embodied-nav-mcp
A language model that can look around a room, go to what it names, count what it sees, and stop when it is done.
The model decides what to do. The sensors decide where things are.
Tools
| Tool | What it does |
|---|---|
get_view |
Four perspective faces from the 360° camera, plus pose |
ground |
Referring expression → pixel box → lidar metres → converter-snapped waypoint |
drive_to |
Publish one map-frame (x, y) and wait until arrived or settled |
stop |
Park so the planner stops chasing the last waypoint |
parse_numerical |
Split a counting question into target / anchor |
count_in_view |
Count this vantage only; each instance is lidar-lifted |
commit_count |
Cluster looks by position; publish the integer |
publish_object |
Lidar AABB of the last ground; publish a 3D marker |
commit_count does not take a number from the model. Two views of four pillows become 4, not 8.
Geometry is the Type 3 stack from Team Xiao Hei’s feat/xiao-hei-instruction-merge: size gate, blind-cone handling, converter settle pose, park-on-exit.
Install
uv sync --extra dev
export ANTHROPIC_API_KEY=...
uv run pytest
Python 3.11+. Live robot/sim needs ROS Jazzy and the autonomy stack. Tests use a fake robot and need neither.
Try it
# Dry run, no ROS
uv run embodied-mcp-agent --fake "How many blue chairs are between the table and the wall?"
uv run embodied-mcp-agent --fake "Find the potted plant on the kitchen island closest to the fridge."
uv run embodied-mcp-agent --fake "Take the path near the window to the fridge."
On a robot (or in the challenge container) with the stack up:
uv run embodied-mcp-challenge
# evaluators still run: ros2 launch dummy_vlm dummy_vlm.launch
embodied-mcp --attach-ros is the same tools over stdio MCP. The challenge node calls them in-process: a robot container has no public URL for a remote MCP connector.
CMU VLN Challenge
Drop-in adapter: examples/cmu_vln/. Package name dummy_vlm is unchanged.
| Type | Tools | ROS out |
|---|---|---|
| Numerical | parse_numerical, count_in_view, commit_count |
/numerical_response |
| Object reference | ground, publish_object |
/selected_object_marker |
| Instruction-following | get_view, ground, drive_to, stop |
/way_point_with_heading |
Allowed sensors: /camera/image, /registered_scan, /terrain_map, /terrain_map_ext, /state_estimation. Arrival is /state_estimation, not /way_point_reached. One question per process.
Configuration
| Variable | Default | |
|---|---|---|
ANTHROPIC_API_KEY |
— | required |
EMBODIED_MCP_MODEL |
claude-opus-5 |
host and pixel-grounding / counting model |
EMBODIED_MCP_BUDGET_S |
540 |
seconds from process start |
XIAO_HEI_IMAGE_TOPIC |
/camera/image |
camera topic |
License
MIT. Vendored geometry and ROS I/O from Team Xiao Hei; see NOTICE.