mcp-phone-use
MCP stdio server that controls an attached Android phone over ADB — the phone twin of mcp-computer-use.
~/.pi/agent/mcp.json -> bun /workspace/mcp-phone-use/server.coffee
The simple interface
| Tool | What it does | Image? |
|---|---|---|
phone_screenshot |
Capture screen; crop {x,y,width,height} + grid to zoom |
yes |
phone_click |
Tap at x,y in the last image's pixels | yes (unless observe:false) |
phone_type |
Type text into the focused field (enter:true to submit) |
yes |
phone_key |
Send a key, a chord (Ctrl+A), a sequence (Ctrl+A,Delete), repeat:N |
yes |
phone_find_element |
Substring search over the live UI tree + tap centers | no |
phone_ui_dump |
Live uiautomator tree (text, resource-id, content-desc, class, bounds, center, focused) |
no |
phone_info |
serial / model / android / screen px | no |
phone_clear_text |
Clear a field (Ctrl+A+Delete, else long-press → Select all → Cut) | no |
phone_long_press, phone_swipe, phone_interact |
long press, swipe, ordered tap→text→keys | yes |
phone_keyboard |
escape hatch: status / attach / detach of the virtual keyboard | no |
Virtual HID keyboard (the important part)
phone_type and phone_key send real keystrokes on a virtual HID keyboard the server creates and manages itself. Callers never attach or detach it.
How it works (all verified on a real device running Android 17, no root, no USB gadget, no BlueZ):
- Android ships
/system/bin/hid(frameworks/base/cmds/hid), which writesUHID_CREATE2to/dev/uhidthroughlibhidcommand_jni. - adb's
shelluid is in theuhidgroup:uid=2000(shell) ... groups=...,3011(uhid),...and/dev/uhidiscrw-rw---- uhid uhid. SELinux isEnforcingand it still works. adb shell hid -is kept alive with a pseudo-JSON stdin stream (register→delay→report), which creates a genuine input device:/dev/input/event4,Classes: KEYBOARD | EXTERNAL, visible indumpsys input,geteventshowsKEY_LEFTCTRL/KEY_AwithMSC_SCAN 000700e0.- Because it is a physical keyboard to the OS, hardware-keyboard semantics work: Ctrl+A select-all, Ctrl+C/V/X, Tab, Escape, normal typing.
Measured comparison on the same field:
| Method | Ctrl+A select-all | Typing | Notes |
|---|---|---|---|
input keycombination 113 29 (adb chord) |
✗ typed a literal a |
✓ | API-level injection; apps ignore the injected meta |
input keyevent 67 |
n/a | one key | works, but no modifiers |
| virtual HID keyboard | ✓ (Ctrl+A then Delete wiped the whole field; Ctrl+C → Delete → Ctrl+V pasted it back) | ✓ (incl. @ as Shift+2) |
real device, real meta state |
The one gotcha: VID/PID
Use a VID/PID with no /system/usr/keylayout/Vendor_XXXX_Product_YYYY.kl, otherwise EventHub maps our HID usages through that vendor layout and every key silently does nothing. The first attempt used a well-known vendor pair (18d1:2c40) that ships on the tested device as a remote-control layout — keys reached the kernel and InputReader (KeyState (pressed): KEY_LEFTCTRL, KEY_A) but nothing reached the app. Default is now vid=0x1234 pid=0x9999 → falls back to Generic.kl (US layout).
Lifecycle
- Attached implicitly by the first
phone_key/phone_type/phone_clear_text. - Auto-detaches after
idleMs(default 240 s) so the phone does not keep a phantom keyboard. - Killed on
SIGINT/SIGTERM; closing thehidprocess' stdin unregisters the device (EOF → unregister). - If
/dev/uhidis denied (locked-down OEM builds), the server silently falls back toinput text/input keyevent/input keycombination.
Screenshot-quota loop
- Read/locate with
phone_find_element/phone_ui_dump— text only, no image. - Send input with
observe:false— no screenshot returned;phone_key/phone_type/phone_clear_textneed noscreenshotId. - Batch:
phone_key { keys:"Delete", repeat:25, observe:false }= 25 backspaces, one call. - Screenshot only when you must see pixels (or need a
screenshotIdfor coordinate taps).
Android 17 quirks (measured)
input long-pressdoes not exist (Unknown command: long-press) → long press = zero-distance swipe with duration:input swipe X Y X Y 1200.input keyeventaccepts key names;input keycombination [-t ms] <codes...>is the chord API but its meta state is ignored by most apps.- uiautomator bounds are
bounds="[l,t][r,b]"(bracketed pairs) —scripts/ui-dump.pyhandles both; it also reportsfocused/selected. KEYCODE_CTRL_LEFTis 113 (not 291). HID side: left-ctrl is modifier bit0x01,Ais usage0x04, Left-Arrow is0x50(verified by cursor-position test).
Layout
server.coffee tool definitions + JSON-RPC wiring
shared/mcp-stdio.mjs minimal newline-delimited MCP stdio host
lib/adb.mjs device resolution, screencap/pull, uiautomator dump, shell, spawnShell
lib/hidkbd.mjs virtual HID keyboard: UHID register/report, HID usage tables, idle auto-detach
lib/phone.mjs taps, long-press, text, keys, clear_text, ui dump (HID preferred, adb fallback)
lib/views.mjs screenshotId registry (120s TTL), crop/point mapping, geometry guard
lib/schema.mjs tiny JSON-Schema subset validator
scripts/screenshot-view.py crop/scale/grid (Pillow)
scripts/ui-dump.py XML -> JSON elements
Run
bun /workspace/mcp-phone-use/server.coffee # stdio MCP server (also launched from mcp.json)
pi mcp list # validate the mcp.json entry
Requires adb and python3 + Pillow (requirements.txt). Multiple devices: set ADB_SERIAL=<serial>.