VISTA Desktop MCP

Built on VISTA: A Visual Harness for Reasoning in an Interactive World by Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, and Kaiming He (2026). See the paper and project website.

This local stdio MCP adapter adds desktop tools to normal Codex sessions. It reuses a vendored copy of VISTA’s SessionController for archived-frame inspection, cropping, and pixel sampling. It does not launch the Linux/Docker game harness, change the model, or require an ARC/API key. This is a local adaptation, not an upstream VISTA Windows desktop feature.

Install

This project is standalone; it does not need the original VISTA checkout. Requires an interactive Windows desktop, Python 3.12, and uv. Clone and install:

git clone https://github.com/yjr-123456/vista-desktop-mcp.git
cd vista-desktop-mcp

uv venv --python 3.12 .venv
uv pip install --python .venv/Scripts/python.exe -r requirements-lock.txt
.venv/Scripts/python.exe -m unittest discover -v
.venv/Scripts/python.exe smoke_test.py
.venv/Scripts/python.exe install_codex.py --with-skill

The installer backs up ~/.codex/config.toml (or $env:CODEX_HOME/config.toml) and adds only [mcp_servers.vista_desktop]. Existing differing entries are not overwritten. Keep this checkout and .venv at their configured paths. Restart Codex after installation; already-running sessions may not discover it. --with-skill also installs vista-computer-use under the Codex skills directory. An existing differing skill is backed up before updating its bundled files.

For a real input test, run smoke_test.py --gui. It briefly opens its own test window, checks entry focus, types Chinese text, tests Ctrl+A/Backspace, then closes the window. It uses a temporary archive that is removed on exit.

Usage

Ask Codex, for example:

使用 vista_desktop 查看当前桌面，放大检查目标窗口，再按我的要求操作。

The MCP server supplies usage instructions automatically. The companion vista-computer-use skill explains workspace binding, coordinate conversion, outcome verification and evidence-based recovery. The compatibility notes compare GitHub/ClawHub skills and document tool mappings and unsupported features. Its twelve tools are:

The skill includes reviewed workflow guidance and desktop_helpers.py for coordinate mapping, window diagnosis, archived-image comparison and image-hash verification. Window focus and frame comparison are also exposed directly by the MCP. Sources, versions, adaptation details and retained MIT licenses are documented in the skill's source notes. No additional third-party computer-use skill or runtime is installed.

Bind the actual conversation workspace first with configure_session, then observe. All input and crop coordinates are original physical pixels, not scaled preview pixels. An inspect response gives both the source region and rendered image dimensions. For a crop, map rendered (u,v) back as x = region.x + u * region.width / image_size.width, and similarly for y. Act once, inspect the result, then proceed. After an action error, observe before retrying because input may have been delivered before capture failed.

Workspace-local multimodal memory

Global MCP processes cannot be assumed to inherit the current conversation's working directory. The default server creates no archive until the agent calls:

{"workspace":"E:/Code_Space/MyProject"}

Use the real task directory with configure_session, not the MCP installation directory. The server requires an existing absolute workspace and returns the canonical session_dir. No user prompt is needed when the directory is already known from the conversation environment.

<workspace>/
  .agents/
    vista-desktop/
      .gitignore
      sessions/
        <session-id>/
          session.json       # schema, provenance, coordinate convention
          frames/
            turn_000_frame_000.png
          history.jsonl      # action metadata, native size, relative path, hash
          memories.jsonl     # agent observations linked to visual evidence
          MEMORY.md          # readable observations with relative frame links
          GUIDE.md           # reusable verified knowledge
          WORKING.md         # objective, current state, next step
          .writer.lock       # OS-managed exclusive writer lease

memories.jsonl/MEMORY.md appear on the first remember call. Frames stay lossless; previews may be scaled. Relative frame paths and SHA-256 hashes keep evidence traceable if the workspace moves. Absolute paths in old history and the creation-time workspace in session.json are provenance; use session_dir plus relative_path after a move. Only the memory subtree is ignored by Git, not the entire .agents directory or project skills.

The generated ID identifies a VISTA memory session, not a Codex thread ID. Repeated binding to the same workspace in the same process retains the current session unless a new ID is supplied. After restart, use sessions to find the old ID and configure_session(workspace=..., session_id=..., resume=true) to continue it. A concurrent writer is rejected. Historical read tools accept session_id within the bound workspace. A different workspace has a separate archive namespace.

Use remember(summary, turns, tags) at meaningful verified milestones, then recall(query) and inspect for retrieval. Recall is text/tag matching, not embedding search. Summaries and GUIDE/WORKING notes are agent-authored; they are not inferred automatically or silently injected after compaction. The skill directs the agent to recover and check them, then observe the current desktop.

Frames are retained until their archive is explicitly removed. Typed text is omitted from action metadata but can be visible in screenshots. No third-party memory service is added; images returned to Codex still participate in normal model/tool processing.

Explicit launch options and old archives

server.py --workspace <absolute-path> prebinds a known workspace. Avoid pinning this in global Codex config if the server is used across projects. --session and --resume can select an existing session for either launch option.

The old %LOCALAPPDATA%/VISTA/desktop/ archives are left intact: their originating workspace cannot be reliably inferred. To read/resume them explicitly, launch server.py --data-dir <old-root> --session <old-id> --resume. --data-dir is also an explicit override for users with another storage convention; it cannot be combined with --workspace. New default sessions no longer write there.

Scope and controls

• Windows interactive primary monitor only; tested at 3840x2160. Secondary monitors, UAC secure desktop, and locked/remote-disconnected sessions are not supported.
• Uses screenshot reasoning and keyboard/mouse input, without adding OCR/UIA.
• Unicode input does not replace the clipboard. Some games and elevated apps may reject injected input; this adapter does not elevate privileges.
• Moving the mouse to a primary-screen corner triggers the PyAutoGUI fail-safe for input actions. observe remains available.
• Avoid simultaneous desktop actions from multiple Codex sessions or manual input while an action is running; the lock serializes only one server process.
• Disable by setting enabled = false in [mcp_servers.vista_desktop], then restart Codex. Remove that table to uninstall registration. A dated configuration backup is kept beside config.toml; avoid restoring it wholesale after other edits.

Codex MCP configuration reference: https://developers.openai.com/learn/docs-mcp

Acknowledgments and citation

The visual-memory foundation comes from VISTA, including archived-frame inspection, crop validation, pixel sampling, and persistent working-memory conventions. This project adds a Windows desktop MCP adapter and a companion Codex skill. It is an independent adaptation, not an official VISTA release.

The vendored SessionController comes from upstream commit c97c354. See vendor/README.md for implementation provenance and THIRD_PARTY_NOTICES.md for redistribution notices. If you use the VISTA work in research, please cite the original authors using the citation supplied in the upstream README:

@misc{han2026vista,
  title         = {{VISTA}: A Visual Harness for Reasoning in an Interactive World},
  author        = {Han, Qiushi and Hu, Keya and Qiu, Linlu and Wu, Cathy and He, Kaiming},
  year          = {2026},
  eprint        = {2610.02200},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2610.02200}
}

License

This project is licensed under the MIT License. The vendored VISTA code retains its original copyright notice and MIT license. Other adapted material retains its respective attribution and MIT notices in the skill's source notes and license directory.

Performance

Default response previews use JPEG quality 90 (4:4:4); original evidence remains full-resolution lossless PNG with SHA-256. observe/play accept preview_format="png" and optional preview_region={x,y,width,height}. Map coordinates through the returned region offset and preview dimensions. Archives remain full-screen. Reuse action-result images instead of redundant observations.

Returned timings_ms separates capture/window-context/archive/preview processing, excluding action waits, MCP transport, model inference and hooks. Move/drag use seconds once as movement duration. Archive hashing reuses encoded bytes without rereading disk.

Run .venv/Scripts/python benchmark_preview.py for seven encoding samples of one live frame, without input or saved screenshots. On 2026-10-25: PNG 32.36 ms / 569326 bytes; JPEG 4.20 ms / 280864 bytes. Screen content affects results; these are not end-to-end timings.

Client hooks and shell startup can dominate end-to-end latency. If calls stall while server timings remain low, diagnose the client launch/hook path separately.

Reconnect/restart the MCP after updating to load the implementation.

Focus recovery and guarded input

windows(focus) returns focused=false, status=focus_denied and a fresh full-screen preview when Windows refuses activation. Inspect that evidence before choosing a recovery action; no automatic click or keyboard fallback occurs. Stale HWND/PID validation remains an error.

play optionally accepts expected_hwnd and expected_pid together. A foreground mismatch returns status=not_applied, reason=foreground_changed and a full-screen preview without sending input. This checks top-level window identity immediately before dispatch; it does not establish field focus or eliminate concurrent races. The skill uses these responses directly, avoiding a redundant observe round trip.

Shared observation context

configure_session(initial_observe=true) optionally returns binding metadata and the first screenshot in one call; the default remains metadata-only. Every new screenshot includes best-effort active_window metadata sampled just after capture. Reuse its HWND/PID for guarded input instead of a separate foreground query. A window-query failure returns active_window=null plus window_context_error and preserves the screenshot. This metadata is not atomic with capture and does not prove field focus. These capabilities apply to desktop applications generally.