dsh-win-computer-use

Windows computer-use toolkit for DeepSeek Harness — give your text-only DSH agent hands on the Windows desktop.

UIA-semantic observation/control, window screenshots, built-in vision API (see images without a multimodal model), human-like bezier cursor motion, and a high-performance Rust helper backend.

Inspired by Anionex/dsh-computer-use (macOS), Harusame64/desktop-touch-mcp (Rust UIA engine) and sbroenne/mcp-windows.

Architecture

DSH Agent → win_* tools → executor (win-cu.ps1 / Rust helper) → Win32/UIA
                                             ↘ built-in vision API (win_see)

Two interchangeable executors, switchable at runtime via config.json (backend: ps1 | rust):

	PowerShell (ps1)	Rust helper (rust)
Startup cost	powershell.exe ~200ms + Add-Type compile	single binary ~10ms
UIA tree fetch (60 elements)	~350ms	~106ms
Focused element	~370ms	~2ms
Vision	curl + PS	built-in (WinHTTP/curl)

Performance reference from desktop-touch-mcp's measured PowerShell → Rust COM migration.

Tools

Tool	Description
win_list_apps	List desktop windows: pid / process / title / bounds / foreground
win_observe	UIA tree: index / type / name / AutomationId / bounds / patterns
win_screenshot	PrintWindow PNG capture
win_click	Semantic-first (Invoke/Select/Toggle by id or index), bezier-cursor coordinate fallback
win_set_value	ValuePattern set (no clipboard)
win_type_text	SendInput Unicode (Chinese OK), auto focus + focus restore
win_press_key	Virtual keys + modifiers, focus restore
win_scroll	Mouse wheel with bezier cursor move
win_drag	Bezier-path drag
win_wait	Poll UIA tree for element text
win_see	Built-in vision: any image → configured vision API → text description
win_launch	Launch an application

Design notes

• Observe then act: use stable AutomationIds for actions — UIA tree indices shift when apps re-render (learned the hard way).
• Visible, natural cursor: coordinate actions move the real cursor along a bezier path (random control points + easing + micro-jitter), so you see it move.
• Focus protection: type/key/scroll bring the target window to foreground, then restore your previous foreground window.
• Built-in vision loop: win_see sends screenshots straight to your configured OpenAI-compatible vision endpoint — works even with a text-only model.

Terminal console (win-cu)

Run win-cu in any PowerShell to open a persistent console (menu + live panel, no flicker):

  >> 视觉       |  VISION_API_KEY   **********
     控制       |  VISION_BASE_URL  https://ai.huan666.de
     运行时     |  VISION_MODEL     grok-4.5
     状态       |  AUTO_SCREENSHOT  [x] 开启
                |  [S] 保存   [T] 测试   [Tab] 字段   [q] 退出

Keys: Up/Down switch menu (right panel live-renders) · Tab next field · Enter edit/cycle · Space toggle · S save · T test · q quit. Edits persist to config.json — the DSH plugin reads it live on every call, no restart needed.

Config

C:\Users\XHY\.win-computer-use\config.json:

{
  "visionApiKey": "sk-...",
  "visionBaseUrl": "https://ai.huan666.de",
  "visionModel": "grok-4.5",
  "language": "zh",
  "autoScreenshot": true,
  "cursorSpeed": "normal",
  "cursorVisible": true,
  "focusRestore": true,
  "clickFallback": true,
  "backend": "rust",
  "rustHelperPath": "C:\\...\\win-cu-helper.exe"
}

Repository layout

src/main.rs       Rust helper (UIA + SendInput + bezier + PrintWindow + built-in vision)
win-cu.ps1        PowerShell backend (default executor)
win-cu-tui.ps1    Persistent terminal console (menu + panel)

License

MIT