dsh-plugin-window-input

A self-contained DeepSeek Harness plugin that lets the agent attach to a window on the local Windows desktop and type into it — or drive the mouse — no core modifications needed.

Load it into a stock DSH via --patch. Works on Windows only.

What you get

Tool	What it does
window_list	Enumerate visible top-level windows (title, class, process name, hwnd), optionally filtered by a query.
window_focus	Bring a window to the foreground by title (substring) or hwnd.
window_type	Type text into a window's input control — the control is auto-detected (focused control → UI Automation → Scintilla → SendInput).
mouse_move	Move the cursor to absolute screen coordinates (x, y).
mouse_click	Click a point (x, y) or an element in a window — give title/hwnd + name (visible label, e.g. "Send") and optional controlType; activation goes through UI Automation when available, otherwise a physical click at the element's position.
mouse_scroll	Scroll: delta ±120 per notch. With title/hwnd the wheel is sent straight to that window's focused control (deterministic, remote-session safe); otherwise injected at the current cursor or x/y.

Example flows

window_list                    → find the "Notepad" window
window_type title="Notepad" text="hello" pressEnter=true

# click the ChatGPT composer's "Send"-style button by label:
mouse_click title="ChatGPT" name="Send"
# scroll the same window down 3 notches:
mouse_scroll title="ChatGPT" delta=-360
# move + right-click at a screen point:
mouse_move x=800 y=450
mouse_click x=800 y=450 button="right"

How window_type finds the input box

Typing into a window needs more than its top-level handle — it needs the control that is waiting for keyboard input. The engine resolves it in layers:

1. Focused control — GetGUIThreadInfo returns the thread's focused child, the authoritative keyboard target. Works for any app.
2. UI Automation ValuePattern — for standard controls (Notepad, editors, chat inputs) it sets the value directly, no focus needed.
3. Scintilla editors (Notepad++, many code editors) — text is inserted cross-process via clipboard + SCI_PASTE (pointer-free and safe; the pointer-based SCI_REPLACESEL/SCI_INSERTTEXT do NOT work cross-process).
4. SendInput (opt-in method="sendinput") — for terminals/apps that only accept keyboard injection. Some environments (remote desktop, security software, restricted tokens) block synthetic input injection.

pressEnter inserts a newline via \n (UI Automation) or SCI_NEWLINE (Scintilla), so it works even where SendInput is blocked.

Requirements

• Windows
• A DeepSeek Harness checkout (pnpm dsh web)

Installation

# 1. clone this repo anywhere
git clone https://github.com/linxsy-code/dsh-plugin-window-input.git

# 2. edit cordis.yml — point name at YOUR clone path (a file:// URL)
#    e.g.  name: 'file:///C:/path/to/dsh-plugin-window-input/src/index.ts'

# 3. from the DeepSeek Harness checkout root:
pnpm dsh web --patch <path-to-this-repo>/cordis.yml

Open http://127.0.0.1:3080. In a new session, ask the agent e.g. "use window_type to type 'hello' into Notepad and press Enter" — or if you also use the @ window menu, pick a window with @窗口.

Self-test & diagnostics

The mouse paths depend heavily on the environment (remote-desktop / restricted sessions may block injected input or drop injected wheel events). Run the suite on your machine to verify the engine really works and which scroll mode to prefer (small windows pop up briefly — that is expected):

powershell -ExecutionPolicy Bypass -File selftest.ps1
# against a different engine copy:
powershell -ExecutionPolicy Bypass -File selftest.ps1 -Engine C:\path\to\win-agent.ps1

It runs three checks and reports RESULT: PASS/FAIL for each:

Check	Proves
test-click	a coordinate click activates a window (mouse injection works)
test-element-click	an element click by visible name dismisses a real dialog
test-scroll	direct WM_MOUSEWHEEL delivery scrolls a document (remote-session safe)

tools/env-wheel-probe.ps1 then answers "are injected wheel events delivered in this session?" — if not, always use mouse_scroll with a title/hwnd.

Files

• src/index.ts — the plugin (registers the six tools, spawns win-agent.ps1).
• win-agent.ps1 — the engine (Win32 P/Invoke + UI Automation + Scintilla messages + mouse, emits JSON).
• cordis.yml — the --patch overlay.
• selftest.ps1 + tools/ — self-test suite & environment diagnostics (not part of the plugin at runtime).

How the mouse works

• Move — SetCursorPos (absolute screen pixels).
• Click — a click is MOUSEINPUT down/up via SendInput (this works in environments where keyboard SendInput is blocked — error 87 — because the block is per-input-type; verified). Element clicks first try a UIA activation pattern (Invoke/Select/Toggle/Expand); if the app exposes none (some environments degrade native Win32 controls to a generic pane), the engine falls back to a physical click at the element's bounding-rectangle center. Control-type is only a hint: matching is name-first so degraded trees still hit.
• Scroll — two delivery modes. direct (the default whenever title/hwnd is given) sends WM_MOUSEWHEEL synchronously to the target window's focused child control — deterministic, and it works in remote/locked-down sessions where injected wheel events are silently dropped. input injects real wheel events at the cursor/point for browsers and apps that only scroll on true input.

Known limitations

• SendInput keyboard input may be blocked in remote-desktop / sandboxed / restricted-token environments (Win32 error 87) — prefer the default UI Automation / Scintilla paths. Mouse SendInput and SetCursorPos are generally not blocked, but injected wheel can be dropped in the same environments — use mouse_scroll with a title/hwnd (direct delivery) there.
• UI Automation SetValue only reaches controls that expose ValuePattern; canvas-only or exotic controls fall back to the focused-control path.
• Element names must be visible text or accessible names; icon-only controls without accessible names can't be matched by name (click them by coordinates instead).
• Windows only.

@窗口 mention (optional, requires the fork)

A companion feature adds a 窗口 (Windows) category to the @ mention menu in the DSH Web composer: type @, pick a window, and it inserts @窗口 <title> that the agent resolves through window_type. Because it touches the client Remote assembly, it lives in a fork of DSH rather than a --patch plugin:

https://github.com/linxsy-code/deepseek-harness   (branch master)

License

MIT © 2026 linxs

---

dsh-plugin-window-input（中文）

让 DeepSeek Harness 的 Agent 能「附着到某个 Windows 窗口往里输入文字」，或直接操控鼠标，无需改动核心代码。

工具

• window_list：枚举可见顶层窗口（标题/类名/进程/hwnd），可按关键字过滤。
• window_focus：按标题（子串）或 hwnd 把窗口切到前台。
• window_type：向窗口输入文字，自动识别输入框（焦点控件 → UI Automation → Scintilla → SendInput）。
• mouse_move：把光标移到屏幕坐标（x,y）。
• mouse_click：点坐标，或点窗口内元素——给 title/hwnd + name（可见文字，如 "Send"）+ 可选 controlType；优先走 UIA 激活模式，不支持时回退为该元素屏幕位置的物理点击。
• mouse_scroll：滚动，delta 每格 ±120。带 title/hwnd 时把滚轮消息直接发给该窗口的焦点控件（确定可靠、远程会话也可用）；否则在光标处（或 x/y）注入滚轮事件。

安装

git clone https://github.com/linxsy-code/dsh-plugin-window-input.git
# 编辑 cordis.yml，把 name 改成你本机克隆路径（file:// URL）
cd <deepseek-harness-checkout>
pnpm dsh web --patch <本仓库>/cordis.yml

打开 http://127.0.0.1:3080，新会话里 Agent 就能用这些工具，例如：

window_list
window_type title="记事本" text="你好" pressEnter=true
mouse_click title="ChatGPT" name="发送"      # 按元素名点击
mouse_scroll title="ChatGPT" delta=-360      # 向下滚动 3 格
mouse_move x=800 y=450 ; mouse_click x=800 y=450 button="right"

已知限制

• SendInput 键盘输入在远程桌面/受限令牌等环境可能被拦截（Win32 87），请用默认的 UIA/Scintilla 路径；鼠标 SendInput/SetCursorPos 一般不受限，但注入式滚轮可能被同一类环境丢弃——这种情况请给 mouse_scroll 带 title/hwnd（直发模式）。
• 元素点击按可见文字匹配，纯图标且无无障碍名的控件匹配不到，改用坐标点击。
• 仅支持 Windows。

自测与诊断

鼠标行为强依赖运行环境（远程桌面/受限会话可能拦截注入或丢弃注入式滚轮）。在你的机器上跑一次自测，确认引擎可用、并知道该用哪种滚动模式（期间会短暂弹出小窗口，属正常）：

powershell -ExecutionPolicy Bypass -File selftest.ps1
# 指定另一份引擎副本：
powershell -ExecutionPolicy Bypass -File selftest.ps1 -Engine C:\path\to\win-agent.ps1

三项检查各输出 RESULT: PASS/FAIL：

检查	验证
test-click	坐标点击能激活窗口（鼠标注入可用）
test-element-click	按可见名称的元素点击能真的关掉一个对话框
test-scroll	直发 WM_MOUSEWHEEL 能真的滚动文档（远程会话也可靠）

tools/env-wheel-probe.ps1 回答「本会话注入式滚轮是否被投递」——若被丢弃，请一律给 mouse_scroll 带 title/hwnd。

@窗口 提及（可选，需 fork）

配套功能可在 @ 弹出菜单里加「窗口」类别列出所有窗口、选中插入 @窗口 <标题> 供 Agent 解析。因涉及客户端 Remote 装配，只提供在 fork 里：github.com/linxsy-code/deepseek-harness（master 分支）。