dsh-plugin-vision

Auxiliary vision for DeepSeek Harness (dsh): analyze images through an external OpenAI-compatible vision endpoint and get a text answer back. Works with any main model — including DeepSeek, whose own API is text-only.

Why

dsh's built-in read_image tool injects the image into the main model's context, which requires the main model to declare image input. The DeepSeek adapter does not (inputModalities: ['text']), so read_image refuses to run with it. This plugin takes the complementary auxiliary vision route: it sends the image to a separate vision model and returns the answer as text. No main-model vision support needed.

	read_image (built-in)	vision_analyze (this plugin)
Image goes to	the main model's context (native)	an external vision endpoint
Main model must be vision-capable	yes	no
Result	image block the model sees	plain text answer

How it works

1. Read the image — a local file path or an http(s) URL (size-capped, magic-byte MIME sniffing).
2. Encode it as an inline base64 data URL.
3. POST an OpenAI-compatible chat/completions request with [{type:text}, {type:image_url}] content.
4. Return the vision model's answer (falls back to reasoning_content for thinking models).

Two capabilities

1. vision_analyze tool

Send an image (local path or URL) to the configured vision endpoint and get a text answer — works with any main model, including text-only DeepSeek.

2. Image-attachment transcription (pasting images in the Web UI)

The Web UI accepts pasted/dropped images, but the DeepSeek adapter rejects image content (UNSUPPORTED_CONTENT). This plugin hooks agent/pre-step — the documented seam for replacing the messages that enter a step — and, when a user message carries image blocks, calls the vision endpoint to describe them, then swaps each image block for a [User-attached image description] text block before the request reaches the model. DeepSeek only ever sees text, so image attachments just work. When the active model route declares image input, transcription is skipped and native vision takes over.

Toggle with attachImages (default true). Transcription failures degrade to an explicit note instead of blocking the turn.

attachMode controls how the vision model is asked about attached images:

• auto (default): when the user wrote text with the image, that text is passed to the vision model verbatim as its question — "who is this?", "translate the text", "what is wrong on this page?" are answered directly, plus a one-line image summary for follow-ups. Without user text, a generic description is produced.
• describe: always use the generic description prompt, ignoring user text.

3. deepseek-vision provider route (pasting images in the Web UI, root fix)

The Web UI's upload preflight rejects images when the selected model does not declare image input — so pasted images never even reach the agent on the plain DeepSeek route. This plugin registers a deepseek-vision provider: a DeepSeekAdapter subclass that declares image input (preflight passes) and transcribes attached images to text at request time before delegating to the real DeepSeek chat-completions endpoint. The main model is still DeepSeek — same endpoint, same key, same models. Select "DeepSeek (vision via plugin)" in the model picker, then paste/drop images as usual.

Toggle with deepseekVision.enabled (default true); the provider id is deepseekVision.providerId (default deepseek-vision).

No-second-port setup: load the plugin into your existing GUI

The vision API key resolves through the harness credentials service first (the Web UI's stored keys) and falls back to the environment — no export needed in the GUI. Write the plugin into the home-level user patch, which applies to every profile (including the web GUI you already run):

# $DSH_HOME/cordis.patch.yml
- insert:
    - id: vision
      name: 'file:///D:/download%20D/Deepcode/dsh-plugin-vision/lib/index.js'

Add ALIBABA_CODING_PLAN_API_KEY to your stored credentials, then restart the GUI once. Use the built lib/index.js — the published CLI loads .ts entries but cannot resolve their .js-suffixed sibling imports from src/.

4. Image archive (auto-save pasted images + index.json)

Every pasted image is automatically saved to ~/.dsh/image-archive/ with a date-numbered name (2026-08-14_120331_001.png), recorded in an index.json manifest (path, sha256, size, source, optional note), and its location is annotated for the model ([图片已存档: …]). Two tools complete the flow:

• image_archive — the agent archives an important image (user preferences, receipts, key data) into a named folder with an optional note: it saves <archiveDir>/<folder>/<name-or-date>.png and updates index.json.
• image_archive_find — search the manifest by name, folder, or note.

Configure with archive.enabled (default true) and archive.dir (default ~/.dsh/image-archive). Deduplicated by attachment id.

Try it

You need a harness checkout (pnpm install && pnpm run build once), then:

# the vision API key comes from the environment, never from config files
export ALIBABA_CODING_PLAN_API_KEY=sk-...        # Windows: $env:ALIBABA_CODING_PLAN_API_KEY="..."
pnpm dsh web --patch D:/download D/Deepcode/dsh-plugin-vision/cordis.yml

Open the Web UI and ask: Analyze C:\path\to\image.png with vision_analyze and tell me what it shows. Or headless:

pnpm dsh --profile headless --patch D:/download D/Deepcode/dsh-plugin-vision/cordis.headless.yml \
  "Use vision_analyze on C:\path\to\image.png and report what it says"

Windows note: plugin paths in cordis.yml must be file:// URLs (file:///D:/...%20...), not bare D:/... paths — the ESM loader rejects the latter as scheme d:.

Configuration

Key	Default	Meaning
baseUrl	https://token-plan.cn-beijing.maas.aliyuncs.com/compatible-mode/v1	OpenAI-compatible endpoint base URL (any such endpoint works).
apiKeyEnv	ALIBABA_CODING_PLAN_API_KEY	Env var holding the API key.
model	qwen3.7-plus	Vision model id on the endpoint.
timeoutMs	120000	Per-call timeout (thinking vision models need headroom).
maxImageBytes	8388608 (8 MB)	Hard cap on image size.
attachImages	true	Transcribe pasted images to text for text-only main models.
attachMode	auto	auto: pass the user's own prompt to the vision model; describe: always generic description.
deepseekVision.enabled	true	Register the deepseek-vision provider route (DeepSeek + image transcription).
deepseekVision.providerId	deepseek-vision	Provider route id shown in the model picker.

Override any key from cordis.yml:

- id: vision
  name: dsh-plugin-vision   # or a file:// path to src/index.ts
  config:
    model: qwen-vl-max
    timeoutMs: 120000

Develop

pnpm install        # installs the published @deepseek-ai peer packages
pnpm typecheck
pnpm test           # 17 vitest cases: MIME sniffing, payload, parsing, source loading, execute

Known limitations

• Image sources are file paths (resolved against the harness cwd) or plain http(s) URLs. Remote URLs are fetched directly without SSRF hardening — restrict this tool to trusted networks if you use it with URL inputs.
• The image is sent to the configured endpoint as-is; oversized images are rejected rather than downscaled (no Pillow dependency). Compress first for very large screenshots.
• Token costs of the vision call are charged to the configured endpoint's plan.

Publish

This project is a bundle (dsh.bundle.patch). Install with dsh plugin add, share on GitHub with the dsh-plugin topic, or npm publish. See PUBLISH.md in the companion scaffold repo for the full checklist.

License

MIT