dsh-vision-proxy

A host plugin for DeepSeek Harness that gives a text-only model eyes.

The harness rejects image uploads for models without the image modality, and a text-only model cannot read an image anyway. This plugin bridges the gap by delegating image understanding to a second, vision-capable model on any provider the harness already routes:

• Proxy provider — registers a vision-proxy route with one synthetic model (<mainModel>-vision) that advertises image input. Select it in the model picker and attached images work normally: each image is described by the vision model, the description is injected into the request as text, and the request is forwarded to the real main model.
• vision tool — the model can also call it directly with a local image path and an optional question, for images that never entered the conversation.

The main model and the vision model are configured independently. Any provider works in either role — the plugin only needs llm to route it, exactly like your other models.

Install

The package is a plain Cordis plugin with no build step. Two ways to mount it:

From npm or a clone — install the package into your profile and add a loader row:

cd ~/.dsh/profiles/web
npm i dsh-vision-proxy        # or: npm i <path-or-git-url>

In cordis.patch.yml:

- insert:
    - id: vision-proxy
      name: 'dsh-vision-proxy'

Local development — junction the repo into the profile's node_modules and add the same row:

New-Item -ItemType Junction -Path "$HOME\.dsh\profiles\web\node_modules\dsh-vision-proxy" -Target "C:\path\to\vision-proxy"

Node resolves the junction's real path, so dependency resolution starts at the repo location. If your profile keeps its @deepseek-ai packages hoisted one level up (~/.dsh/profiles/node_modules, the pnpm layout), junction that directory into the repo's parent node_modules as well:

New-Item -ItemType Junction -Path "C:\path\to\vision-proxy\..\node_modules\@deepseek-ai" -Target "$HOME\.dsh\profiles\node_modules\@deepseek-ai"

An npm-installed copy (npm i <repo>) does not need this — the package then lives inside the profile tree where the hoisted packages resolve naturally.

Restart the harness. The picker gains a Vision Proxy provider with one entry per text-only model the harness can route (<model>-vision).

Configuration

All config lives in the vision-proxy settings namespace (settings.yaml), the same file where your provider routes live:

vision-proxy:
  mainProvider: opencode-go        # preferred route for the main (text) side; empty = scan live providers
  visionProvider: opencode-go      # route for the vision model; empty = use the main side's provider
  visionModel: mimo-v2.5           # the vision model used for every image
  fallbackVisionModels: []         # optional: models tried in order if the primary call fails
  visionMaxTokens: 2048            # output cap per vision call
  descriptionPrompt: "Describe this image in full detail: ..."   # instruction sent with every vision call

Notes:

• No automatic fallback. fallbackVisionModels is empty by default. Every fallback attempt is a paid model call, so switching models on failure is strictly opt-in. A fixed visionModel fails loudly instead of silently spending.
• mainProvider disambiguates when the same model id exists on several routes (e.g. deepseek-v4-flash on both opencode-go and deepseek-official). Empty means the first live provider that serves the model wins.
• The vision side must be a model the harness can route and that accepts image input. The harness validates this at call time and reports a clear error otherwise.

Switching models

• Main model — pick a different proxy entry in the model picker. deepseek-v4-pro → deepseek-v4-pro-vision, and so on. No config, no restart; each entry inherits the real model's context and reasoning metadata.
• Vision model — either edit settings.yaml, or tell the model "switch the vision model to kimi-k3" and it will call the vision_config tool, which persists the change immediately.

Usage

• Attached images — switch the session to a <model>-vision entry in the model picker and attach normally. Each image is described before the request goes to the main model. If a description fails, a placeholder text is injected instead of failing the turn.
• Local images — ask the model to look at a file, e.g. "what does screenshot.png show?" It will call the vision tool. Supported formats: png, jpg, jpeg, webp, gif (subject to the harness attachment limits, ~5 MB by default).

How it works

user attaches image ──▶ vision-proxy/<model>-vision selected
                          │
                          ▼
                  vision-proxy adapter (this plugin)
                          │  for each image block:
                          │    llm.stream({ provider: visionProvider, model: visionModel,
                          │                  messages: [image, prompt] })   ← one paid call per image
                          │  image block → [image, described by <model>: ...]
                          ▼
                  llm.stream({ provider: <main route>, model: <main model> })  ← the real model

The main route is derived from the proxy model id and resolved against the live provider catalog — the session's default selection is never consulted, so choosing a proxy model cannot make the plugin delegate to itself. The proxy model's context-window and reasoning metadata are inherited from the real model at request time, so the harness budgets the route exactly like the model it runs.

Limitations

• The proxy route is a dynamic adapter: sessions that ran under <model>-vision need this plugin loaded to replay. Restart the harness with the plugin mounted and it is always there.
• One vision call per image, priced per your vision provider.
• No GUI settings page in this version; edit settings.yaml (or the settings document) instead.

License

MIT