dsh-vision-bridge — DSH Web GUI dynamic vision bridge

English | 中文

A cordis plugin for the DSH (DeepSeek Harness) Web GUI that enables pasted images: the UI accepts image attachments, and the main model (DeepSeek) decides on demand which vision model to call — qwen3-vl-flash for visual Q&A and qwen-vl-ocr for plain text extraction — via Alibaba Bailian's OpenAI-compatible endpoint.

Features

• Pasted images in the GUI — registers a deepseek-vision wrapping route declaring inputModalities: ['text', 'image'], so the chat box accepts image paste.
• Dynamic vision (zero pre-translation) — pasted images are not pre-transcribed; the image block is replaced with a hint (sha256 attachment id) and the main model writes the vision prompt per question/context before calling the tool. No extra calls, no upfront guessing.
• Two vision tools:
  • describe_image(image, prompt) — visual Q&A (qwen3-vl-flash), prompt written by the main model on each call.
  • extract_text(image) — OCR (qwen-vl-ocr), extracts all text in reading order.
• Three image input forms — http(s) URL / local file path / pasted attachment id (sha256:…).
• Robustness — MIME sniffing (PNG/JPEG/GIF/WebP), 20 MB cap, explicit 401/403 hints, timeout and empty-response guards.

What it does

Capability	Status
GUI accepts pasted images	✅
Visual Q&A via describe_image (qwen3-vl-flash)	✅
OCR via extract_text (qwen-vl-ocr)	✅
Prompt decided by main model per request (no fixed template)	✅
URL / local path / sha256 attachment inputs	✅
MIME sniffing + size limits + clear errors	✅

Architecture

The plugin is a single-file cordis native plugin that wraps the official deepseek-official adapter into a deepseek-vision route, and calls Bailian directly from the tool implementations.

flowchart LR
  U[User pastes an image] --> G[GUI allows it<br/>inputModalities text+image]
  G --> W[deepseek-vision route<br/>wraps deepseek-official]
  W --> H[hintBlocks: image block →<br/>text hint with sha256 id]
  H --> M[Main model DeepSeek<br/>sees the hint]
  M -->|visual Q&A| D[describe_image<br/>qwen3-vl-flash]
  M -->|extract text| E[extract_text<br/>qwen-vl-ocr]
  D --> B[Bailian compatible-mode<br/>/chat/completions]
  E --> B
  B --> R[Text result back into chat]

Key design point: the vision prompt is fully decided by DeepSeek on every call, based on the user's question and conversation context — asking about the "amount" vs. the "tax number" on the same invoice sends different prompts, maximizing information and minimizing calls.

Usage

Drop the plugin into your DSH Web profile plugin directory:

cp vision-bridge.js ~/.dsh/profiles/web/plugins/

It depends on the llm and tools services and requires the official llm-deepseek adapter (deepseek-official) to be loaded — the plugin wraps it to register the deepseek-vision route.

Configure your Bailian API key in ~/.dsh/.env (either works):

DASHSCOPE_API_KEY=sk-xxxx
# or
VISION_API_KEY=sk-xxxx

Get a key from the Alibaba Cloud Bailian console.

Development

No build step — the plugin is a single plain-JS file. Edit, restart the web server, done. The model names are constants at the top of the file (VISION_MODEL / OCR_MODEL); swap them for other Bailian models (e.g. qwen-vl-max, qwen3-vl-plus) and restart.

Notes

• Attachments must have been sent in the session before they can be read (the plugin looks them up by sha256 in the DSH attachment store).
• This plugin only solves seeing images — the main DeepSeek model has no vision input, so DSH's built-in read_image tool is not usable on its own.

License

MIT