dsh-plugin-multimodal

English | 中文

Image recognition and image generation for DeepSeek Harness (dsh-plugin, a Cordis plugin).

The plugin is just tools: image_recognize, image_generate, and vision_providers call whichever external API models the deployer declares — vision via multimodal chat/completions; image generation via OpenAI-compatible images/generations or the Aliyun DashScope native async-task protocol. No model is bundled or defaulted — a tool invoked without a configured model fails with a clear, actionable error. It never guesses and never fails silently.

Demo

Image recognition	Image generation

Features

• image_recognize — analyze an image (local path / http(s) URL / data URI) through a configured vision model: describe, OCR, read charts/screenshots, review content, …
  • Local files are inlined as base64 data URIs (25 MiB cap)
  • prompt (what to ask), max_tokens / temperature, per-call provider selection
• image_generate — generate or transform an image through a configured image model, save it to disk, and return its file path plus a Web-renderable URL when the static server is enabled
  • Text-to-image: OpenAI-compatible images/generations (prefers b64_json, falls back to URL download), or protocol: dashscope-native for the DashScope async task API (submit → poll → download), needed when an OpenAI-compatible gateway exposes no image routes (some Aliyun deployments)
  • Image-to-image: pass an image argument (source image) to generate from it (edit / variant / style transfer). OpenAI-compatible endpoints use the image field; DashScope native uses base_image_url
  • Watermark-free by default: OpenAI-compatible requests carry watermark: false (Doubao Seedream etc. support it; endpoints rejecting the param are retried without it). Set watermark: true on the provider to keep the vendor's mark
  • Inline in the Web chat: a built-in static server (default 127.0.0.1:3081) exposes generated images as absolute http(s) URLs; the tool returns Markdown ![](url) lines the model can paste into its reply so the GUI renders them (the GUI's Markdown renderer allows only absolute http(s) image URLs — local paths, relative links and data URIs never render; the harness webserver serves no arbitrary files, and writing into the frontend dist directory does not work, tested)
  • Real-format extensions: base64 payloads get their extension from the magic bytes (PNG/JPEG/WebP/GIF) so Content-Type matches the content
  • size (e.g. 1024x1024), n (1–4, auto-suffixed filenames), output_path (directory or file)
  • Default save location: <caller workspace>/generated/, timestamped filenames
• vision_providers — list every configured external model (id, kind, model, endpoint, apiKey status) so the model can pick a provider id
• Multiple models — the providers list accepts any number of entries; each tool call may select one via provider, defaulting to the first of the matching kind
• Not configured → clear error — no provider of the kind, unknown provider id, empty apiBase, model, or apiKey all raise actionable Chinese errors when the tool is called; missing connection settings never prevent DSH Web from starting

Install into a DSH profile

The DSH plugin command delegates installation to pnpm, so pnpm must be on PATH regardless of how DSH itself is launched. Build this local checkout before installing it (the package entry point is lib/index.js):

cd C:/projects/dsh-harness/dsh-multimodal
pnpm install
pnpm run build

Then use the command that matches how you run DSH:

# DSH run through npx (no global `dsh` command)
npx @deepseek-ai/dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal

# DSH run from a deepseek-harness source checkout (run from that repo root)
pnpm dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal

# DSH installed with a global `dsh` command
dsh plugin --profile web add file:C:/projects/dsh-harness/dsh-multimodal

The package also has a prepare lifecycle script so Git-hosted installs build before use. A local file: checkout should still be built explicitly before add, which makes the required runtime artifact easy to verify.

Then restart the Web profile with the same launcher (npx @deepseek-ai/dsh web, pnpm dsh web, or dsh web) and open a new session (tool sets are fixed per session). The in-package cordis.patch.yml is the deployment config source (in-package config overrides outer layers; settings.yaml and user profile patches only fill undeclared keys).

⚠️ Applying config changes (important): the web profile uses nodeLinker: hoisted, so a file: plugin is copied into the profile (not symlinked). After editing cordis.patch.yml you must resync:

1. Delete <DSH_HOME>\profiles\web\node_modules\dsh-plugin-multimodal (the snapshot dir)
2. Run <your DSH launcher> plugin --profile web install
3. Restart the Web profile with the same launcher

Re-running dsh plugin add alone will not recopy existing content; deleting the snapshot first is what makes install reliable.

Usage

1. Configure external API models (required)

Fill in apiBase, apiKey, and model in the providers list of the in-package cordis.patch.yml (none of these connection values has a working default). The plugin and DSH Web still start while they are empty; validation happens when the corresponding tool is called. The protocol is OpenAI-compatible, so any vendor works:

providers:
  # Image recognition: multimodal chat/completions
  - id: my-vision
    kind: vision
    apiBase: https://api.openai.com/v1        # or https://api.siliconflow.cn/v1 etc.
    apiKey: sk-xxxx                            # your API key
    model: gpt-4o                              # or Qwen/Qwen2.5-VL-72B-Instruct etc.
    # maxTokens: 1024      # optional
    # temperature: 0.2     # optional

  # Image generation: ONE provider covers text-to-image AND image-to-image
  # (Volcengine Ark Doubao Seedream uses this protocol too)
  - id: my-image
    kind: image
    apiBase: https://api.openai.com/v1        # or https://ark.cn-beijing.volces.com/api/v3
    apiKey: sk-xxxx
    model: gpt-image-1                         # or doubao-seedream-5-0-pro-260628 etc.
    # i2iModel: doubao-seedream-3-0-i2i-250528 # optional: i2i model used when an
    #                                          #   image arg is passed; omit when the
    #                                          #   model handles both (e.g. Seedream 5.0 Pro)
    # watermark: false      # default false (watermark-free), no need to set
    # size: 1024x1024       # optional default size

  # Image generation: Aliyun DashScope native protocol (use when the
  # compatible-mode gateway exposes no image routes; tested: wanx2.1-t2i-turbo
  # is watermark-free). Aliyun t2i/i2i are separate models → i2iModel.
  - id: my-image-wanx
    kind: image
    protocol: dashscope-native
    apiBase: https://dashscope.aliyuncs.com
    apiKey: sk-xxxx
    model: wanx2.1-t2i-turbo
    i2iModel: wanx-v1                          # i2i model (input image must be a public http(s) URL)
    size: '1024*1024'      # DashScope sizes use an asterisk

id must be unique; the same endpoint may host several models under different ids; vision and image generation may point at different vendors. image_generate does not distinguish t2i/i2i: passing an image argument switches to i2iModel (or the same model when unset); omitting it means text-to-image.

2. Verify the plugin is active

Open a new session and ask: "Do you have image_recognize / image_generate tools?" — or have the model run vision_providers to list the configured models.

3. What the model can do

• Recognize — "识别这张图 C:\path\to\photo.png 里有什么" → image_recognize
• Text-to-image — "画一只戴帽子的橘猫，保存到 output 目录" → image_generate returns the saved file path
• Image-to-image — "把这张图 https://.../a.png 里的猫变成蓝色（用 my-image-wanx）" → image_generate with image + provider
• Inspect — "你现在能用哪些图像模型？" → vision_providers

When nothing is configured

DSH Web and the plugin remain available. Only the invoked tool fails, e.g.:

multimodal: provider "my-vision" 未配置 apiBase。请在插件包内 cordis.patch.yml 的 providers 中填写 OpenAI 兼容端点，然后重启 Web profile。

Configuration

Key	Default	Meaning
providers	[] at schema level; the bundled patch contains two empty editable examples	External API models. Each entry: id (unique), kind (vision / image), apiBase, apiKey, and model (required when calling the provider), protocol (image generation only: openai = images/generations (default); dashscope-native = Aliyun DashScope async task API), optional i2iModel / watermark (default false) / size / maxTokens / temperature
outputDir	empty (= <caller workspace>/generated)	Default directory for generated images; relative paths resolve against the caller workspace, absolute paths are used as-is
staticPort	3081	Built-in static image server port (127.0.0.1); generated images are exposed at http://127.0.0.1:<port>/<file> for the Web GUI to render via markdown. 0 disables it (paths only); port conflicts degrade gracefully

image_recognize args: image (required: path / URL / data URI), prompt, provider, max_tokens, temperature. image_generate args: prompt (required), image (optional: image-to-image source, path / URL / data URI), provider, size, n (1–4), output_path (directory ends with / or \).

Development

npm install --cache ./.npm-cache   # build & test deps
npm run build                      # tsc → lib/
npm test                           # unit tests (stubbed API, no network)
node scripts/verify.mjs            # mount check: loads, tools register, error contract

Peer dependencies (@deepseek-ai/cordis, @deepseek-ai/dsh-tools, and @deepseek-ai/schemastery) are also development dependencies for local builds and tests. An installed plugin consumes the compatible peer versions supplied by its DSH profile/runtime.

Known limitations

• 25 MiB inline cap for local images — compress or use a URL for larger files
• Generated-image format detection — PNG/JPEG/WebP/GIF base64 payloads use their detected extension; unknown base64 formats fall back to PNG, while downloaded URL results use the response Content-Type
• DashScope image-to-image needs a public image URL — base_image_url rejects local paths and data URIs (tested). Upload the source image to a public location (Alibaba Cloud OSS / any image host) first and pass the URL. Some models (e.g. wanx2.1-imageedit) are pickier about URLs — wanx-v1 is the most reliable for i2i (tested)
• Old sessions don't see new tools — open a new session after install/upgrade
• Non-standard compatible endpoints — the plugin retries once without response_format; if the endpoint still errors, the remote message is surfaced

License

MIT