dsh-vision-patch

A plugin with two halves on the llm-pi-ai settings namespace. The client card on the Models page declares which models accept image input, per model and per route, and picks a per-route vision fallback model. The host half uses that fallback to describe images so a turn never dies on an image the main model cannot read.

The card renders inside every provider card the Models page draws for a llm-pi-ai route (saved rows, first-run setup rows, add-provider drafts) and writes through the settings Remote. The host half adds no HTTP route of its own; it registers a single agent/pre-step interceptor.

What it does

• Per-model checkbox — input: ['text', 'image'] on the model's stored entry. Unchecked removes the input key entirely, so the model inherits the installed catalog's capability (it never writes ['text'], which would forbid images outright).
• Route-wide checkbox — defaultInput: ['text', 'image'] (set/unset via path ops) for every model on the route that declares no list of its own.
• Vision fallback model — a per-route dropdown of the route's vision-capable models (those that declare images themselves or inherit them from the route). The picked model is stored as visionFallbackModel.
• Model filter — a search box appears once a route lists more than 8 models; the count line shows how many models currently declare images.
• Live refresh — the card re-reads after each write and follows settings/document-updated, so edits made elsewhere on the page converge.

A provider with no stored profile (never saved from the Models page) shows guidance text instead of checkboxes.

Vision fallback (host)

A provider route in the llm-pi-ai settings namespace may carry a visionFallbackModel field: { provider?: string; model: string }. The provider defaults to the route itself; model names a vision-capable model on it. An absent field turns the feature off for that route. The Models card's dropdown writes this field; the host reads the same stored profile, so the two sides always agree.

When the host's agent/pre-step interceptor sees a step whose user message contains an image content block AND the step's route (from agent.options.provider) has a configured visionFallbackModel, it:

1. streams the turn to the fallback vision model via ctx.llm.stream({ provider, model, messages, signal }) to describe the image(s);
2. replaces each image block in the user message with a text block holding that description; and
3. returns the rewritten messages, so the main model continues on the description.

It does not inspect the active main model's modality: it describes whenever a fallback is configured and an image is present. If the describe fails, times out, no fallback is configured, or ctx.llm is unavailable, the interceptor delegates unchanged (next()), so the image falls through to the harness's default handling and the turn is never blocked.

Durability & logging

When the interceptor describes an image it rewrites the user's own message, replacing each image content block with a text block holding the description. That rewritten text is not stamped with a plugin source kind. The whole host seam is validated only against mocks — there is no live host to confirm a new, non-catalog source kind is accepted — and rewriting an already-present user message (rather than appending a fresh one) risks clobbering the harness's own source marker on that message. Durability of the injected wording is therefore the harness's call; this plugin makes the injection observable instead, via the host logger:

• info when a describe is attempted (an image is present, a fallback model is configured, and ctx.llm.stream is available);
• debug when the describe yields no usable text and the interceptor delegates unchanged (next());
• debug when a description is actually injected and the turn enters rewritten;
• debug when a describe is skipped because one is already in flight for the same agent (the per-agent concurrency guard).

All logging is guarded (ctx?.logger?.info?.(...) / debug / warn?.(...)), so an absent logger is harmless. A silent describe is never silent in the logs: if the user believes an image was described but no text assembled (for example the real ctx.llm.stream chunk shape differs from the mock), the debug line names the agent that fell back to next() unchanged.

Install

dsh plugin --profile web add github:ryasrk/dsh-vision-patch

The profile's dsh.profile.bundles list picks the package up automatically (its package.json declares dsh.bundle.patch). Restart the web host so the new bundle layer composes, then reload the Models page.

Layout

Path	Role
cordis.patch.yml	inserts the dsh-vision-patch loader row
lib/index.js	host entry — apply(ctx) registers the agent/pre-step interceptor via lib/host.js
lib/host.js	host helpers: the agent/pre-step handler that describes images and rewrites the turn, plus the settings fallback reader
client/client.js	browser bundle: the card (image-input checkboxes + vision fallback dropdown), registered on the settings.models.provider-card slot keyed llm-pi-ai
package.json	dsh.bundle.patch + dsh.client (web platform, inject edges)

The client bundle requires only react (a platform seed word); it needs no build step and has no runtime dependencies.

How it writes

All writes go through ctx.remote.settings against the llm-pi-ai namespace, carrying the revision the last read observed (settings-conflict on a lost race):

• route-wide toggle → mutate path ops on ['providers', <route>, 'defaultInput']
• vision fallback → mutate path ops on ['providers', <route>, 'visionFallbackModel'] (set the { model } choice, unset to clear)
• per-model → update merge restating the whole stored models list (path ops cannot address array elements), every untouched field merge-preserved.

Writes that land return the namespace's post-write view, which folds straight back into the card.

Uninstall

dsh plugin --profile web remove dsh-vision-patch

(Then restart the web host so the bundle layer leaves the composition.)