dsh-bench

English · 中文

Run one task across several models, tool surfaces, or context configurations, and get back a controlled-experiment table: which arm actually passed, what each cost, and where a slower arm's time went. When the experiment's design or its result is unsound, the report says so to your agent directly — a task every arm solves along the same trajectory measures typing speed, not capability.

Built for DeepSeek Harness.

The plugin registers three tools: bench_run to measure, bench_history to compare against past runs, and bench_gc to reclaim the session logs the trials leave behind. Each trial is a fresh one-shot spawn subagent that sees only the task text — no parent history — so the arms are comparable.

Why

The plugin ecosystem has plenty of tools that observe a single session — usage dashboards, context breakdowns, trajectory diagnostics — and none that compare two. Without a comparison there is no way to answer whether a cheaper model, a smaller tool surface, or a shorter output budget actually changes the outcome; there are only opinions about it.

Every number here already exists in the harness. The plugin drives the matrix and folds the results:

what	where it comes from
steps, model wall time	sessionStats session-projection unit
tokens	ctx.tokenMeter.measure(session)
tool calls and mix	tool/call events in the child's durable log
stop reason	SubagentResult.stopReason

Mount

Nothing in $DSH_HOME needs editing — the plugin ships its own overlay. Copy the template and point it at wherever you cloned this repo:

cp bench.patch.yml.example bench.patch.yml
# then edit the `name:` line to your own absolute path
dsh --profile web --patch /path/to/dsh-bench/bench.patch.yml

Verify the composition first — this boots nothing and costs nothing:

dsh --profile web --patch /path/to/dsh-bench/bench.patch.yml --dump-config

Path gotcha. A relative name: inside a --patch overlay resolves against the profile directory ($DSH_HOME/profiles/<name>), not against the patch file — unlike an agent preset, where it resolves against the preset directory. A plugin living outside the profile must therefore use an absolute specifier, and on Windows Node's ESM loader accepts an absolute path only as a file:// URL.

The plugin is loaded from outside the profile, so it sits outside the HMR watch root: changes need a restart, not a reload.

Use

bench_run defaults to a dry run and returns the plan without sending anything. Pass confirm: true to spend tokens.

{
  "task": "Read package.json and list every script.",
  "variants": [
    { "label": "pro" },                                       // inherits the caller's route
    { "label": "flash", "model": "deepseek-v4-flash" },
    { "label": "no-shell", "model": "deepseek-v4-flash", "deny": ["bash", "pwsh"] }
  ],
  "repeat": 3,
  "check": ["node", "verify-scripts.mjs"],
  "confirm": true
}

Ran 9/9 trials.

**Outcome**

| variant  | provider/model             | runs | pass | completed | outcome |
| ---      | ---                        | ---: | ---: | ---:      | ---     |
| pro      | deepseek/deepseek-v4-pro   | 3    | 3/3  | 3/3       | ok      |
| flash    | deepseek/deepseek-v4-flash | 3    | 3/3  | 3/3       | ok      |
| no-shell | deepseek/deepseek-v4-flash | 3    | 1/3  | 3/3       | ok      |

**Cost & latency** — medians across runs

| variant  | steps | tool calls | tokens | llm ms | ttft ms | tok/s | tool ms | tool mix /run |
| ---      | ---:  | ---:       | ---:   | ---:   | ---:    | ---:  | ---:    | ---           |
| pro      | 7     | 8          | 24448  | 8124   | 620     | 96    | 1180    | glob×3, read×2, grep×1 |
| flash    | 4     | 4          | 11047  | 3011   | 450     | 198   | 640     | read×3, glob×1 |
| no-shell | 6     | 7          | 15220  | 5340   | 470     | 191   | 910     | read×5, glob×2 |

There is no UI: bench_run is a model-facing tool, so you invoke it by asking the agent for it in the composer.

Why an arm is slow, not just that it is

llm ms alone cannot tell waiting from generating, and the two have opposite fixes. It is therefore split using the harness's own figures:

• ttft ms — mean first-token latency per step. A gap here at comparable token volume is provider queueing: a different gateway or a different hour may erase it.
• tok/s — decode throughput. A gap here is mostly the model — but shared batching degrades throughput under load too, so read it as an upper bound rather than a constant (two runs an hour apart moved one arm's throughput -17% and another's +19%).
• tool ms — wall time inside tools, outside the model entirely.

A measured 0 renders as —: no real trial costs zero tokens or waits zero milliseconds, so a literal zero means "not measured", never "instant".

The report then attributes each slower arm's extra time against the fastest one, because "2.6× the first-token latency and a third less throughput" does not say which half to act on:

**Where the extra time went**

- **mimo** spent 11427ms more than `flash`: 48% waiting (+5432ms), 52% generating (+5995ms)

Roughly half of that gap may disappear on another gateway or at another hour, and the rest never will. The two components do not sum to the raw llm ms delta — framing and per-step overhead live outside both — so the split is a share of what was actually attributed.

"Generating" is a verdict on the model only when both arms wrote about the same amount. When they did not, the report splits it again, pricing the volume delta at the fastest arm's own rate:

- **deny-shell** spent 25361ms more than `full`: 20% waiting (+5093ms), 80% generating (+20268ms)
  - the generating share splits again: +23281ms from output volume (10.5x the decode tokens,
    280 → 2932), -3013ms from throughput — mostly WHAT IT WROTE, not how fast it writes;
    the lever is the trajectory or the tool surface, not the model

Without that second line, an arm that generates faster (129 vs 114 tok/s) and simply writes ten times as much reads as "80% generating" — which invites blaming the model for what the tool surface did.

History

Every confirmed run is recorded through the storage seam ($DSH_HOME/storages/bench_runs.json under the json backend), and bench_history reads it back:

{ "limit": 10, "task": "package.json" }   // both optional

It lists past runs newest-first and, for any task run more than once, reports how each arm drifted:

**Drift since the previous run of the same task**

- `mimo` on "find the bigger package.json" — ttft -58%, tok/s = (08-16 23:40 → 08-17 09:15)

That line is the whole reason the ledger exists. One measurement cannot separate a slow endpoint from a busy minute — two can. Throughput holding while latency moves is provider queueing; both moving together is load reaching the decode path, or a route change. And when two arms drift in OPPOSITE directions in the same window, the cause is not your network — it is one endpoint's own load.

A dry run records nothing. Without the storage seam the run still completes and says it was not recorded.

Reclaiming what the trials left behind

Every trial is a real session with a durable log, and nothing in the harness expires them. One matrix of 2 arms × repeat 3 leaves six directories under $DSH_HOME/sessions/<workspace>/, permanently. bench_gc reclaims them:

{}                                  // report only: how many exist, how much space
{ "confirm": true }                 // MOVE them to $DSH_HOME/bench-trash/
{ "purge": true, "confirm": true }  // empty the trash — irreversible

Targets come from the ledger, never from the filesystem. A bench child's session header is {"origin": "subagent", "delegationDepth": 2, "agentPreset": "standard"} — indistinguishable from any other subagent you started, with no label to match on. So each run now records its trial session ids, and bench_gc touches only directories this plugin can prove it created. Runs recorded before that are reported as unreclaimable rather than silently omitted from the count.

Reclaiming moves; it does not delete. purge is the separate irreversible step. The trash is a sibling of the sessions root, not a child of it: the persistence backend reads every directory under its root as a workspace slug, and a trash folder inside would become one.

Two things it deliberately leaves alone: the session projection cache ($DSH_HOME/storages/session_projcache.json) keeps entries for reclaimed ids — it belongs to another plugin and its schema is not this one's to edit — and the ledger rows themselves, since bench_history is the record of what was measured, not of what is still on disk.

The ablation axis

Each variant takes suppress: a list of agent/pre-step context sources stripped from that arm's trials. It answers the question every harness user decides blind — is this injected section worth its tokens?

"variants": [
  { "label": "full" },
  { "label": "no-skills", "suppress": ["skill-catalog"] },
  { "label": "no-agents", "suppress": ["agent-instructions"] },
  { "label": "bare",      "suppress": ["skill-catalog", "agent-instructions"] }
]

**Context ablation**

- `full` — full context
- `no-skills` — suppressed skill-catalog: 1 message(s) stripped per run
- `bare` — suppressed skill-catalog, agent-instructions: 2 message(s) stripped per run

The common kinds are skill-catalog (the available-skills reminder) and agent-instructions (the AGENTS.md / CLAUDE.md digest); a composition may inject others.

Two properties matter more than the feature itself:

• A suppression that matches nothing is loud. Ablation's failure mode is silent — no error, just an arm that quietly ran with full context and looked like evidence. When a requested kind strips zero messages, the report says so and names the kinds it actually saw, so a typo cannot masquerade as a null result.
• Only this run's children are touched. The filter matches on the child's durable parentSession, not on a global flag, so a bench run never strips context from a concurrent session in another workspace.

Pair it with check, or the numbers only show that less context is cheaper — which was never in doubt.

When the task itself is the problem

If every arm walks the same trajectory — same steps, same tool calls — the report says so outright. That table is easy to misread as "these models are equivalent", when it usually means the task left no room for them to differ: spell out a file path and every model degenerates to one read, and the run measures nothing but typing speed. Ambiguity is what makes strategies visible; a task with none has no discriminating power, whatever its table looks like.

Cost is not quality

completed means the child finished its turn — not that it did the task correctly. The check command is the only quality signal here: an argv (never shell-interpreted; pass your own shell if you want one) run in the workspace after each trial, where exit 0 is a pass. It receives the trial's final text in $DSH_BENCH_OUTPUT, plus $DSH_BENCH_LABEL, $DSH_BENCH_RUN, and $DSH_BENCH_STOP_REASON.

Without it a run measures cost only, and the report says so. An arm that is three times cheaper while failing its check is not the cheaper arm — it is the wrong one.

Tool surface as an axis

Each variant takes allow (keep only these) and deny (remove these), so "does this model still succeed without a shell?" is a row rather than a guess.

Scoping is applied by filtering the assembled catalog, with a guard denying execution as a backstop. It is deliberately not SubagentStartRequest.toolFilter: that becomes a tools.restrict(), which masks global tools only and fails on scope-local names — and in a stock profile every interesting tool (bash, pwsh, read, glob, todo_write, web_search, ask_user_question) is registered by the agent preset, not globally. A restrict-based filter therefore cannot touch them.

A few tools are denied in every trial by default (denyTools):

• ask_user_question — a one-shot bench child has no human behind it, so a question can only stall the trial and inflate its wall time; an arm that stopped to ask never attempted the task, yet its row would sit in the same columns as an arm that finished one.
• bench_run, bench_history, bench_gc — a trial is an ordinary agent and sees the whole catalog, including these. A child that starts its own matrix spends real money outside maxTrials, which caps one call and not a tree of them; and a trial that reclaims sessions would delete the log of the run measuring it.

This is a guard rail, not a prohibition. Comparing how well two models drive a benchmark is a real experiment, so naming a tool in an arm's allow lifts it out of the default deny — a stated intent beats a safe default, an unstated one does not. Allowing and denying the same name in one arm is rejected outright rather than silently resolved.

A name that was never offered to a trial is reported per name, with the full offered catalog beside it — the check that catches deny: ["bash"] on a machine whose shell tool is pwsh.

The axis is worth an arm because removing one tool moves the whole trajectory, not just one call. On "count the lines in every .py file under this directory", deny: ["pwsh"] took the arm off a single shell command and onto read×4.7, glob×2.3, subagent, workflow: 2 → 5 steps, 8425 → 16310 tokens, ~6s → ~46s. Both arms still passed 3/3 — the answer was reachable without a shell, at six times the cost.

Aggregates are medians and counts, never means. Trajectory outcomes are closer to Bernoulli than Gaussian, and a mean over a bimodal sample reports a value no run produced. Below repeat: 3 the report warns that the median columns are a single sample rather than a trend.

Config

key	default	meaning
providerName	spawn	Subagent provider. fork would inherit the caller's conversation and destroy comparability.
maxTrials	24	Hard cap on variants × repeat per call.
trialTimeoutMs	300000	Per-trial deadline. Without one a single wedged route holds the whole matrix open — a broken provider once spent five of a six-minute run on retry backoff alone.
checkTimeoutMs	60000	Per-check deadline.
denyTools	["ask_user_question", "bench_run", "bench_history", "bench_gc"]	Denied in every trial regardless of variant. An arm's allow lifts a name out of this list.
sessionsRoot	$DSH_HOME/sessions	Where bench_gc looks for trial sessions. Only needed if the persistence backend was given a non-default root; nothing reports it back at runtime.

Deliberate limits

• Sequential trials. Concurrency would pollute llmMs with queueing delay and make the wall-time columns incomparable.
• No preset axis. SubagentStartRequest carries no preset override — children join their parent's preset — so that axis needs ctx.agents.create() instead of the subagent seam.
• First-trial failures are fatal. A bad provider name, model route, or depth budget fails the whole call rather than burning the matrix against the same fault. Later failures are recorded as notes and the run continues.
• A timed-out trial still reports its cost. The tokens were spent; only completed and outcome mark it as unfinished.

Status

P0, verified against a live 4-trial run (2 arms × repeat 2) on 2026-08-15.

What that run established:

• Composition, module load, tool registration, JSON-Schema acceptance at prompt assembly, output validation, and report rendering all work.
• No approval deadlock. The children executed 12 tool calls (glob, read, grep) with no gate stalling the parent — the delegated policy carries down as child-agent.ts implies.
• One real bug, since fixed: Cordis gates service access, not merely plugin activation. Reading ctx.tokenMeter without declaring the dependency throws cannot get property "tokenMeter" without inject, so a defensive try/catch degraded every metric column to 0. Optional seams now arrive through ctx.inject(…); a composition lacking them still reports tool mix and completion counts, with a note naming what is missing.
• Retry backoff dominates wall time when a route is broken. A failing arm cost roughly 5 of the run's 6 minutes: 2 repeats × 2 llm/retry attempts each, with backoff between them. A per-trial timeout belongs in P1.

That run also produced the tool's first genuine finding, which is the whole point: one arm scored 0/2 completed with zero tool calls, its children ending in turn/end → error, "Stream ended without finish_reason" (TRANSPORT). A broken provider route is invisible in normal chat — it merely feels slow — but next to a working arm it is unmissable.

P1 added what the second live run showed was missing:

• Question tools denied by default. Both arms of that run called ask_user_question — with no human behind a one-shot child, that is pure wall-time contamination, and it made the 5.8× latency gap unattributable. (The first implementation went through toolFilter and, as a later run proved, never denied anything; see the ablation section.)
• check command. That run could only conclude "flash is cheaper", never "flash is better", because nothing verified either arm's answer.
• Per-trial timeout, after the earlier six-minute run spent five minutes on one broken route's retry backoff.
• Reporting fixes. The tool-mix column was a run total sitting beside a per-run tool-call median — two units in one table. It is now per-run. Below repeat: 3 the report also states that its "median" is a single sample.

A later run then compared two models on a task with real location work. Both passed 3/3 with identical steps and tool calls and token counts within 6%, differing only in wall time — which raised the question llmMs could not answer, and produced the ttft/throughput split and the discriminating-power warning above. It also retired an earlier hypothesis: one arm's heavy shell exploration in the very first run never reproduced, and was a single unreplicated sample rather than a strategy.

P2 added the ablation axis, and its first live run immediately exposed two failures that were silent by construction:

• Tool scoping had never worked. SubagentStartRequest.toolFilter becomes a tools.restrict(), which masks global tools only, while a stock profile registers every interesting tool through the agent preset. The run reported ask_user_question is not registered while children had been calling it all along — and the earlier reading that P1's deny "worked" was wrong: the tool had stopped appearing for an unrelated reason. Scoping now filters the assembled catalog, with a guard denying execution behind it.
• Attribution broke on opposite signs. It divided by the sum of two components that can disagree, printing 292% waiting (+-852ms) for an arm that waited longer while generating faster. Opposite signs now report signed durations and no split.

That run also showed why the "matched nothing" diagnostics matter more than either axis: skill-catalog is not a source kind this composition injects, so an arm labelled no-skills had quietly run with full context — and without the warning its row would have read as evidence that skills do not matter. bench_run and bench_history were denied to trials in the same pass, after the offered-tool list revealed a trial could start its own matrix.

The first run to make tool scoping change behaviour then broke the attribution one more way. A deny: ["pwsh"] arm took six times the wall time and the report called it "80% generating" — but that arm generated at a higher tok/s and merely wrote ten times as much, so the label pointed at the model when the cause was the tool surface. The generating share now splits again into volume and throughput whenever the two arms' output lengths differ by more than 15%.

What is verified, and how

Every feature here was written against a live run, but not every path has fired in one. Failure paths are covered by the mock suite only, which is stated rather than glossed:

capability	status
subagent trials, metric folding, report rendering	live
check pass/fail, ledger write, bench_history drift	live
ttft / throughput / tool-time split, attribution	live, including the opposite-sign case and the volume-vs-throughput sub-split
context ablation (suppress)	live — 3 messages stripped per run
"matched nothing" diagnostics for both axes	live — caught a skill-catalog kind this composition never injects, and a deny: ["bash"] on a machine whose shell is pwsh
tool scoping (allow / deny)	live, and shown to change behaviour — on a "count the lines in every .py file" task, deny: ["pwsh"] pushed the arm off pwsh×1.3 onto read×4.7, glob×2.3, subagent, workflow: 2 → 5 steps, 8425 → 16310 tokens, ~6s → ~46s. Both arms still passed 3/3, so the finding is reachable without a shell, at 6x the cost
bench_gc reclaim / purge / already-gone / pre-id ledger rows	fixture — a real sessions tree on disk, driven end to end, asserting an unrelated session survives; not yet run against $DSH_HOME
per-trial timeout, check failure, missing seams, provider without the capability	mock only — no live run has hit them

Roadmap

• P3 — trajectory diff, a preset axis via ctx.agents.create(), and concurrency.

License

MIT