dsh-prompt-regression

Wording is behavior — so make prompt wording a tested contract.

A DeepSeek Harness community plugin (skill) that snapshots the deterministic surface of your prompts — SKILL.md files, system prompts, agent presets — and diffs them against a recorded baseline, turning prompt edits into reviewable, gated changes.

DeepSeek Harness's own dsh-prose-standard establishes that prompts and model-visible strings are behavior. This skill is the missing mechanical layer: the same test-driven discipline your code gets, applied to your wording.

What it catches

Drift class	Why it matters
Frontmatter changes (name, description, invocation flags)	Reroutes skill invocation — a widened description silently changes when agents load the skill
Fenced code blocks	The commands the model will actually run changed
Heading outline	Breaks inbound anchors other skills and docs link to
Truth-source links	The wiring to authority files changed
Per-section prose hashes	Rewritten guidance shows up per section — signal without verbatim-diff noise

DSH-native convention checks

Snapshot diffing sees changes; it cannot see whether your skill tree conforms to the harness's own conventions. The second script, scripts/dsh_audit.py, ports two of those conventions into mechanical checks — every rule traced to its committed source in the harness repo, so behavior matches the official specifications instead of paraphrasing them.

1. Sources-of-truth wiring (truth-sources)

Harness skills wire themselves to owning documents via relative links in Sources of truth (read, don't re-summarize) sections. When an owner moves or gets deleted, the skill silently strands. This check resolves every relative link in every SKILL.md against the skill's directory, exempts inline-code templates (documentation about link syntax), and fails loudly:

$ python3 scripts/dsh_audit.py truth-sources .agents/
truth-sources: 1 broken link(s)
  skills/dsh-translate-docs/SKILL.md:66 [sources-of-truth] broken link: 'deleted-owner.md' does not exist

Broken wiring exits 1 — the harness's own doc-site-sync rule: fail, never silently absorb.

2. CoT-leakage probes (cot-leakage)

A faithful port of dsh-trim-cot-leakage's recall batteries: every probe pattern with its taxonomy class (the 8-class system from dead design-session citations to authoring-language slips), the official exclusion rules (vendor/, archived notes, the calibration skill's own directory), and the known false-positive families shipped as inline hints:

$ python3 scripts/dsh_audit.py cot-leakage .agents/skills
cot-leakage: 18 probe hit(s) — every hit needs semantic judgment
  class 3 — change narration and version stamps (12 hit(s))
    dsh-find-simplifications/SKILL.md:95  'An added-then-removed feature...'  ← 'no longer'
  class 4 — review choreography (5 hit(s))
    dsh-prose-standard/SKILL.md:81  'After the user decides...'  ← 'reviewer'

The batteries over-match by design — that is the official philosophy, kept intact: exit 0 with a class-grouped report, judgment stays with you (--strict flips any hit to exit 1). The two judgment calls that are safe to mechanize are mechanized: Chinese-language files skip the authoring-language-slip probes, and same-document anchors are verified against the heading set.

Verified against the harness itself

The checks run on the official repo as ground truth: all 11 skills' link wiring passes clean; the probe report on the skills tree surfaces 18 hits, every one of them a legitimate over-match of the kind the official batteries also produce (skills discussing review workflow naturally contain probe words); and injected fixtures covering all 8 taxonomy classes plus the 4 known false-positive families are caught and hinted respectively — 32/32 tests green.

Install

Drop the skill into your harness skills directory:

# project-level
cp -r . .agents/skills/dsh-prompt-regression
# or user-level
cp -r . ~/.dsh/skills/dsh-prompt-regression

Requires Python 3.9+ (stdlib only, zero dependencies).

Usage

# 1. record the baseline (once)
python3 scripts/prompt_regression.py snapshot skills/

# 2. edit a SKILL.md, then check what drifted
python3 scripts/prompt_regression.py diff skills/
# exit 0: clean   exit 1: drift report

# 3. classify every drift (the skill guides this), then accept
python3 scripts/prompt_regression.py snapshot skills/

# DSH-native convention checks (see above)
python3 scripts/dsh_audit.py truth-sources .agents/   # broken wiring → exit 1
python3 scripts/dsh_audit.py cot-leakage .agents/     # probe report, exit 0

Sample drift report:

CHANGED  skills/dsh-foo/SKILL.md
  frontmatter:
    -  "description": "Use when reviewing a pull request"
    +  "description": "Use when reviewing a pull request or preparing one"
  code block added   [bash]: gh stack merge <stack-number> --yes --merge
  prose changed under heading: Workflow

2 file(s) drifted. Review the diff above, then run `snapshot --name baseline.json` to accept.

CI gate

- run: python3 scripts/prompt_regression.py diff skills/ --snapshot-dir .prompt-snapshots

Exit 1 fails the job. Accepting a change = committing a new baseline in the same PR as the source edit.

How it works

The script extracts, per matched file:

• frontmatter routing fields
• heading outline
• fenced code blocks (language + full body)
• link targets
• per-section prose hashes (prose grouped under each heading, fenced blocks and lists excluded — they're captured separately)

Everything is stored as deterministic JSON. No model calls, no command execution — extraction and comparison only. Semantic judgment about what a drift means stays with the agent, guided by the skill's classification workflow.

Why not just diff the file?

A raw git diff on prose buries the three fields that actually matter (routing, commands, anchors) inside paragraph rewrites. This tool separates the deterministic surface (diffed exactly) from prose (hashed per section), so a pure rewording reads as one line while a changed command reads as a changed command.

License

MIT