dsh-maestro-core

Supervisor for DSH Web resilience — Phase 1 Guard & Report + Phase 3 Auto-Resume & Auto-Reload — together with the settings store, the Maestro settings card, the tool guard and the harness-to-harness sync engine that used to be four separate packages.

Runs outside the pnpm → sh → node tree (systemd daemon) to survive tree crashes, plus inside dsh web as a host+client Cordis plugin to auto-resume interrupted sessions and auto-reload the browser after restart.

• Daemon: Polls :3080 every 3s, keeps last-known-good (LKG) snapshots (~/.dsh/.supervisor/lkg/, rotate 3, sha256 verify, df >500MB guard), auto-rollbacks on crash (debounce 60s, flock lock), writes report-<ts>.md (health + git diff + log tail), and notifies via Telegram (loose, never blocks).
• Host plugin: runAutoResume() 8s after boot — findInterrupted (tail 100) + findDanglingOpenTurns (full scan for recent sessions) within autoResumeWithin (default 5m) → agents.resume({resumeSessionId, agentOptions: {provider,model}}) recovered from request/context → followup('continue'). Loopback RPC POST /dsh-maestro-supervisor-resume/{scan,resume} (reachable from loopback; connection.isLoopback reports it) for the daemon (resumeViaRpc).
• Client plugin: Hybrid auto-reload — fetch HEAD / polling 1s on offline/WebSocket close/visibilitychange → 200 → location.reload(). Served as window.__ModuleLoader__.load bundle at /plugins/@ddtcorex/dsh-maestro-core/client.js via dsh.client.

Modules

One package, four host rows and one client bundle.

Module	Source	Row id	Channel	What it does
Supervisor	src/host/*.ts	maestro-supervisor	/dsh-maestro-supervisor-resume (loopback-reachable)	Crash polling, LKG snapshots, auto-resume, auto-reload
Store	src/host/store/	—	—	The namespaced settings.json every other module reads and writes
Config	src/host/config/	maestro-config	/dsh-maestro-config	maestroConfig service over the store, backing the Maestro settings card
Guard	src/host/guard/	dsh-maestro-guard	/dsh-maestro-guard	Rule classification, the approval gate and its journal
Sync	src/host/sync/	dsh-maestro-sync	/dsh-maestro-sync	Backup, restore, retention GC and two-machine sync

Row names for the absorbed modules are subpaths of this package (@ddtcorex/dsh-maestro-core/lib/<module>/index.js), which is why exports["./lib/*"] exists: a row name the exports map cannot resolve is skipped silently at boot.

The store is also published on its own (@ddtcorex/dsh-maestro-core/store) and can be vendored into a consumer with scripts/vendor-store.mjs, which writes one self-verifying file with a body hash.

Registered tools

Host tools registered with ctx.tools.register:

Tool	Module	What it does
dsh_web_restart	supervisor	Schedule a supervised dsh web restart (consent-gated)
dsh_web_restart_status	supervisor	Report the state of the last restart request
dsh_web_dryboot	supervisor	Dry-boot on an ephemeral port with an isolated DSH_HOME
dsh_web_gc	supervisor	Preview, then reap, verified dry-boot orphans
maestro_session_health	supervisor	Session-log health scan (re-encode or quarantine)
maestro_resume_tool_health	supervisor	Tool-view health of a resumed session
maestro_repair_session_preset	supervisor	Re-link an agent that joined no preset
maestro_guard_status, maestro_guard_stats, maestro_full_scan	guard	Guard state, counters and a full rule scan
maestro_sync_status, maestro_sync_check_machines, maestro_sync_preview, maestro_sync_apply, maestro_sync_pull, maestro_sync_push, maestro_sync_bidirectional_preview, maestro_sync_bidirectional_apply, maestro_sync_tunnel_restore	sync	Two-machine sync, preview first
maestro_backup_preview, maestro_backup_apply, maestro_backup_gc_preview, maestro_backup_gc_apply, maestro_restore_preview, maestro_restore_apply	sync	Backup, retention GC and restore, preview first

Install

One package, one command:

dsh plugin --profile web add @ddtcorex/dsh-maestro-core

In a workspace checkout, build and link instead:

pnpm --dir packages/dsh-maestro-core install
pnpm --dir packages/dsh-maestro-core build   # tsc host + tsc client + esbuild bundle -> lib/ + lib/client.js
pnpm --dir packages/dsh-maestro-core verify  # tsc --noEmit host + client
pnpm --dir packages/dsh-maestro-core test    # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
# link:() the checkout into ~/.dsh/profiles/web/package.json, then
pnpm --dir ~/.dsh/profiles/web install

pnpm build is required after any src/ change; lib/ is gitignored build output. The client needs both tsc steps and the build-client.mjs bundle — plain tsc alone leaves lib/client.js as a bare ES module and dsh web will fail with exports no "./client" bundle.

The package declares dsh.client (platform: web, inject: ["@deepseek-ai/dsh-client-connection", "@deepseek-ai/dsh-client-ui-slots"]) so the browser half is auto-loaded — no extra dsh.client flag needed.

Pre-flight (required): before adding to a live profile's bundles, dry-boot must pass:

DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 &
# wait for "dsh web: http://127.0.0.1:<port>" and curl 200, then kill
# This catches load-time failures (missing lib/index.js, stale build, bad cordis.patch.yml)
# that no in-code try/catch can catch. See dsh-safe-restart skill.

This exact failure class caused dsh web outages on 2026-08-27 (missing lib/index.js). See AGENTS.md Conventions.

Systemd daemon (optional, for crash detection outside the tree)

bash packages/dsh-maestro-core/scripts/install-systemd.sh
systemctl --user daemon-reload
systemctl --user enable --now dsh-web-supervisor
systemctl --user status dsh-web-supervisor
journalctl --user -u dsh-web-supervisor -f

The template leaves Environment=TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID commented — uncomment via systemctl --user edit dsh-web-supervisor if you want Telegram, otherwise it logs only.

To run without systemd (foreground, for debugging):

node packages/dsh-maestro-core/lib/index.js daemon   # poll every 3s
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js resume --within 5m   # list interrupted sessions

Configuration

All autoResumeWithin values are minutes when given as number (e.g. 5 → 5 minutes). Strings support 30s/5m/1h. Precedence (highest first):

1. Cordis config (cordis.patch.yml config: or apply(ctx, config)) — explicit per-install.
2. Env DSH_SUPERVISOR_AUTO_RESUME / DSH_SUPERVISOR_RESUME_WITHIN (bare 5 in env → 5m for ergonomics).
3. Supervisor config ~/.dsh/.supervisor/config.json (autoResumeEnabled, autoResumeWithin).
4. Maestro settings ~/.dsh/dsh-maestro-config/settings.json (domains.supervisor.*).
5. Default: true / 5.

Key	Type	Default	Env	File	Notes
autoResumeEnabled	boolean	true	DSH_SUPERVISOR_AUTO_RESUME (1/true/yes/on/enabled vs 0/false/no/…)	config.json: autoResumeEnabled, settings.json: domains.supervisor.autoResumeEnabled	false → notify only
autoResumeWithin	number (minutes) or string (5m)	5	DSH_SUPERVISOR_RESUME_WITHIN	config.json: autoResumeWithin, settings.json: domains.supervisor.autoResumeWithin	Window for findInterrupted/findDangling (mtime + event.time)

Example ~/.dsh/.supervisor/config.json:

{
  "autoResumeWithin": 5,
  "autoResumeEnabled": true
}

CLI

node packages/dsh-maestro-core/lib/index.js --help
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js daemon   # poll 3s, debounce 60s
node packages/dsh-maestro-core/lib/index.js resume [--within <dur>]   # list interrupted sessions, e.g. 5m, 30s, 1h
node packages/dsh-maestro-core/lib/index.js boot-guard acquire|release --pid <pid>   # boot.lock + boot-boundary, used by the safe-restart script

The CLI has no logs or rollback command: rollback runs inside the daemon, and reports are files under ~/.dsh/.supervisor/reports/.

RPC (reachable from loopback, connection.isLoopback)

# Scan (findInterrupted only, tail 100)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST \
  -H 'content-type: application/json' \
  -d '{"type":"client-request","rpcId":"r1","method":"scan","payload":{"withinMs":300000}}'
# → {"type":"server-response","rpcId":"r1","result":{"ok":true,"value":{"scanned":425,"interrupted":[]}}}

# Resume (re-attaches agent + followup continue, recovers provider/model)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume -X POST \
  -H 'content-type: application/json' \
  -d '{"type":"client-request","rpcId":"r2","method":"resume","payload":{"ids":["--example-project--/session-abc"]}}'
# → {"type":"server-response","rpcId":"r2","result":{"ok":true,"value":{"resumed":["--example-project--/session-abc"]}}}
# or {"ok":false,"error":{"code":"bad-request","message":"resume requires at least one session id"}}

The daemon uses resumeViaRpc() (supervisor.ts) which POSTs the same envelope to http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume with fetch and validates server-response + rpcId + result.ok.

Auto-Resume Details

• When: apply() sets setTimeout 8000 after boot, then runAutoResume() — only safe right after fresh boot when dsh web is sole owner (an open turn found then cannot belong to a still-running generation). resumeInterrupted additionally checks agents.get(sessionId) and skips if already live.
• What: findInterrupted (tail 100, looks for turn/end with reason.kind === 'interrupted' at time within window) + findDanglingOpenTurns (full scan for recent sessions, looks for turn/start without matching turn/end at time within window) → merged = Set([...interrupted, ...dangling]) → resumeInterrupted for each id.
• How: agents.get(sessionId) if live → followup('continue'); else sessionPersistence.load(sessionId) → find request/context with provider/model → agents.resume({resumeSessionId, agentOptions}) → followup('continue'). If load fails, still resumes without agentOptions (degrades). Returns string[] resumed and logs sent continue trigger.
• Subagents: findDangling does full log scan for recent sessions (mtime within window, 1-2 files) — not tail — because subagent b6487e33 had its only turn/start at seq 6 at the very beginning of a 1906-line log, missed by tail -100. findInterrupted stays tail 100 (interrupted closer is always at tail). The mtime pre-filter keeps full scans cheap (previously 5.5s for 425 sessions without it).

Auto-Reload Details (Hybrid)

• Client (src/client/auto-reload.ts, lib/client.js via window.__ModuleLoader__.load): ctx.effect hooks WebSocket (patches window.WebSocket to catch close for same-origin DSH ws), offline/online, visibilitychange → setInterval(fetch HEAD / 1s) when down → 200 → location.reload() (once, reloading guard). Also checks HEAD / on load in case the page was opened while down.
• Host (supervisor.ts pollHealth 3s + notify, plugin.ts runAutoResume): health check + restart + notify is the host half; together with client polling they cover manual, supervisor, and systemd restarts without F5. No extra host push channel needed — client polling is primary, host health is secondary; the window.__ModuleLoader__ bundle is served at /plugins/@ddtcorex/dsh-maestro-core/client.js via ClientModuleRegistry (dsh.client + exports["./client"]).

Verification

# Build & unit
pnpm --dir packages/dsh-maestro-core verify   # host + client
pnpm --dir packages/dsh-maestro-core test     # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
curl -s http://127.0.0.1:3080/plugins/@ddtcorex/dsh-maestro-core/client.js | grep -c "window.location.reload"  # 2

# Live
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3080/  # 200
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST -H 'content-type: application/json' -d '{"type":"client-request","rpcId":"t","method":"scan","payload":{"withinMs":300000}}' | head -c 200
node --input-type=module -e "import {findDanglingOpenTurns} from './packages/dsh-maestro-core/lib/resume.js'; console.log(await findDanglingOpenTurns(undefined,{withinMs:5*60*1000}))"
# Create a real dangling: pnpm --dir deepseek-harness dsh --profile headless "Run bash synchronously sleep 60" & sleep 4; kill $!; node -e "...findDangling..."  # should be 1
# After restart, it should have turn/end interrupted → continue → turn2

Troubleshooting

Symptom	Cause	Fix
Cannot find package '.../dsh-maestro-core/index.js'	pnpm build not run or lib/ stale	pnpm --dir packages/dsh-maestro-core build && pnpm --dir ~/.dsh/profiles/web install
exports no "./client" bundle / client bundle not found	Missing lib/client.js or exports["./client"]	pnpm build (runs tsc + tsc -p tsconfig.client.json + node scripts/build-client.mjs), check package.json exports and dsh.client, test -f lib/client.js, curl .../client.js
EADDRINUSE on a fixed port of dsh web	A previous dsh web still holds the listener tree: :3080 (LAN proxy), :3081 (public proxy), :3082 (raw webserver); :3000 is unbound	`ss -tlnp
uses .jsonl but backend is zstd	Hand-written session.jsonl while backend is zstd	Use zstd -c plain.jsonl > session.jsonl.zstd or JsonlSessionPersistence API. Never hand-write opposite encoding — listArtifacts checks every project dir on boot and one stray file blocks all of dsh web.
first frame is not exactly one header line	zstd without type: session header	Use toHeaderLine + compressZstdFrame(header) + compressZstdFrame(body) as in encodeMaterialization.
findDangling 0 but subagent still open	Tail window too small (before 63b7719)	Fixed: findDangling now full-scans recent sessions (mtime within window). findInterrupted stays tail 100.
resumed: [] or RESUME FAILED	agents.resume failed (no persistence, no provider/model)	Check session.jsonl.zstd exists and zstd -d -c ... | head -n 1 is valid header. request/context with provider/model is recovered — if missing, still resumes without agentOptions.
RESUME SKIPPED	No session within autoResumeWithin window	Increase autoResumeWithin to 10/"10m", check config.json and DSH_SUPERVISOR_RESUME_WITHIN, verify with withinMs: 60*60*1000.
Page does not reload after restart	lib/client.js not served or browser cache	`curl .../client.js
dangling found but agents.get says live	Session still live in current process (not a crash)	findDangling is only safe right after fresh boot when dsh web is sole owner. resumeInterrupted skips if agents.get is live — correct, wait for next boot.

Known Issues

• Manual session.jsonl vs zstd: One stray opposite-encoding file under ~/.dsh/sessions/ blocks the entire workspace listArtifacts on every boot (encodingMismatch). See Troubleshooting.
• Tail vs full scan: Before 63b7719, b6487e33 subagent missed because its only turn/start was at seq 6 at the very beginning of a 1906-line log. Fixed, but if you add a new scan variant, reuse readSessionAllLines with mtime pre-filter.
• Port map: :3080 is the LAN proxy (PIN login), :3081 the public proxy, :3082 the raw dsh web webserver; :3000 is unbound. Inspect with ss -tlnp; never pkill -f "dsh web".
• Client bundling: lib/client.js must be window.__ModuleLoader__.load wrapper via scripts/build-client.mjs, not bare export. Add new client files under src/client/ and ensure tsconfig.client.json includes them, then pnpm build.
• Config precedence: cordis.patch.yml config: > env (DSH_SUPERVISOR_RESUME_WITHIN bare 5 → 5m) > config.json > settings.json > default. See getAutoResumeEnabled() and getResumeWithinMs() in plugin.ts and supervisor.ts.

Development

pnpm --dir packages/dsh-maestro-core verify
pnpm --dir packages/dsh-maestro-core test
pnpm --dir packages/dsh-maestro-core build

For daemon changes: DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 + corrupt settings.json → assert report + rollback.

Telegram

Loose by default: notifier.ts tries import('@ddtcorex/dsh-maestro-notifier'), then TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID env, then console.log. Enable via systemctl --user edit dsh-web-supervisor → uncomment Environment=TELEGRAM_* → daemon-reload + restart.

Hard mode (optional): package.json add "@ddtcorex/dsh-maestro-notifier": "workspace:^0.1.0" + pnpm-workspace.yaml packages: ["../dsh-maestro-notifier"] → pnpm install links it.

See Also

• Spec: <workspace-root>/docs/specs/2026-09-13-supervisor-restart-resilience-design.md
• Skill: skills/dsh-safe-restart/ (restart-dsh-web.sh with dry_boot_and_verify() and --auto)
• Client bundling: dsh-maestro-mobile (scripts/build-client.mjs pattern)