TeachReplay

Teach once, replay anywhere.

🌐 Project website

TeachReplay is a harness-agnostic Teach-by-Demonstration engine: show a workflow once — clicks, typed values, shell commands — and it becomes a reusable, parameterized skill that can be replayed later with different inputs and an explicit success/failure verdict on real computers.

---

What is TeachReplay?

Teach → Record → Compile → Replay → Verify

1. Teach — start a recording and demonstrate the task once on a computer.
2. Record — the recorder watches the computer's semantic state (URLs, visible text, element roles and values) and diffs it into a versioned trajectory. Password fields stay masked; every value passes secret redaction.
3. Compile — a deterministic compiler turns the trajectory into a parameterized skill: your demonstrated values become inputs with your values as defaults. GUI steps are grounded semantically (role + name, never raw coordinates); shell commands become real CLI steps with their recorded exit codes.
4. Replay — the engine re-snapshots per step, matches targets semantically, acts, and verifies each step's effect. One bounded retry, an optional model-assisted recovery hook, then explicit failure.
5. Verify — success requires the recorded condition (URL or confirmation text) to hold. "The engine stopped" is never success.

Package overview

Package	Purpose
@teachreplay/core	trajectory schema · recorder · parameterized skill compiler · parameter substitution · replay engine · verifier · file stores · createTeachRuntime orchestration
@teachreplay/remote	generic SSH Linux computer backend (Xvfb + Chrome DevTools) with GUI + shell channels — not provider-specific
@teachreplay/mock	deterministic in-memory demo computer for tests and local demos
@teachreplay/adapter-dsh	DeepSeek Harness plugin registering teach_* tools

Architecture

The core owns all Teach-by-Demonstration logic and depends on nothing but Node.js — an enforced invariant (independence test). Adapters contribute only backends, stores, and event sinks:

Integration	Status
Standalone (createTeachRuntime + file stores + mock/remote backends)	✅ included — see the standalone demo
OpenMausBot (TeachReplay-OpenMausBot)	✅ thin adapter — the v0.1 integration rebuilt on the core
DeepSeek Harness (deepseek-ai/deepseek-harness)	✅ adapter targets the real DSH plugin API — see below

TeachReplay for DeepSeek Harness

@teachreplay/adapter-dsh adds teach-by-demonstration tools to DeepSeek Harness agents: teach_start / teach_stop / teach_compile / teach_replay / teach_shell — the same Record → Compile → Replay → Verify engine, no core logic duplicated.

• Targets the real DSH plugin API: @deepseek-ai/cordis / @deepseek-ai/dsh-tools are peerDependencies, pinned as devDependencies (4.0.1 / 0.1.1-rc.2, the exact pair DSH resolves at dsh-v0.1.1-rc.2) so this repo's own build compiles the plugin against the real defineTool/Plugin.Object types — no local shims. A prior DSH-native rewrite verified inside a full DSH workspace checkout (see the adapter README) is what this version is based on
• Installation: pnpm add @teachreplay/adapter-dsh — DSH's sub-packages are published to npm under the next dist-tag while DSH is in developer preview; see the adapter README (⚠️ DSH is in developer preview with compatibility-breaking changes; re-verify per release)
• Demo: examples/dsh-demo runs teach_start → demonstrate → teach_stop → teach_compile → change parameters → teach_replay → success locally — against the mock computer through DshTeachSession directly, not through DSH's real tool runtime

(DSH does not accept external pull requests while in developer preview — see its own CONTRIBUTING.md — so this ships as an independent package rather than an upstream PR.)

Quick start

git clone https://github.com/Ottohere-Mourn/TeachReplay.git
cd TeachReplay
pnpm install
pnpm build
pnpm demo        # standalone: record → compile → change parameters → replay → verify
pnpm test        # 50 tests, including the zero-harness-dependency invariant
pnpm typecheck

The demo runs entirely on the built-in demo computer — no credentials, no network, no harness.

Real evaluation

scripts/mini-benchmark.mjs runs 8 tasks / 14 replays against a real remote Linux computer (GUI + CLI, SSH env-configured). Every success is checked against ground truth on the machine, not just the replay's own verdict. Latest real run (results in scripts/teach-benchmark-results.json):

Metric	Result
Verdicts correct	14/14
Normal replay success	12/12
Changed-parameter replays	6/6
GUI + CLI workflow	success, processed output matches
Severe UI drift / missing element	detected as explicit failures (2/2)
Ground-truth mismatches	0

An honest small-scale sanity check — not a claim of state of the art.

Limitations

• Recording observes state transitions (polled, ~500 ms), not input events — very fast actions can coalesce. Slow, deliberate demonstration is the intended input.
• Clicks are inferred from state changes; ambiguous transitions record no click rather than a wrong one.
• Model-assisted recovery is an optional hook (ModelBackend) — adapters wire the agent.
• Replay speed is SSH-round-trip bound (~4 s/step on the evaluation box).
• DSH is in developer preview with compatibility-breaking changes — re-verify per release.

Attribution and license

Apache-2.0 (see LICENSE). TeachReplay was extracted from TeachReplay v0.1, developed inside OpenMausBot (Apache-2.0) — see NOTICE for the derived-work attributions, including the CDP helper derived from OpenMausBot's computer-use tooling. Record-to-skill is not a new idea — TeachReplay's value is the working, verified combination of demonstration → structured trajectory → parameterized skill → executable replay → explicit verification across GUI + CLI.