dsh-langfuse

Langfuse LLM observability for DeepSeek Harness — one OpenTelemetry trace tree per agent session, with feedback scores and subagent lineage.

Send every agent session to Langfuse as an OpenTelemetry trace tree — turn → trace, model step → generation, tool call → tool span — with GenAI semantic-convention attributes, token usage, time-to-first-token, and error statuses.

✨ Features

• 🧵 Full session tracing — every turn, generation, and tool call lands in Langfuse with model, provider, usage (incl. cache-read & reasoning tokens), and TTFT.
• 🚚 Content-Length transport — spans are delivered as a single write with an explicit Content-Length header, never chunked. Some Langfuse deployments sit behind gateways that mangle chunked POST bodies (400 Failed to parse OTel JSON Trace); this transport sidesteps that class of bugs entirely.
• 👍 Feedback scores — /feedback records become TEXT scores on the session's latest turn trace.
• 🔗 Subagent lineage — child-session turn traces link to the parent session's trace (same-process).
• 🎚️ Three sharing modes — FULL (live), FEEDBACK_ONLY (export only when the user records feedback), DISABLED (default; nothing leaves the process).
• 🔒 Opt-in inputs — generation input (system prompt / tools / prompt) is exported only when includeGenerationInput is on.
• 🛡️ Fail-loud config — bad URLs, missing keys, and invalid bounds throw at plugin load, before any transport exists.

🚀 Quick Start

dsh plugin --profile web add dsh-langfuse

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
# optional: self-hosted or non-EU region
export LANGFUSE_HOST=https://langfuse.example.com

dsh web   # restart the running instance, then take a turn

With a key present the backend runs in FULL mode; without one it is DISABLED. LANGFUSE_TELEMETRY_MODE=FEEDBACK_ONLY narrows sharing to feedback-gated release, and LANGFUSE_INCLUDE_GENERATION_INPUT=1 exports generation inputs.

The bundled patch disables the base profile's session-telemetry-otel row (the telemetry seam accepts exactly one backend per context) and mounts this one. dsh --profile web --dump-config shows the composed result without booting.

As an explicit cordis.yml row

- id: session-telemetry-langfuse
  name: dsh-langfuse
  config:
    mode: FULL                 # FULL | FEEDBACK_ONLY | DISABLED (default)
    exporter:
      url: https://cloud.langfuse.com/api/public/otel/v1/traces
    auth:
      publicKey: !!js process.env.LANGFUSE_PUBLIC_KEY
      secretKey: !!js process.env.LANGFUSE_SECRET_KEY
    processor: {}              # optional; passed verbatim to BatchSpanProcessor
    includeGenerationInput: false

⚙️ Configuration

Field	Default	Meaning
mode	DISABLED	FULL exports every session live; FEEDBACK_ONLY exports the canonical session log only when feedback is recorded; DISABLED constructs nothing.
exporter.url	—	Full OTLP traces endpoint (…/api/public/otel/v1/traces). Required unless DISABLED.
exporter.timeoutMillis	10000	Per-request timeout.
auth.publicKey / auth.secretKey	—	Langfuse project key pair (Basic auth). Required unless DISABLED.
processor	{}	Passed verbatim to BatchSpanProcessor (scheduledDelayMillis, maxQueueSize, …). Retries (bounded backoff, 6 attempts) live in the exporter.
includeGenerationInput	false	Export langfuse.observation.input (system prompt / tool schemas / model / user prompt).
maxAttributeChars	16384	Per-attribute payload ceiling; longer payloads are truncated with a …[truncated] marker (the canonical log keeps the full bytes).
shutdownTimeoutMillis	3000	Deadline on the SDK's shutdown drain.
feedbackScoreName	user-feedback	Langfuse score name for /feedback pushes.

🧭 What appears in Langfuse

dsh session event	Langfuse concept
turn/start / turn/end	trace (turn N; error end reasons → ERROR status)
step/start + request/header + assistant/message	generation (step T.S) — model, provider, output, gen_ai.usage.* tokens
first assistant/chunk of a step	langfuse.observation.completion_start_time (TTFT)
tool/call + tool/result	tool span (arguments → input, result → output, isError → ERROR)
user/message	trace input
feedback/record	TEXT score on the session's latest turn trace
subagent session turns	trace links to the parent session's trace
agent-error ops record	exception event + ERROR status
every other event type (todo, plan, compaction, …)	span event on the open turn

Correlation attributes use this package's own vocabulary (dsh.turn_idx, dsh.step_idx, dsh.event_seq, …) so multiple telemetry backends can share one Langfuse project without attribute collisions.

🏗️ How it works

• A telemetry-seam backend. The harness logs everything model-visible to the canonical session log; this plugin implements SessionTelemetryBackend (@deepseek-ai/dsh-session-telemetry), so it captures every event — including ones it has never heard of — with the seam's consent semantics and redaction waterfall for free.
• OTel SDK + custom exporter. Spans are built with BasicTracerProvider + BatchSpanProcessor, serialized with @opentelemetry/otlp-transformer, and delivered by a custom SpanExporter with Content-Length and bounded retries.
• A folding timeline. SessionTimeline replays the flat record stream into span trees keyed by (session.id, turn, step). Span boundaries always use the record's own timestamp — live capture and canonical-log replay produce identical trees.
• At-most-once delivery (inherited from the seam): the loss window is the crash window; receivers correlate on langfuse.session.id + dsh.turn_idx + dsh.event_seq.

🧪 Development

npm run check   # biome + typecheck + vitest (unit & seam integration) + build

30 tests cover config validation, the folding timeline, the transport (Content-Length, no Transfer-Encoding, retry/backoff), score pushes, and a full seam integration that drives a real SessionTelemetryCoordinator and asserts the wire payload.

⚠️ Limitations

• One backend per context (the seam throws on duplicates — the bundled patch disables the official OTLP-logs row).
• Subagent lineage is same-process best-effort (a resumed parent's trace id is unknown across restarts).
• No durable delivery (at-most-once).
• Score pushes are fire-and-forget (failures only log).

📄 License

MIT