dsh-agent-evaluator

An agent evaluation plugin for DeepSeek Harness: import test sets, run agents, score results, and generate reports.

Chinese

Install

cd /path/to/deepseek-harness
pnpm dsh plugin --profile web add ./dsh-agent-evaluator-0.10.0.tgz
pnpm dsh web

If dsh is installed globally, use it instead of pnpm dsh. Cloning the repository or using a git URL also works.

Usage

Web UI: Settings → Agent Evaluation, or run /eval commands:

/eval import /path/to/suite.jsonl
/eval model provider/model
/eval judge provider/model
/eval run suite --parallel 4 --retries 1
/eval progress <jobId>
/eval report <jobId>

Subcommands: import, list, drop, model, models, judge, run, resume, kill, runs, progress, report.

Headless / CI:

cd /path/to/deepseek-harness
pnpm dsh --profile eval eval import suite.jsonl
pnpm dsh --profile eval eval model provider/model
pnpm dsh --profile eval eval run suite --parallel 4

Features

• Imports JSONL / JSON / CSV / TSV and directories, adapting common benchmark field names.
• Runs each case in an isolated agent with concurrency, retries, and timeouts.
• Scorers: exact, contains, llm.
• Persists reports under $DSH_EVAL_HOME/eval/reports, with checkpoint/resume support.
• Web panel provides model pickers, live progress bars, and pass/fail charts.

Documentation

• INSTALL.md / INSTALL.zh.md — installation
• USAGE.md / USAGE.zh.md — full usage
• AGENTS.md / AGENTS.zh.md — architecture and maintenance

Development

node --test test/*.test.js

Integration tests and client builds require a deepseek-harness checkout; see AGENTS.md.

MIT License, see LICENSE.