dsh-kb-manager

English · 简体中文

Import once. Retrieve with confidence.

dsh-kb-manager is a local knowledge-base lifecycle plugin for DeepSeek Harness (dsh): multi-format import → smart chunking → pure-TypeScript vector indexes → hybrid search (vector + BM25 → RRF → optional rerank) → citation tracing—plus snapshots/rollback, directory-watch sync, and portable .kbpack archives.

Ask in natural language. The plugin exposes 16 Agent tools and an optional Web panel (KB list, import wizard, search bench, source locator)—no separate RAG server required.

Why dsh-kb-manager?

Capability	What it changes
Multi-format import	PDF (page-aware; scan pages flagged), DOCX, Markdown, HTML (Readability), CSV, JSON, TXT, plus URL fetch.
Smart chunking	fixed / recursive (default) / semantic; heading breadcrumbs, table-as-chunk, page/paragraph provenance.
Pure-TS indexes	Flat (exact) + HNSW (approx)—no native addons.
Hybrid retrieval	Dual-path recall → RRF (k=60) → optional rerank; debug: true returns per-stage scores.
Offline fallback chain	Local / OpenAI-compatible embedding → built-in HashEmbedder; rerank failure falls back to RRF order.
Snapshots & rollback	Manifest + index backup; embedding-model mismatch refuses unsafe restore.
.kbpack portability	tar.gz of source + chunks + index + metadata; create_new / merge import.
Directory sync	chokidar + SHA-256 change detection + catch-up scan; deleted watch dirs pause without cascading deletes.
Read-only mode	read_only: true makes write tools return read_only_mode immediately.
Agent-friendly tools	Few-shot examples embedded in every tool description.

Architecture

flowchart LR
  A[Documents / URLs] --> B[Parse]
  B --> C[Chunk]
  C --> D[Embed]
  D --> E[Vector index<br/>Flat / HNSW]
  C --> F[BM25]
  E --> G[Hybrid search]
  F --> G
  G --> H[RRF]
  H --> I[Optional rerank]
  I --> J[Citations + debug scores]
  E --> K[Snapshots / .kbpack]
  C --> K

Install

[!NOTE] Requires an existing DeepSeek Harness installation.

From GitHub

dsh plugin --profile web add github:xiaoshi7915/dsh-kb-manager

On git install, the prepare script runs tsdown for a self-contained build (no monorepo context needed).

Build from source

git clone https://github.com/xiaoshi7915/dsh-kb-manager.git
cd dsh-kb-manager
npm install
npm run build
dsh plugin --profile web add .

Validate the composed profile, restart DSH, and refresh the Web UI:

dsh --profile web --dump-config
dsh web

Then try:

Create a knowledge base named “project-docs”, import this PDF, and search for how authentication works. Show me the source chunk.

How it works

1. Create a KB (create_kb) with an embedding model (default offline hash-embed-v1, or any OpenAI-compatible name when embedding_api_base is set).
2. Import local files or URLs (import_document) → parse → chunk → embed → index.
3. Search with search_kb / multi_kb_search: vector + BM25 → RRF → optional rerank; use debug: true to inspect stage scores.
4. Trace hits with get_chunk (surrounding context + provenance metadata).
5. Optionally snapshot before risky edits, restore later, or ship a .kbpack to another machine.
6. Optional auto_sync_dir watches a folder and incrementally updates the target KB.

Data lives under storage_path (default ~/.dsh/kb-manager/). The Web client reads the same service surface for overview, import, search, and source location.

Agent tools

Tool	Description	Key params
create_kb	Create a knowledge base	name*, description*, embedding_model?, tags?
list_kbs	List all knowledge bases	—
get_kb	KB details (model / docs / chunks / storage)	kb_id*
delete_kb	Delete an entire knowledge base	kb_id*
import_document	Import a local file or URL	kb_id*, source*, metadata?
list_documents	List documents (optional status filter)	kb_id*, status_filter?
delete_document	Soft-delete chunks; mark index dirty	kb_id*, doc_id*
search_kb	Hybrid search; optional debug scores	kb_id*, query*, top_k?, filters?, rerank?, debug?
multi_kb_search	Cross-KB search with source_kb	kb_ids*, query*, top_k?
get_chunk	Chunk text + neighbors for tracing	kb_id*, chunk_id*
rebuild_index	Full rebuild via shadow index swap	kb_id*
create_snapshot	Create a version snapshot	kb_id*, note?
restore_snapshot	Roll back to a snapshot	kb_id*, snapshot_id*
export_kb	Export .kbpack or JSON	kb_id*, format?, output_path*
import_kb	Import a pack (create_new / merge)	file_path*, merge_strategy?
get_kb_stats	Doc / chunk / index / avg-length stats	kb_id*

Write tools (create_kb, delete_kb, import_document, delete_document, rebuild_index, create_snapshot, restore_snapshot, import_kb) return { success: false, error: 'read_only_mode' } when read-only. Domain errors are returned as { success: false, error: <code>, message } (not thrown).

When the Agent uses it

1. “Put this PDF / doc / page into the knowledge base” → import_document
2. “Search the KB for X” → search_kb
3. “Find Y across all knowledge bases” → multi_kb_search
4. “Where did this citation come from?” → get_chunk
5. “Retrieval feels wrong—show stage scores” → search_kb({ debug: true })
6. “Back up before a big change” → create_snapshot
7. “Roll back to before the change” → restore_snapshot
8. “Pack this KB for another machine” → export_kb + import_kb
9. “Will local file edits sync?” → directory watch incremental sync
10. “How big is this KB?” → list_kbs / get_kb_stats

Configuration

Defaults work offline. Override in a trusted profile (id: kb-manager):

Field	Default	Notes
storage_path	~/.dsh/kb-manager/	Storage root; ~ expands to the home directory
default_embedding_model	hash-embed-v1	Offline default; or an OpenAI-compatible model name
embedding_api_base	''	OpenAI-compatible endpoint; empty → HashEmbedder
embedding_api_key	''	Embedding API key
chunk_size	512	Chunk size (characters)
chunk_overlap	50	Overlap length
chunk_strategy	recursive	fixed / recursive / semantic
top_k	5	Default hit count
enable_rerank	true	Enable post-RRF rerank
rerank_endpoint	''	Remote rerank; empty → built-in rule reranker
index_type	hnsw	hnsw / flat
auto_sync_dir	''	Watch path; empty disables
auto_sync_kb_id	''	Target KB for sync
auto_sync_interval	300	Catch-up interval (seconds)
max_file_size_mb	100	Per-file size cap (MB)
read_only	false	Disable write tools

Example:

- id: kb-manager
  config:
    storage_path: ~/.dsh/kb-manager/
    default_embedding_model: text-embedding-3-small
    embedding_api_base: https://api.openai.com/v1
    chunk_strategy: recursive
    index_type: hnsw
    enable_rerank: true
    read_only: false

Boundaries

• Reads/writes under storage_path; import_document / import_kb also read user-specified paths or URLs; export_kb writes to output_path.
• Unsupported formats return unsupported_format; oversized files are rejected by max_file_size_mb.
• Fully local by default; network only when embedding_api_base / rerank_endpoint or URL import is configured.
• export_kb does not mutate KB data and remains available in read-only mode.
• Pure-TS indexes: no native vector DB process to operate.

Differentiating vs typical RAG stacks

Capability	This plugin	Common RAGFlow / Dify / kotaemon / pdfkb-mcp setups
KB snapshots & rollback	✅	Usually missing
.kbpack (source + chunks + index + metadata)	✅	Usually missing
Directory watch + catch-up scan	✅ chokidar + SHA-256	Partial / missing
Per-stage retrieval debug scores	✅ vector / BM25 / RRF / rerank	Often opaque
Full offline embed fallback	✅ HashEmbedder	Often needs external services
Read-only mode	✅	Rare

Layout

dsh-kb-manager/
├── package.json  cordis.patch.yml  tsconfig.json  tsdown.config.ts  vitest.config.ts
├── awesome-entry.yml  README.md  README_ZH.md
├── assets/readme/          # hero.png (paste generated banner here)
├── src/
│   ├── index.ts            # Plugin entry (name / inject / Config / apply)
│   ├── config.ts           # Schemastery config schema
│   ├── core/  parse/  chunk/
│   ├── embed/  index/  search/
│   ├── kb/                 # KBService, snapshots, sync, kbpack
│   ├── tools/              # 16 Agent tools
│   └── client/             # Web panel
└── tests/

Development

npm install        # install deps (prepare builds automatically)
npm run build      # tsdown dual entry → lib/index.js + lib/client.js
npm test           # vitest
npm run typecheck  # tsc --noEmit

License

MIT © 2026 xiaoshi7915