dsh-web-fetch-playwright

中文 · npm · GitHub

A DeepSeek Harness (DSH) plugin that gives the built-in web_fetch tool a Playwright/CDP backend: pages are rendered in a real browser, denoised with Readability + DOMPurify + Turndown + GFM, and returned as Markdown.

Features

• Real browser rendering — loads the page the way a user sees it, so client-side rendered (SPA) content is captured, not just the raw HTML.
• Denoise pipeline — Mozilla Readability extracts the article, DOMPurify removes layout/noise tags (nav, sidebar, footer, ads, forms), and Turndown with the GFM plugin converts to Markdown with the same style options as the shipped tool-web renderer.
• Two backends — launch a local Playwright browser, or drive an already-running browser over its DevTools Protocol (CDP) endpoint.
• Browser resolution — a configured path, a playwright CLI on $PATH, or the bundled playwright-core; CDP needs no local browser at all.
• Isolated sessions — every fetch uses its own browser context. Local launches close their browser per fetch; the CDP backend keeps one shared connection to the remote browser and each fetch opens an isolated context (a tab) inside it, closed when done.
• Live configuration — a settings card (设置 → 插件 → 插件配置) edits the backend, denoise toggle, and concurrency; changes apply to the next fetch without a restart.
• Budget-aware — per-fetch deadline (45s); concurrency is backend-priced (maxConcurrency, default 4 local browsers / 50 CDP tabs; queued fetches fail fast with a retry hint after 20s instead of hanging); image/font/media subrequests aborted; body capped at 100k chars.

How it works

Half	Location	Responsibility
Host (server)	src/	Registers the fetch provider (id playwright) into ctx.web; cordis.patch.yml pins the web seam's fetchProvider to it and enables the web_fetch tool with a 60s budget.
Browser (client)	src/client/	Registers the Playwright 网页爬取 configuration card, which hot-writes the settings section into $DSH_HOME/settings.yaml.

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = playwright
        ├─ local: resolve (path → $PATH → bundled playwright-core) → chromium.launch
        ├─ cdp:   connectOverCDP(endpoint)
        ├─ page.goto → settle (networkidle, best-effort) → page.content()
        ├─ denoise: jsdom → Readability → DOMPurify → Turndown(GFM)
        └─ Markdown (or raw HTML when denoise is off)

Requirements

• DSH web profile (dsh web), Node.js ≥ 20.
• For the local backend: a Playwright installation with Chromium, a Chromium-family browser binary, or playwright-core with a browser in the default cache.
• For the CDP backend: any browser already running with --remote-debugging-port (e.g. chromium --headless --remote-debugging-port=9222).

Installation

From the npm registry (prebuilt — no build permission needed):

dsh plugin --profile web add dsh-web-fetch-playwright

From a GitHub repository (source — pnpm runs the prepare build; allowlist the package in profiles/web/pnpm-workspace.yaml if pnpm blocks the build script):

dsh plugin --profile web add github:chendefine/dsh-web-fetch-playwright

Or through the DSH plugin marketplace (设置 → DSH插件市场) — the repo carries the dsh-plugin topic and is indexed automatically.

After a bundle plugin is added to the profile layer stack, restart dsh web for it to load; uninstall with dsh plugin --profile web remove dsh-web-fetch-playwright and restart again.

Configuration

The settings card (设置 → 插件 → 插件配置 → Playwright 网页爬取) edits the web-fetch-playwright settings section live:

Field	Default	Description
backend	local	Radio: Local Playwright or Remote CDP endpoint, each with its own nested input.
playwrightPath	(blank)	Local backend: path to a playwright executable or a Chromium-family browser binary. Blank = discover on $PATH, then fall back to the bundled playwright-core.
cdpEndpoint	127.0.0.1:9222	Remote backend: host:port, http(s)://… or ws(s)://….
denoise	true	Run the denoise pipeline; off returns the full rendered HTML for the tool layer to convert.
maxConcurrency	(auto)	How many fetches may render at once (1–200). Blank = backend default: 4 for local (each slot launches a browser) / 50 for CDP (each slot is just a tab in the already-running remote browser). Beyond the limit, fetches wait briefly; if no slot frees within 20s they fail with WEB_FETCH_TIMEOUT and a hint to retry or raise this setting, rather than hanging until the tool budget aborts.

Local backend resolution order:

1. The configured path (auto-detected as Playwright CLI or browser binary).
2. A playwright executable on $PATH (its package knows that installation's browser registry).
3. The bundled playwright-core — requires PLAYWRIGHT_BROWSERS_PATH or browsers in the default cache; otherwise the error suggests playwright install chromium.

CDP mode needs no local browser: the provider holds one shared connection for its lifetime (reconnecting automatically if it drops, and reconnecting to the new endpoint when the setting changes), and every fetch leases a fresh isolated context — a tab in your browser — that closes when the fetch completes. Concurrency therefore counts tabs, which is why the CDP default is high (50). Unloading the plugin drops the shared connection.

Development

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm test        # vitest run (browser smoke self-skips without a browser)
pnpm build       # tsc declarations + tsdown (host ESM + client module-registration bundle)

Repository layout:

src/
├── index.ts               # host entry: registers provider + settings section
├── config.ts              # schemastery schema, CDP endpoint normalizer
├── provider.ts            # WebFetchProvider: navigation, deadline, semaphore, caps
├── markdown.ts            # denoise pipeline (Readability + DOMPurify + Turndown/GFM)
├── playwright-resolve.ts  # local backend discovery (path / $PATH / bundled core)
├── types.ts               # structural Playwright types (runtime module discovered dynamically)
└── client/                # browser half: settings card, form model, locales
tests/                     # unit + provider + browser integration (self-skipping)

See CONTRIBUTING.md for the development and release workflow, and SECURITY.md for the security model and reporting policy.

Security

Same stance as the built-in HTTP provider: no SSRF / private-network protection is implemented — anything the browser can reach, this provider can fetch. The CDP endpoint is configured from the settings page with no loopback restriction, so only expose the settings page to trusted environments. Fetched pages are rendered locally; no data is sent anywhere beyond the target page itself.

License

MIT © 2026 chendefine