Merge remote-tracking branch 'origin/master' into web-e2e-interactions
# Conflicts: # .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml # .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md # .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.zh.md # apps/web/tests/scaffold.ts
This commit is contained in:
Tianyi Cui
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: 796c0812b91f059e52fc82238802bd12e1e3a93f
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 3e725b92cda3beaf47f6c3d3f8dfb2b02dc1ae7e
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: 78a01652ef35f371c35c059033cd28f29f5bf94e
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: b5fb29c63ab10352d50ef6ba9ce7b65e92989387
|
||||
|
||||
@@ -6,7 +6,7 @@ English | [中文](2026-07-24-web-gui-browser-e2e-lane.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → `bootHost`'s agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits.
|
||||
The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → the host agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -16,33 +16,33 @@ The web GUI ships as a real assembled chain — chromium page → client plugin
|
||||
|
||||
A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners.
|
||||
|
||||
`launchWebScaffold()` boots the real web composition: the shipped `apps/cli/cordis.yml` through the vendored Loader's include boot — the same tree and mechanism `AppCLIEntry` drives for `dsh web` (the config-tree boot landed upstream on 2026-07-25, superseding this lane's earlier in-process `startHost` assembly and resolving the original Loader-ization question in favor of Loader-izing). Divergences ride include patches over the SAME shipped tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`.
|
||||
`launchWebScaffold()` boots the real web composition from the shipped `apps/cli/cordis.yml` through the vendored Loader's include mechanism — the same tree and mechanism `AppCLIEntry` drives for `dsh web`. Divergences ride include patches over that tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`. Setup rollback and ordinary close both dispose the Cordis tree before removing the two owned temp roots, attempt every cleanup independently, and report cleanup failures without masking the setup failure.
|
||||
|
||||
Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER. (The first round's `BootHostOptions.llm: 'deepseek' | false` seam was superseded by the config-tree boot and removed with `bootHost`'s web role.)
|
||||
Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER.
|
||||
|
||||
`seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair).
|
||||
|
||||
### Determinism rules
|
||||
|
||||
The barrier stack for a prompted turn, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible); (3) any log harvest after `host.dispose()`. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
|
||||
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
|
||||
|
||||
No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly.
|
||||
|
||||
Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, `data-*` attributes, and visible text.
|
||||
Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; interaction selectors anchor on roles, `data-*` attributes, and visible text, while the frame and conversation-region captures use the existing CSS-module local-name anchors.
|
||||
|
||||
### Expected outputs
|
||||
|
||||
One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride `host.ctx` session events inline (which tools ran, `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
||||
One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
||||
|
||||
The typecheck plane split is structural: `apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` are host-plane programs (they boot the host spine), so they are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
||||
The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
||||
|
||||
### Modes and fixtures
|
||||
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Every prompting scenario's fixture was recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Every prompting scenario's fixture was recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
||||
|
||||
### Scenarios
|
||||
|
||||
1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (bash `tool/call`, completed `turn/end`, >10 chunk events).
|
||||
1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events).
|
||||
2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed.
|
||||
3. **`live-interactions`** — one tool-free recorded turn serves three replay-only scenarios through override sidecars whose CONTENT is authored in the spec and minted as a per-run file in a spec-owned temp dir (the derived success entry for the retry append is re-derived from the fixture via `deriveReplayScript`, never copied into a committed sidecar). Cancel: a `{ patches }` `hang` with a `readyFile` marker — the marker's existence proves the stream is parked mid-turn before the test clicks Stop, making mid-stream cancellation deterministic by construction (`turn/end` reason `aborted`, composer re-enabled). AUTH error: a pre-chunk `throw` outside llm-retry's retryable set (`turn/end` reason `error`, zero `llm/retry` events, composer recovers). SERVER retry: `throw` at call 0 + the fixture's own success appended at 1, proving llm-retry end-to-end in the browser via the durable `llm/retry` record (`request/header` logs only on change, so attempt count is invisible there).
|
||||
4. **`question-composer`** — the shipped composition's resident `ask_user_question` takeover: a recorded turn blocks mid-step on the real userInteraction seam, the composer (`[data-question-key]`) renders in the browser, the test answers through it (the ONE sanctioned place a drive step reacts to model content: the turn cannot complete without the answer, in record and replay alike), and the tool result carries the chosen label. Golden: the composer's stable waiting state.
|
||||
@@ -68,9 +68,9 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.
|
||||
|
||||
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on `host.ctx` events keep the world-verification duty.
|
||||
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty.
|
||||
|
||||
**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions; the bin's thin glue is covered by the keyless CLI smokes. Becomes free only if the web host is ever Loader-ized — declined in review, with the app-assembly ruling reaffirmed.
|
||||
**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-only replay branch plus environment plumbing in the shipped CLI. The in-process scaffold already loads the same `apps/cli/cordis.yml`; only argv, profile JSON, and `AppCLIEntry` glue remain outside it, and the keyless CLI smokes cover those paths.
|
||||
|
||||
**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose.
|
||||
|
||||
@@ -84,7 +84,7 @@ The lane itself: `pnpm run test:web` runs every scenario keylessly alongside the
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins bootHost's composed prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors.
|
||||
- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins the web composition's prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors.
|
||||
- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`).
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
- **Web error surface**: the client consumes no `agent/error` frames and a pre-chunk failure freezes no partial, so a non-retryable provider failure renders no error copy — the user sees the send simply stop. The AUTH scenario pins the current contract (no crash, composer recovers, turn logged `error`) and `FIXME(web-error-surface)` marks where visible error text gets asserted once the UI grows an error rendering.
|
||||
|
||||
@@ -6,7 +6,7 @@ Status: implemented
|
||||
|
||||
## 问题
|
||||
|
||||
Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → `bootHost` 的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。
|
||||
Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → host 端的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。
|
||||
|
||||
## 决策
|
||||
|
||||
@@ -16,33 +16,33 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
一个普通的共享 fixture 模块（[测试政策认可的形态](../../../../docs/testing.md)），不是包：值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中；剩下的只是启动接线和浏览器胶水，而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。
|
||||
|
||||
`launchWebScaffold()` 启动真实 web 组合：经 vendored Loader 的 include boot 加载交付的 `apps/cli/cordis.yml`——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制（配置树 boot 于 2026-07-25 在上游落地，取代了本车道第一轮的进程内 `startHost` 组装，也把当初的 Loader 化问题裁定为「Loader 化」）。差异全部经 include patch 骑在同一棵交付树上，即 ACP `cordis.snapshot.yml` 模式的进程内表达：临时 `persistenceRoot`；禁用 `workspace-context`（录制的 fixture 不得嵌入本仓库的 AGENTS.md）；禁用 `session-title-llm`（其发后不管的标题调用会与循环争抢会话的回放游标）；webserver 行钉到端口 0 加已构建 dist；无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行，boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行，使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致；`dsh web` bin 自身的胶水（argv、profile json、AppCLIEntry）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。
|
||||
`launchWebScaffold()` 通过 vendored Loader 的 include 机制，从交付的 `apps/cli/cordis.yml` 启动真实 web 组合——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制。差异全部经 include patch 覆盖在这棵树上，即 ACP `cordis.snapshot.yml` 模式的进程内表达：临时 `persistenceRoot`；禁用 `workspace-context`（录制的 fixture 不得嵌入本仓库的 AGENTS.md）；禁用 `session-title-llm`（其发后不管的标题调用会与循环争抢会话的回放游标）；webserver 行钉到端口 0 加已构建 dist；无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行，boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行，使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致；`dsh web` bin 自身的胶水（argv、profile json、AppCLIEntry）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。初始化回滚和正常关闭都会先对 Cordis 树执行 dispose（资源释放），再删除 scaffold 持有的两个临时根目录；每项清理都会独立尝试，并会报告清理失败而不掩盖初始化失败。
|
||||
|
||||
无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录（providers-catalog）模式填充开放的 seam——绝不用 catch-all：适配器行被禁用后不存在任何适配器，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置（发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置）。选择直接安装而非插入回放插件行是刻意的：直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置，任何离群的流式调用都会以 NO_ADAPTER 大声失败。（第一轮的 `BootHostOptions.llm: 'deepseek' | false` seam 已随配置树 boot 取代 `bootHost` 的 web 角色而移除。）
|
||||
无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录（providers-catalog）模式填充开放的 seam——绝不用 catch-all：适配器行被禁用后不存在任何适配器，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置（发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置）。选择直接安装而非插入回放插件行是刻意的：直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置，任何离群的流式调用都会以 NO_ADAPTER 大声失败。
|
||||
|
||||
`seedSession()` 通过真实持久化 API 播种冷会话——一次性 `Context` 挂载 `SessionStore` + `SessionPersistenceJsonl` 指向 host 的根目录，`create()` + `append()`，一次 `utimes` 回拨保证侧栏顺序确定（`semantic-checkpoint.snapshot.ts` 先例）——绝不裸写文件，因此播种器对桶哈希、文件名编码、压缩一无所知，host 的 zstd 默认值也无需任何启动开关。种子在播种时即校验（可解析、以 `turn/end` 结尾——未闭合的最终轮次会被恢复（resume）的崩溃修复改写）。
|
||||
|
||||
### 确定性规则
|
||||
|
||||
提示一轮对话的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）；（3）任何日志采收都在 `host.dispose()` 之后。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。
|
||||
回放模式下浏览器断言的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）。录制模式下，日志采收在 `whenIdle()` 之后、scaffold 释放之前进行，此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。
|
||||
|
||||
不做单次瞬态 DOM 断言：从回放产出到 React 提交的每一跳都可能合并分片，采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言（模型可见 ⟺ 已记录，使日志成为权威证据）。`dsh-llm-replay` 的可选 `paceMs`（默认缺省 = 突发）只是让浏览器观察到真正增量 SSE 的真实感旋钮；正确性绝不依赖它，且节奏等待期间中止会即时取消。
|
||||
|
||||
每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；选择器只锚定 role、`data-*` 属性和可见文本。
|
||||
每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；交互选择器锚定 role、`data-*` 属性和可见文本，而 frame 与会话区采集则使用既有的 CSS 模块局部类名锚点。
|
||||
|
||||
### 预期输出
|
||||
|
||||
每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在 `host.ctx` 会话事件上（哪些工具运行了、`turn/end` 完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。
|
||||
每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在根上下文的会话事件上（哪次工具调用产生了哪项已持久化的工具结果、`turn/end` 是否完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。
|
||||
|
||||
类型检查平面切分是结构性的：`apps/web/tests/{scaffold,support,replay-round-trip.e2e,seeded-history.e2e}.ts` 是 host 平面程序（它们启动 host 主干），因此被排除出注册在 client 侧的 `apps/web` 工程，逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。
|
||||
类型检查平面切分是结构性的：启动 host 主干的三个文件（`scaffold`、`replay-round-trip.e2e` 和 `seeded-history.e2e`）被排除出注册在 client 侧的 `apps/web` 工程。这三个文件及其共享的 `support.ts` 逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。
|
||||
|
||||
### 模式与 fixture
|
||||
|
||||
`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。每个发起提示的场景，其 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。
|
||||
`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。每个发起提示的场景，其 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点，不含当次运行的 `rpcId`）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。
|
||||
|
||||
### 场景
|
||||
|
||||
1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（bash `tool/call`、完成的 `turn/end`、>10 个分片事件）。
|
||||
1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（这次 bash 调用的已持久化工具结果严格等于 `WEB_E2E_OK\n`、完成的 `turn/end`、>10 个分片事件）。
|
||||
2. **`seeded-history`**——冷播种一份已录会话；侧栏列出它（分组行 → 会话行，默认折叠），打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用，因此没有任何绑定约束；record 模式实时驱动同一轮（真实 `read` 工具读取播种的工作区文件）来产出种子。
|
||||
3. **`live-interactions`**——一段不含工具调用的已录轮次经覆写 sidecar 承载三个仅回放的场景：sidecar 的内容本身写在 spec 里，每次运行时在 spec 自有的临时目录中生成文件（重试追加所用的派生成功条目经 `deriveReplayScript` 从 fixture 重新派生，绝不复制进已提交的 sidecar）。取消：一个带 `readyFile` 标记的 `{ patches }` `hang`——标记文件的存在证明流在测试点击 Stop 之前已停驻在轮次中途，使流中取消按构造即确定（`turn/end` 原因为 `aborted`，输入框重新启用）。AUTH 错误：一次落在 llm-retry 可重试集合之外的分片前 `throw`（`turn/end` 原因为 `error`，零条 `llm/retry` 事件，输入框恢复可用）。SERVER 重试：第 0 次调用 `throw` + 在第 1 次调用处追加 fixture 自身的成功条目，凭持久的 `llm/retry` 记录在浏览器中端到端证明 llm-retry（`request/header` 仅在变化时记录，因此尝试次数在那里不可见）。
|
||||
4. **`question-composer`**——已交付组合中常驻的 `ask_user_question` 接管：一段已录轮次在真实的 userInteraction seam 上阻塞于步骤中途，提问输入框（`[data-question-key]`）在浏览器中渲染，测试经它作答（这是驱动步骤对模型内容作出反应的唯一获准之处：没有这个回答，轮次无法完成，record 与 replay 皆然），工具结果携带所选的 label。预期输出：提问输入框稳定的等待态。
|
||||
@@ -68,9 +68,9 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决：驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%，且两个场景就上工厂是从单一消费方过度泛化，真正共享的逻辑已从受门禁的包中导出。重启条件：出现第二个 web 形态消费方，或 ≥6 个场景的内联分支被证实各自漂移；届时包边界将画在无浏览器一侧。
|
||||
|
||||
**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在 `host.ctx` 事件上的世界状态断言保住了验证世界的义务。
|
||||
**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在根上下文事件上的世界状态断言保住了验证世界的义务。
|
||||
|
||||
**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在产品 bin 里加测试模式分支和环境变量管道，而进程内路线用的是零产品改动的导出生产函数；bin 的薄胶水已由无密钥 CLI 冒烟覆盖。只有 web host 某天 Loader 化它才免费——评审中已否决，并重申了应用内组装的裁定。
|
||||
**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在交付的 CLI 中增加测试专用回放分支和环境变量管道。进程内 scaffold 已加载同一份 `apps/cli/cordis.yml`；只剩 argv、profile JSON 和 `AppCLIEntry` 胶水不在其覆盖范围内，而这些路径已由无密钥 CLI 冒烟覆盖。
|
||||
|
||||
**为可测试性改 wire 协议。** 已否决：契约已有第一等的无密钥同构 seam（`InProcessApiClient(toFetchHandler(api))`），逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因，测试一条不再交付的 wire 会颠倒该层的存在意义。
|
||||
|
||||
@@ -84,7 +84,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
## 暂缓
|
||||
|
||||
- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 bootHost 组装的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。
|
||||
- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 web 组合的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。
|
||||
- **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。
|
||||
- **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。
|
||||
- **Web 错误表面**：客户端不消费任何 `agent/error` 帧，分片前的失败也没有可冻结的部分输出，因此不可重试的提供方失败不渲染任何错误文案——用户看到的只是发送就此停住。AUTH 场景钉住当前契约（不崩溃、输入框恢复可用、轮次记录为 `error`），`FIXME(web-error-surface)` 标记了待 UI 长出错误渲染后断言可见错误文本的位置。
|
||||
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-25-scriptable-llm-wire-fault-server.md: 0795f71f0eab1a107740aaa8cba6fa04b1fbd306
|
||||
2026-07-25-scriptable-llm-wire-fault-server.zh.md: a0e3c98d729adcc74e6d98933ab2beb538f71cb3
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: Scriptable LLM wire fault server
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-25-scriptable-llm-wire-fault-server.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Adapter unit tests use local HTTP servers to classify individual provider failures, while retry tests use an in-process scripted `LlmAdapter` to prove closed-step recovery. Neither boundary provides a reusable server for running the shipping HTTP adapter, agent loop, and retry policy together, and neither lets a developer point an existing app at deterministic transport faults by changing only its base URL and API key.
|
||||
|
||||
Connection refusal, a reset before the first event, clean EOF without `[DONE]`, a valid content-less completion, and a reset after partial output have different adapter and recovery outcomes. Treating them as one generic mock failure hides whether the provider boundary preserved the distinction and whether failed chunks remained outside committed model history.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` is a support package with an importable Node HTTP server and a standalone CLI. It accepts OpenAI-compatible root and `/v1` chat-completions paths, validates an optional bearer token, captures requests, and consumes one explicit behavior per accepted request. Script exhaustion fails loud; repetition requires `repeatLast`.
|
||||
|
||||
Request behaviors cover socket reset, post-header disconnect, partial disconnect, stall, valid empty completion, clean truncated streams, malformed payloads, representative HTTP failures, complete text/reasoning/tool-call responses, slow streaming, and max-token completion. A true `connection_refused` is a CLI listener-lifecycle phase because a bound request handler cannot refuse its own TCP connection.
|
||||
|
||||
The `random` script entry performs a new weighted selection for every request. The server exposes and logs its unsigned 32-bit seed, accepts caller-supplied relative weights, and ships a success-heavy stress profile that mixes transport, protocol, provider, timeout, and semantic-empty outcomes. The profile is configurable test pressure rather than an estimate of production incident frequency; `connection_refused` remains outside the request-level pool.
|
||||
|
||||
The server reports wire facts only and does not classify retryability. Real-composition tests route it through `dsh-llm-deepseek`, `dsh-agent-loop`, and `dsh-llm-retry`: connection refusal, hard disconnect, partial reset, idle timeout, and a valid content-less completion recover under the existing default policy; clean partial EOF remains `STREAM_CLOSED` and is not retried by default. The package does not change those policies.
|
||||
|
||||
## Verification
|
||||
|
||||
Package tests exercise every request behavior, split UTF-8 request decoding, HTTP validation without script consumption, script exhaustion/repetition, stalled-connection teardown, CLI parsing and delay bounds, IPv6 base URLs, random seed reproducibility, weight validation, single-result telemetry, lifecycle cleanup, and the invariant companion under the per-file coverage gate. The retry integration suite proves exact request counts, numbered retry steps, request-body identity, failed partial-chunk isolation, semantic-empty recovery, clean-EOF classification, timeout recovery, true refused-connection recovery after delayed listener startup, and bounded exhaustion through the real HTTP/SSE adapter.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Implement the server in Python** — rejected because Node's standard HTTP and socket APIs expose every required fault, while TypeScript keeps the server, CLI parser, tests, package build, lint, and coverage inside the repository's existing toolchain. A second runtime would add environment and subprocess dependencies without increasing wire isolation.
|
||||
|
||||
**Keep separate inline mock servers in adapter tests** — rejected because those fixtures cannot be launched by an existing app and would duplicate behavior sequencing, randomization, telemetry, and connection cleanup across suites. A support package gives tests a shared implementation without promoting it to product API.
|
||||
|
||||
**Use only an in-process `LlmAdapter` mock** — rejected because it bypasses fetch, HTTP status/header parsing, SSE framing, socket termination, and the adapter idle watchdog: the exact boundaries this test infrastructure exists to exercise.
|
||||
|
||||
**Change retry defaults with the server** — rejected because the server reveals existing semantics rather than deciding policy. Extending recovery to `STREAM_CLOSED` requires a separate decision with its own cost, latency, and duplicate-generation trade-offs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Developers can reproduce fault sequences by changing only provider URL/key configuration, and automated tests can keep socket-level failures deterministic through explicit scripts and seeds. The same wire fixture now exposes gaps between hard resets, clean truncation, and recovered empty completions without splicing attempts or modifying model history.
|
||||
|
||||
The server adds a support package, executable build entry, and behavior vocabulary that must remain compatible with both direct tests and CLI examples. Arrival-ordered scripts are intentionally shared across clients, random defaults are stress weights rather than operational truth, and exact connection refusal requires coordinating the client attempt with the pre-listen interval.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: 可脚本控制的 LLM（大语言模型）协议层故障服务器
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-25-scriptable-llm-wire-fault-server.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
适配器单元测试使用本地 HTTP 服务器对各类提供方故障逐一分类，重试测试则使用进程内的脚本化 `LlmAdapter` 证明已关闭步骤的恢复能力。这两个边界都无法提供可复用的服务器，以便同时运行交付版本的 HTTP 适配器、agent loop（智能体循环）和重试策略；开发者也无法仅修改现有应用的 base URL 与 API key，就让应用连接到确定性的传输故障。
|
||||
|
||||
连接遭拒、首个事件前连接被重置、未收到 `[DONE]` 即正常 EOF、合法但无内容的完成，以及输出部分内容后连接被重置，会产生不同的适配器与恢复结果。把它们统一视为普通 mock 故障，会掩盖提供方边界是否保留了这些区别，以及失败请求的分片是否确实没有进入已提交的模型历史。
|
||||
|
||||
## 决策
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` 是一个支持包（package），提供可导入的 Node HTTP 服务器和独立 CLI（命令行界面）。它接受兼容 OpenAI 的根路径和 `/v1` chat-completions 路径，校验可选的 bearer token，捕获请求，并对每个已接受请求消耗一个显式行为。脚本耗尽时快速失败；只有设置 `repeatLast` 才会重复最后一个行为。
|
||||
|
||||
请求行为覆盖 socket 重置、发送 header 后断开、发送部分内容后断开、停滞、合法空完成、正常关闭但被截断的流、畸形 payload、典型 HTTP 故障、完整的文本/推理/工具调用响应、慢速流式输出以及达到 token 上限的完成。真正的 `connection_refused` 由 CLI 的监听器生命周期阶段实现，因为已经绑定端口的请求处理器无法拒绝自身的 TCP 连接。
|
||||
|
||||
脚本项 `random` 会为每个请求重新执行一次加权选择。服务器公开并记录其无符号 32 位 seed，允许调用方提供相对权重，并内置一套偏重成功结果的压力测试配置，将传输、协议、提供方、超时和语义空结果混合在一起。该配置用于提供可调的测试压力，并非对生产事故发生频率的估算；`connection_refused` 仍不进入请求级随机池。
|
||||
|
||||
服务器只报告协议层事实，不判断是否可重试。真实组合测试让请求依次经过 `dsh-llm-deepseek`、`dsh-agent-loop` 和 `dsh-llm-retry`：在现有默认策略下，连接遭拒、硬断开、部分输出后重置、空闲超时以及合法的无内容完成均可恢复；正常关闭的部分输出 EOF 仍归类为 `STREAM_CLOSED`，默认不重试。该包不会改变这些策略。
|
||||
|
||||
## 验证
|
||||
|
||||
包测试覆盖所有请求行为、跨分片 UTF-8 请求解码、不消耗脚本的 HTTP 校验、脚本耗尽与重复、停滞连接清理、CLI 解析及延迟边界、IPv6 base URL、随机 seed 可复现性、权重校验、单结果遥测、生命周期清理，以及逐文件覆盖率门禁下的配套不变式插件。重试集成套件通过真实 HTTP/SSE（Server-Sent Events）适配器，验证准确的请求次数、带编号的重试步骤、请求体完全一致、失败的部分分片不会泄漏、语义空结果恢复、正常 EOF 分类、超时恢复、监听器延迟启动后从真实连接遭拒中恢复，以及有界重试耗尽。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**使用 Python 实现服务器**：不予采纳。Node 的标准 HTTP 与 socket API 足以暴露所有所需故障，而 TypeScript 可以让服务器、CLI 解析器、测试、包构建、lint 和覆盖率全部留在仓库现有工具链中。引入第二套运行时会增加环境与子进程依赖，却不能增强协议隔离。
|
||||
|
||||
**在适配器测试中继续使用各自独立的内联 mock 服务器**：不予采纳。这些 fixture（测试前置数据）无法作为独立服务器启动并供现有应用连接，还会让不同测试套件重复实现行为编排、随机化、遥测和连接清理。支持包让测试共享同一套实现，又不会将其提升为产品 API。
|
||||
|
||||
**仅使用进程内的 `LlmAdapter` mock**：不予采纳。它会绕过 fetch、HTTP 状态与 header 解析、SSE 分帧、socket 终止以及适配器的空闲看门狗，而这正是这套测试基础设施要覆盖的边界。
|
||||
|
||||
**随服务器一起修改默认重试策略**：不予采纳。服务器用于揭示既有语义，而非决定策略。是否将恢复能力扩展到 `STREAM_CLOSED`，需要单独决策，并权衡成本、延迟和重复生成风险。
|
||||
|
||||
## 后果
|
||||
|
||||
开发者只需修改提供方 URL/key 配置即可复现故障序列；自动化测试则可通过显式脚本和 seed，让 socket 层故障保持确定性。同一套协议 fixture 现在可以暴露硬重置、正常截断与恢复后的空完成之间的差异，而不会拼接多次尝试的内容或修改模型历史。
|
||||
|
||||
服务器新增了一个支持包、可执行构建入口和行为词汇，三者必须同时兼容直接测试与 CLI 示例。按请求到达顺序执行的脚本有意由所有客户端共享；随机模式的默认值代表压力测试权重，而非实际运行规律；精确模拟连接遭拒时，需要让客户端尝试与监听开始前的时间区间协调一致。
|
||||
Reference in New Issue
Block a user