deepseek-harness/.agents/notes/proposed/testing/2026-07-24-web-gui-browser-e2e-lane.md

Agent Note: Keyless browser e2e lane for the web GUI

Status: proposed

Problem

The web GUI ships as a real assembled chain — chromium page → nine client plugin bundles → HTTP unary RPC + two SSE streams → toFetchHandler/apiproxy → bootHost's agent loop, tools, and JSONL persistence — and no test exercises that chain keylessly and deterministically. The GUI testing system covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and a tier-3 smoke pair, but the keyless smoke (apps/web/tests/smoke-fixture.e2e.ts) drives FixtureApiClient behind ?fixture — no host, no wire, no agent loop — while the full-chain smoke (smoke-real.e2e.ts) needs DEEPSEEK_API_KEY and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of docs/testing.md — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless stream-json, and TUI transcript surfaces; the web surface is the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits.

Proposal

Add a keyless, deterministic browser e2e lane under apps/web/tests/, driven by recorded session-log fixtures replayed through @deepseek-ai/dsh-llm-replay, asserting the rendered accessibility tree plus in-process world state. No new package; no product-code change except (open question 1) an LLM composition knob.

Harness: apps/web/tests/harness.ts

A plain shared-fixture module (the testing-policy sanctioned shape), not a package: the gate-worthy logic this lane needs — replay derivation, session parsing, log scrubbing, persistence — already lives in gated packages (dsh-llm-replay, dsh-acp-snapshot, dsh-session-persistence-jsonl); what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners.

launchWebHarness() boots the real web assembly in-process from the exported production functions — startHost({ boot: { persistenceRoot: <tmp>, workspaceContext: false, cwd: <tmp workspace> } }), installLlmReplay(host.ctx, { file, childFiles, providers }), mountWebPlugins(host.ctx), createHostWebPluginRegistry, startWebServer({ port: 0, distIndex, apiHandler: host.handler, webPlugins }) — and returns { baseUrl, host, workspaceCwd, close }. This is the web analog of the TUI suite mounting the production bundle in-process (TUI snapshots): the real entry boundary (dsh web bin arg-parsing and dist resolution) stays held by the existing keyless CLI smokes in smoke-real.e2e.ts, and the web surface has no cordis.yml to bypass — assembly is written in the app per the GUI layering decision. Replay runs in providers-catalog mode with a contextWindow (the TUI suite's PROVIDERS shape), never catch-all: with no adapter registered, catch-all mode would make compact-basic's resolveModelContext throw into its post-step catch every step, spamming warnings and silently disabling the pressure path instead of proving it inert.

seedSession(host, fixtureText) seeds cold sessions through the real persistence API — a throwaway Context mounting SessionStore + SessionPersistenceJsonl against the host's root, create() + append(), one utimes for deterministic sidebar order (the precedent is examples/acp-agent/tests/semantic-checkpoint.snapshot.ts) — never raw file writes, so the seeder needs no knowledge of bucket hashing, filename encoding, or compression. Seeds are validated at seed time (parseable, seq-contiguous, ending in turn/end) so fixture drift fails loud at the earliest resolvable point rather than as silently dropped frames in the client; a seed not ending in turn/end would be mutated by resume's crash repair.

Determinism rules

The barrier stack, in order, for a prompted turn: (1) host-side await agent.whenIdle() under a timeout — the idle flip happens after both the turn/end append and the persistence flush, so one await covers turn completion and durability; (2) browser settled poll — streaming node detached, composer restored, final text visible; (3) log harvest only after host.dispose(). An in-process turn/end listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync), and file polling is banned (slow on NFS, superseded by whenIdle). For history-open scenarios the barrier is a poll for the last expected message's text, then a poll-until-equal aria capture. networkidle is banned outright — it never resolves while an SSE stream is open.

No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling [data-streaming] is a race by construction. Streaming incrementality is asserted from the persisted assistant/chunk events (model-visible ⟺ logged makes the log the authoritative proof), optionally corroborated by an in-page MutationObserver latch armed before send — observers cannot miss a commit; polls can. dsh-llm-replay gains an opt-in paceMs config field (default absent = today's instant yield) as a realism knob so the browser observes genuinely incremental SSE; correctness never leans on the pace.

Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Harness close() asserts every replay script was fully consumed (all scripts bound, every cursor at end), converting silent underruns and shifted bindings into crisp diagnostics; this is a small additive stats handle on installLlmReplay. No vitest retry on the lane — a retried-green race is a flake deferred, and only the chromium launch itself may retry inside the harness, logged. One chromium per file, fresh browser context per scenario, one host per scenario; viewport pinned; selectors anchor on roles, data-* attributes, and visible text only.

Expected outputs

One committed golden per scenario: a normalized ariaSnapshot() of the conversation region only (ui.expected.md) — uuid/cwd/duration tokens normalized, sidebar and other time-bearing chrome structurally excluded, captured poll-until-equal at the settled milestone. The accessibility tree is the mechanization of the client rule "assert what the user would see, never class names": it survives CSS-module hash churn, styling rewrites, and DOM restructuring, and a wholesale component rewrite refreshes it keylessly. Alongside the golden, three or four targeted role/text anchor assertions (heading level, pre code content, tool-row accessible name) keep green anchors under a semantics-preserving rewrite so a churned golden diff is reviewable against surviving anchors. World-state assertions ride host.ctx session events inline (which tools ran, turn/end completed, no error) instead of a second committed log golden: the persisted-log surface is already pinned by the ACP/headless/TUI suites through the same loop and persistence plugins, and re-pinning it here would double refresh cost for no new regression class. playwright gets pinned exactly in apps/web/package.json — the aria format is Playwright-owned, the one snapshot format in this repo we do not own, so version bumps must be deliberate bump-and-refresh commits.

Modes and fixtures

DSH_SNAPSHOT selects replay (default, keyless), record (with key), or refresh (keyless), as inline branches in the specs — the TUI suite's shape, not a suite-factory: at two scenarios the acp-snapshot factory machinery (scenario tables, pinning classes, Windows sidecars, packed-row stabilization) has no owner, and the genuinely shared parts are already exported (scrubRequestHeaders, normalizeSessionLog, parseSessionLog, installLlmReplay). Each scenario script splits into drive steps (type, send, whenIdle-generic waits — run in all modes, never waiting on model-content selectors) and interaction/assertion steps (expand reasoning, aria capture — replay/refresh only), so record mode cannot hang on a live model answering with a different tool count. Record = drive + harvest the in-memory session.header + session.events (the TUI rawSessionLog shape — no file decompression, so bootHost needs no compression knob) + scrub via scrubRequestHeaders + a mandatory keyless refresh to regenerate ui.expected.md. Web fixtures scrub headers everywhere and pin nowhere, matching the TUI precedent; whether the web surface must instead own a header-class pin per the pinned-header discipline is open question 3. Seeds are recorded fixtures under the same inventory and refresh discipline as replay fixtures, never hand-authored one-offs, so DSH_SNAPSHOT=refresh heals every committed surface after intentional shape churn and only assistant/chunk-shape churn escalates to re-record. A TUI-style afterAll fixture guard holds the inventory closed (expected files present, every fixture scrub-fixed-point, no orphan directories).

Demo scenarios

1. fresh-round-trip — new session, prompt, replay streams reasoning + markdown + a bash tool call that really executes (echo in the temp workspace) + final text. Asserts settled markdown semantics (heading, code block), the tool card row, composer restore, the aria golden, and inline world state (bash tool/call + completed turn/end in the session events). The keyless version of the with-key W5 flows.
2. seeded-history — a recorded session seeded cold; the sidebar lists it, opening it renders tool cards and collapsible reasoning purely from the log. This exercises the implicit cold-resume attach (session.history resumes via agentFor), cold summaries, history pagination views, and the client fold of historical events — the surface nothing else covers — with zero model calls, so no replay-binding constraints at all. A follow-up-prompt-after-resume scenario is deliberately deferred until the history/live stitch path changes or regresses.

Lane wiring and CI stance

The lane rides vitest.web.config.ts (pnpm run test:web, serial), which stays gate-exempt exactly as its header comment records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise recorded in the GUI testing system note and therefore requires its own Agent Note cross-linked from that note, staged as: non-required CI job first, promotion criteria measured (consecutive green runs, wall time, zero-retry flake budget, browser cache strategy on the enterprise runners, whether the runner images carry the chromium system libraries). Deferred out of this proposal; a TODO(ci-browser) marks the seam. Scenarios are posixOnly initially. Docs updated in the same implementation PR: the testing policy names apps/web/tests/snapshots/ as the web surface's snapshot home with its divergent DSH_SNAPSHOT=… pnpm run test:web commands, the GUI testing note's tier map gains the lane (and drops its stale references to the deleted missions/scripts/verify-* files), packages/client/AGENTS.md's check ladder mentions it, and the dsh-acp-snapshot README's "ACP-specific by design" sentence is corrected — this lane is the third consumer of its normalizers.

Open questions

1. LLM seam. Two viable shapes. (a) BootHostOptions.llm?: 'deepseek' | false — an assembly toggle in the one module that owns assembly, matching the existing workspaceContext: Config | false shape and the reserved-knob sentence in start.ts; llm: false mounts no adapter, the harness fills the open seam on host.ctx, misuse fails loud at the first stream with NO_ADAPTER, and keyless boots stop needing any key. (b) Zero product change — a placeholder DEEPSEEK_API_KEY env var satisfies llm-deepseek's load-time presence check (twice-precedented in-tree) and replay intercepts ahead of the mounted adapter. (a) is cleaner semantics and honest keylessness at the cost of a test-motivated product field; (b) is free but satisfies a fail-loud check with a lie and leaves a dead adapter mounted. Recommendation: (a), shaped minimal.
2. Loader-izing dsh web. Making the web host cordis.yml-driven like every example would give the ACP-style cordis.snapshot.yml replay overlay for free and align with "everything is a plugin", but it reverses the settled "assembly is written in the app" ruling and is a product-architecture decision on its own merits — its own proposal if wanted; this lane does not need it.
3. Header-class pin. Strict reading of the pinned-header discipline wants one web scenario pinning bootHost's composed prompt + tool schemas (a header class no ACP scenario covers); the TUI precedent scrubs everywhere and pins nowhere. Cheap middle: pin sidecars on fresh-round-trip at record time. Recommendation: follow TUI now (scrub-only), revisit when the web assembly's header diverges further from the repl composition it mirrors.
4. Golden breadth. Full conversation-region aria golden (adopted above) versus targeted assertions only. The golden is the "assembled transcript" duty for user-visible changes; the cost is a keyless refresh on every component rewrite. Recommendation: keep the golden + anchors.
5. Client settled signal. A data-dsh-busy attribute derived from the object layer's pending-RPC/active-stream state would replace multi-condition settled polls with one selector. Presentation-plane observability, no session-log leak — but the current polls suffice for two scenarios. Recommendation: defer until a settled-poll flake actually appears.

Prior art

Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's LIBRECHAT_TEST_RUN_HOOK fake model; ai-chatbot's MockLanguageModelV3 + simulateReadableStream; continue's scripted mock provider classes) — which is what dsh-llm-replay already is. Browser-level SSE interception cannot exercise incremental rendering (route.fulfill delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural.

Alternatives considered

Browser-network SSE interception (page.route). Rejected: route.fulfill cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested.

Mock HTTP provider at DEEPSEEK_BASE_URL. Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job.

Growing the ?fixture client. Rejected: tier separation — FixtureApiClient exists to test the client shell without a server; everything below the client API seam stays untested by construction.

A packages/support/web-snapshot package with a defineWebSnapshotSuite factory. Rejected for now: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios the factory generalizes from one consumer while the genuinely shared logic already lives exported in dsh-llm-replay/dsh-acp-snapshot. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.

A committed normalized-session-log golden as a second expected surface. Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers, against the tier discipline. Inline world-state assertions on host.ctx events keep the world-verification duty.

Spawning the dsh web bin with a DSH_SNAPSHOT replay branch. Rejected for now: it needs a test-mode branch plus env plumbing in the product bin where the in-process route uses exported production functions with zero product change; the bin's thin glue is covered by the keyless CLI smokes. Becomes free if the web host is ever Loader-ized (open question 2).

Changing the wire protocol for testability. Rejected: the contract already has a first-class keyless isomorphic seam (InProcessApiClient(toFetchHandler(api))), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose.

Real-model browser tests as the keyless lane. Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.

Acceptance criteria

• pnpm run test:web keyless passes the two scenarios deterministically (no vitest retry), alongside the existing smoke pair, on a checkout with built client bundles and the frontend dist.
• Replay asserts: aria golden equality at the settled milestone, anchor role/text assertions, inline world-state event assertions, zero pageerrors, zero connection-loss/gap-repair console warnings, all replay scripts fully consumed at teardown.
• DSH_SNAPSHOT=record with a key re-records fresh-round-trip (drive steps only), rewrites its session.jsonl scrubbed, and a follow-up DSH_SNAPSHOT=refresh regenerates ui.expected.md keylessly; refresh alone heals goldens after intentional non-chunk shape churn.
• The seeded scenario renders history through the real cold-resume path with zero model calls and leaves the seed fixture byte-identical (closedness validated at seed time).
• Fixture guard holds the snapshot inventory closed; failure produces a bundle under .artifacts/ (screenshot, console, pageerrors, persistence copy, actual-vs-expected aria).
• Docs land in the same PR: testing.md web-lane entry, GUI testing note tier map + stale verify-script cleanup, client AGENTS.md ladder, acp-snapshot README correction, this note moved to implemented/ rewritten in present tense.

Risks

• The aria format is Playwright-owned — the one committed snapshot format the repo does not control; a version bump can churn every golden. Mitigated by an exact version pin in apps/web and a documented bump-and-refresh procedure; residual risk accepted.
• Replay's first-call-order binding stays fragile under concurrent browser-driven sessions; the lane constrains scenarios to one prompting session each (the seeded scenario prompts none), and the teardown consumption assertion turns violations into diagnostics rather than surreal transcripts.
• compact-basic shares the session's replay cursor — a pressure-triggered summarize would consume a script entry; inert for small fixtures under the 128k catalog window, and the consumption assertion catches it if a fixture ever grows past the threshold.
• CI remains browserless for now, so the lane guards regressions only where it is run (locally and in any future non-required job) until the CI reversal is separately decided; the runner images' chromium-library situation is unverified.
• Record-mode nondeterminism is contained but not eliminated by the drive/assert split: a live model may still produce a transcript whose replay violates a scenario's assertions, requiring prompt tuning at record time (bounded by terse prompts and a chunk-count warning in record mode).
• jsdom-lane overlap: component-level rendering is already covered per-plugin; this lane must stay at assembled-transcript altitude (whole-region golden + anchors) or it starts re-testing tier 2 and paying double maintenance.