refactor(acp-example): derive llm-replay script from the session JSONL

Per a design revision, the per-scenario snapshot fixture becomes EXACTLY the
persisted session JSONL (<scenario>/session.jsonl) rather than a hand-authored
llm.json. The log already holds all LLM behavior (assistant/chunk carries every
StreamChunk) AND all harness behavior (tool/call, tool/result, turn/*, usage),
so one artifact drives replay and doubles as a behavioral golden.

llm-replay becomes replay-only (the record-tee is removed; recording is now
"run the real agent once and harvest the .jsonl", done by the harness in a
later commit). deriveReplayScript(events) groups assistant/chunk by (turn,step)
in log order — exact because the loop makes one ctx.llm.stream() call per step
and tags each chunk with the current (turn,step). The two failure modes the log
can't express (a thrown stream — no terminal finish; cancel/hang — timing) use
an optional replay.override.json sidecar.

Hardens against a Codex review finding: a derived group is only valid if it
ends in a `finish` chunk. A group without one is the fingerprint of a thrown
stream() and is NOT silently replayed as a clean stop — deriveReplayScript
throws, naming the (turn,step), so a missing sidecar override fails loud.

Updates the unit tests (parse/derive/load helpers, sidecar override, finish-
terminated grouping, HMR), the example README, and the RFC prose to the JSONL
format. Two goldens (stdout transcript + re-persisted JSONL) and the harness
wiring land in the next commit.
This commit is contained in:
Tianyi Cui
2026-06-19 02:44:33 +08:00
parent 1a1ce734ba
commit c182543dd5
4 changed files with 352 additions and 262 deletions
+1 -1
View File
@@ -32,7 +32,7 @@ The editor sets each session's `cwd` to the project it opens; the agent's bash t
## Snapshot tests (record-once / replay-deterministic)
This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized stdout transcript against a committed golden file. The model is made deterministic by `src/llm-replay.ts`, a function/namespace plugin that installs an `llm/stream` waterfall listener: in `record` mode it tees the real model's `StreamChunk`s into a per-scenario `llm.json` (flushed atomically after each call); in `replay` mode it short-circuits the waterfall and serves those chunks back, so replay needs no API key. Each fixture entry is a discriminated record — `{ kind: 'chunks' | 'throw' | 'hang' }` — so both LLM failure branches (throw vs. finish-error) and cancellation replay faithfully. See [docs/rfc/implemented/2026-06-19-acp-snapshot-tests.md](../../docs/rfc/implemented/2026-06-19-acp-snapshot-tests.md) for the full design.
This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `src/llm-replay.ts`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`<scenario>/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`". The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `<scenario>/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). See [docs/rfc/implemented/2026-06-19-acp-snapshot-tests.md](../../docs/rfc/implemented/2026-06-19-acp-snapshot-tests.md) for the full design.
## MVP limitations