Tianyi Cui
4209e4af3f
Model-driving ACP snapshot scenarios shipped both session.jsonl (the replay fixture) and session.golden.jsonl (the expected re-persisted log). For recorded scenarios the normalized fixture and golden were byte-identical — pure duplication. Remove session.golden.jsonl entirely: every model scenario now has at most one committed session-log artifact, session.jsonl, which doubles as the replay source AND the expected produced log. The snapshot test compares the replay run's persisted log against the session.jsonl fixture, normalizing BOTH sides — but each against its OWN volatile values, not a shared context. A raw harvested fixture bakes in the recording run's session id / cwd / timestamps, distinct from the live replay run's; since normalizeSessionLog scrubs cwd by exact string match, the fixture must be normalized against its own header (new fixtureContext helper) or its stale recorded cwd would leak unscrubbed and the compare would fail. The session side uses a normalized-string toEqual, NOT toMatchFileSnapshot, so a run never overwrites the fixture. Authored override scenarios (error-finish, cancel) now hold their expected produced log in session.jsonl. Verified llm-replay ignores the fixture for model chunks when an override exists: loadReplayScript() returns the override array and never reads config.file, so committing the full expected log there does not affect replay behavior. The required-fixture guard is now per-kind: every scenario needs input.json + stdout.golden.jsonl; model scenarios need session.jsonl; authored ones additionally need replay.override.json. Updates the ACP-snapshot-tests RFC to the reduced fixture set and moves the proposing RFC proposed -> implemented.
36 lines
4.1 KiB
Markdown
36 lines
4.1 KiB
Markdown
# RFC: Use `session.jsonl` as the only snapshot session-log artifact
|
|
|
|
Status: implemented (proposed and accepted 2026-06-20)
|
|
|
|
## Problem
|
|
|
|
Model-driving ACP snapshot scenarios ship both `session.jsonl` and `session.golden.jsonl`. For normal recorded scenarios, `session.jsonl` is the replay fixture harvested from a real run, and the replay test normalizes the newly persisted log and compares it to `session.golden.jsonl`. In the current fixtures, the normalized recorded log and normalized golden are identical for the ordinary recorded scenarios.
|
|
|
|
Authored override scenarios (`error-finish`, `cancel`) currently use `replay.override.json` to drive model behavior and keep `session.jsonl` as a minimal dummy fixture, while `session.golden.jsonl` holds the expected persisted log. The override file is a JSON array of `ReplayEntry` objects: `{ "kind": "chunks", "chunks": StreamChunk[] }`, `{ "kind": "throw", "chunks": StreamChunk[], "message": string, "code": string, "status"?: number }`, or `{ "kind": "hang" }`. That split is also unnecessary: when an override sidecar exists, `llm-replay` replaces the derived script and does not need `session.jsonl` for model chunks, so `session.jsonl` can still be the expected session-log artifact for the scenario.
|
|
|
|
## Proposal
|
|
|
|
Remove the `session.golden.jsonl` concept entirely. Every scenario has at most one committed session-log artifact, `session.jsonl`:
|
|
|
|
- For recorded scenarios, `session.jsonl` remains the raw harvested log. Replay still derives model chunks from it, and the snapshot test compares the replay run's normalized persisted log against normalized `session.jsonl`.
|
|
- For authored override scenarios, `replay.override.json` drives model behavior and `session.jsonl` holds the expected produced session log. The replay adapter ignores the fixture for model chunks when the override exists, so the same file can be the expected log without affecting replay behavior.
|
|
- For no-model scenarios, `session.jsonl` can stay as the minimal fixture needed to boot `llm-replay`; no session-log comparison is needed unless the scenario creates a persisted session.
|
|
|
|
Stdout goldens remain unchanged; they are the editor-facing projection and are not redundant with the session fixture.
|
|
|
|
## Acceptance criteria
|
|
|
|
- `session.golden.jsonl` disappears from the snapshot harness, fixtures, orphan guards, and docs.
|
|
- The snapshot test derives the expected session log from `session.jsonl` for every model scenario.
|
|
- Authored sidecar scenarios commit their expected produced log in `session.jsonl`; `replay.override.json` remains the model-behavior override.
|
|
- Orphan-fixture guards understand which files are required by scenario kind.
|
|
- The [ACP snapshot tests RFC](../../implemented/testing/2026-06-19-acp-snapshot-tests.md) is updated to describe the reduced fixture set.
|
|
|
|
## What we give up
|
|
|
|
Reviewers lose one artifact name that made the expected persisted log visually separate from the replay fixture. The stdout golden still protects the editor transcript, and comparing replay output to `session.jsonl` preserves the loop/persistence regression check without duplicating files.
|
|
|
|
## Implementation note
|
|
|
|
The comparison normalizes BOTH sides, but each against its OWN volatile values, not a shared context. A raw harvested `session.jsonl` bakes in the recording run's session id, cwd, and timestamps; the replay run produces fresh ones. `normalizeSessionLog` scrubs cwd by exact string match, so normalizing the fixture against the *replay* run's cwd would leave the recorded cwd in the header unscrubbed and the compare would fail. The harness therefore derives the fixture's normalize context from its OWN header line (`{ type:'session', id, cwd }`) — `fixtureContext()` in `acp.snapshot.ts` — so both sides scrub to the same `{{sessionId}}`/`{{cwd}}` tokens. An authored fixture copied from the old golden already carries the normalized header (`id:'{{sessionId}}'`, `cwd:'{{cwd}}'`), which yields those tokens as the volatile values and scrubs idempotently. The session-log side uses a plain normalized-string `toEqual`, NOT `toMatchFileSnapshot`, so a run never overwrites the fixture.
|