Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

test(acp): snapshot system prompts as Markdown

Browse Source
This commit is contained in:
Tianyi Cui
2026-07-11 22:24:20 +08:00
parent ee3e4ea8f9
commit 9339622d3b
20 changed files with 543 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md
+1 -1
View File
@@ -51,7 +51,7 @@ The ACP server app loads `@deepseek-ai/dsh-llm-deepseek`, whose `apply` throws w
A snapshot run asserts **two** normalized surfaces, because the harness's external surfaces are distinct:
1. The **stdout transcript** — the framed `session/update` JSON-RPC the editor sees. Catches regressions in the ACP bridge's event→update translation (`streamSessionEventUpdate`). Compared against a committed `stdout.golden.jsonl`.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is BOTH the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against ITS OWN volatile values (the fixture's read from its header line) and the comparison is on normalized form. Request-header CONTENT (the composed system prompt + tool schemas) is additionally scrubbed to `{{system}}`/`{{tools}}` tokens on both sides — in the stored fixtures too — for every scenario except the one that pins it ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)). For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is both the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against its own volatile values (the fixture's read from its header line) and the comparison is on normalized form. Every stored JSONL additionally scrubs composed prompt text to `{{system}}`; each header class's pinning scenario stores that prompt readably in `system-prompt.golden.md` and keeps the complete tool schemas in its JSONL, while other scenarios scrub schemas to `{{tools}}` ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)). For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
The two are genuinely additive: stdout is the bridge's *lossy projection* of the log (it drops `assistant/message.usage`, `step/*`, exact `seq`/`time`, and renders tool I/O differently), so a loop/tool/turn-structure regression can change the JSONL while leaving the stdout projection identical, and a bridge-translation regression can change stdout while the JSONL is untouched. Asserting the JSONL equality also echoes the proposed [universal replay fixture](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md) idea.