Merge branch 'codex/simp-session-log-representation' into codex/simp-snapshot-fixture-inventory
# Conflicts: # docs/rfc/proposed/process/2026-06-20-discover-package-inventory.md # examples/acp-agent/tests/acp.snapshot.ts
This commit is contained in:
@@ -37,9 +37,9 @@ defineAcpSnapshotSuite({
|
||||
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template. Each pinning directory's generated `system-prompt.golden.md` is the reviewable snapshot of the normalized composed prompt; `session.jsonl` stores `"system":"{{system}}"` while retaining the complete tool list. A pin whose scenario legitimately changes its header mid-run declares `expectedHeaderChanges`; the Markdown snapshot then records each later full prompt under a `request/header change` marker.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's Markdown prompt snapshot from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, and prompt snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
|
||||
`suite.ts` imports Vitest, so use this package only inside a Vitest run. The ACP-specific script queues permission answers by stable option kind and maps them to current option ids; a missing answer cancels, while an unavailable kind fails the scenario after cancelling the agent request. It can also set session config options or assert that unknown ids and values are rejected in the transcript.
|
||||
|
||||
## Model Experience
|
||||
|
||||
|
||||
@@ -1,18 +1,8 @@
|
||||
/**
|
||||
* Shared subprocess harness for ACP snapshot suites. A library module driven by
|
||||
* the suite factory in ./suite.ts (and directly by harness-level specs); each
|
||||
* example's `*.snapshot.ts` names its own agent-under-test paths.
|
||||
*
|
||||
* It boots the REAL agent bin subprocess via the cordis Loader (so the
|
||||
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
|
||||
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
|
||||
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
|
||||
* and — in record mode — harvests the persisted session JSONL after a graceful
|
||||
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
|
||||
* stdout frames and the session-log events into stable, snapshot-able text.
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* Shared ACP snapshot subprocess harness. It boots the real agent bin through the Cordis
|
||||
* loader, drives deterministic ACP JSON-RPC over stdio, captures protocol-pure stdout, and
|
||||
* harvests persisted session logs after graceful shutdown. Normalization stays in
|
||||
* `normalize.ts`; suite registration stays in `suite.ts`.
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/harness
|
||||
*/
|
||||
|
||||
@@ -66,16 +56,10 @@ export interface AgentUnderTest {
|
||||
}
|
||||
|
||||
/**
|
||||
* One step of a scenario's deterministic input script (`input.json`). The
|
||||
* harness interprets these in order. `newSession` captures the server-issued
|
||||
* (random) session id into a `{{sessionId}}` variable that later steps
|
||||
* reference, since a committed file cannot know the id in advance.
|
||||
*
|
||||
* `promptAndCancel` sends a prompt WITHOUT awaiting its response, waits until
|
||||
* the client observes the first streamed `agent_message_chunk` (so the emitted
|
||||
* frames deterministically precede the cancellation), then cancels the turn —
|
||||
* the only way to exercise a cancel deterministically (a plain `prompt` step
|
||||
* awaits the response, which a cancel/hang scenario would block on forever).
|
||||
* One step of a scenario's deterministic input script (`input.json`). The harness interprets
|
||||
* these in order. `newSession` captures the server-issued (random) session id into a
|
||||
* `{{sessionId}}` variable that later steps reference. `promptAndCancel` sends without awaiting,
|
||||
* waits for the first streamed message, then cancels, making transcript order deterministic.
|
||||
*/
|
||||
export type InputStep =
|
||||
| { op: 'initialize'; terminalOutput?: boolean }
|
||||
@@ -92,16 +76,9 @@ export type InputStep =
|
||||
export interface InputScript {
|
||||
steps: InputStep[]
|
||||
/**
|
||||
* Ordered answers for the agent's `session/request_permission` round-trips,
|
||||
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
|
||||
* by option KIND: option ids are agent-issued randoms a committed script
|
||||
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
|
||||
* kind → the offered `optionId` at answer time. A request beyond the queue
|
||||
* (or with no queue at all) is answered `cancelled` — the stub behavior a
|
||||
* scenario without approvals relies on. A scripted kind the request does
|
||||
* not offer REJECTS the run: the scenario scripted an impossible click,
|
||||
* and {@link runScenario} throws once the in-flight step settles (the
|
||||
* agent itself just sees `cancelled`, so it cannot absorb the bug).
|
||||
* FIFO permission answers selected by stable option kind; the harness maps each kind to the
|
||||
* agent-issued option id. Exhaustion cancels, while a kind the agent did not offer fails the
|
||||
* scenario.
|
||||
*/
|
||||
permissionAnswers?: PermissionAnswer[]
|
||||
}
|
||||
@@ -201,8 +178,6 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
const stderrChunks: string[] = []
|
||||
try {
|
||||
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
|
||||
// Copied into the temp cwd so the agent's bash tools see it; the goldens
|
||||
// normalize the cwd, so the seeded paths stay stable across runs.
|
||||
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
|
||||
await cp(opts.workspaceDir, cwd, { recursive: true })
|
||||
}
|
||||
@@ -229,10 +204,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
child.stderr.setEncoding('utf8')
|
||||
child.stderr.on('data', (c: string) => stderrChunks.push(c))
|
||||
|
||||
// Tee raw stdout: accumulate the bytes for the golden + purity check, and ALSO
|
||||
// feed the same bytes to the SDK client through a passthrough. Buffer the raw
|
||||
// bytes (not per-chunk utf8 strings) and decode once at the end, so a
|
||||
// multibyte sequence split across two 'data' events can't corrupt the golden.
|
||||
// Tee the same raw bytes to the golden and SDK client. Decode once at the end so a UTF-8
|
||||
// sequence split across stream chunks cannot corrupt the transcript.
|
||||
const passthrough = new Readable({ read() {} })
|
||||
child.stdout.on('data', (buf: Buffer) => {
|
||||
rawBuffers.push(buf)
|
||||
@@ -255,13 +228,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
// Permission answers are consumed FIFO across the whole run; exhaustion
|
||||
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
|
||||
const permissionQueue = [...input.permissionAnswers ?? []]
|
||||
// A scenario bug detected inside a client callback (a scripted permission
|
||||
// kind the agent never offered). It cannot fail the run from in there: a
|
||||
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
|
||||
// a tolerant agent treats that as a denial and carries on — the run (or
|
||||
// worse, a record) would absorb the impossible click silently. So the
|
||||
// callback answers `cancelled` (a well-defined path for the agent),
|
||||
// captures the error here, and the step loop fails the run on it.
|
||||
// A callback throw would become only an RPC error the agent could absorb. Record an
|
||||
// impossible permission choice here, answer cancelled, and fail the outer scenario.
|
||||
let scriptError: Error | undefined
|
||||
const makeClient = (_agent: AcpAgent): Client => ({
|
||||
sessionUpdate(params: SessionNotification): Promise<void> {
|
||||
@@ -356,10 +324,8 @@ async function runStep(
|
||||
return
|
||||
}
|
||||
case 'newSessionExpectError': {
|
||||
// The bridge rejects a session/new that widens the workspace scope
|
||||
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
|
||||
// surfaces that as a rejected RPC; swallow it so the run completes and the
|
||||
// error frame is captured in the transcript.
|
||||
// The bridge rejects a session/new that widens the workspace scope (non-empty
|
||||
// additionalDirectories / mcpServers — unimplemented).
|
||||
await client.newSession({
|
||||
cwd,
|
||||
mcpServers: [],
|
||||
@@ -379,10 +345,8 @@ async function runStep(
|
||||
case 'promptExpectError': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
|
||||
// The model fails this turn (a recorded provider error), so the bridge
|
||||
// answers the prompt with a JSON-RPC error and the SDK rejects. That
|
||||
// rejection IS the expected editor experience — swallow it so the run
|
||||
// completes and the stdout transcript (the error frame) is captured.
|
||||
// The model fails this turn (a recorded provider error), so the bridge answers the prompt
|
||||
// with a JSON-RPC error and the SDK rejects.
|
||||
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
|
||||
() => { /* expected: the turn failed and the bridge returned an error */ })
|
||||
@@ -391,13 +355,8 @@ async function runStep(
|
||||
case 'promptAndCancel': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
|
||||
// Dispatch the prompt WITHOUT awaiting (a hang fixture never resolves on
|
||||
// its own). To pin frame order deterministically, wait until the client
|
||||
// has OBSERVED the hang's streamed agent_message_chunk before cancelling —
|
||||
// so those update frames always precede the cancelled prompt response in
|
||||
// the transcript (without this, the late chunk and the response race).
|
||||
// Then cancel and await the prompt, which the bridge settles as
|
||||
// `cancelled` once the abort propagates.
|
||||
// A hang fixture never resolves alone. Wait for its streamed chunk before cancellation
|
||||
// so updates deterministically precede the cancelled prompt response.
|
||||
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
|
||||
await client.cancel({ sessionId })
|
||||
@@ -483,14 +442,8 @@ async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
|
||||
})
|
||||
}
|
||||
}
|
||||
// Primary (no parentSession) first, then children by ascending createdAt. A
|
||||
// scenario has exactly one top-level session. In the synchronous cut sibling
|
||||
// children are created strictly sequentially, so their createdAt values are
|
||||
// strictly ordered; the recordedId tiebreak only keeps a degenerate
|
||||
// same-millisecond collision (unreachable here) deterministic. This harvest
|
||||
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
|
||||
// so session.<n>.jsonl maps to the same child on record and replay — replay
|
||||
// re-sorts childFiles by the same key, so the two stay consistent.
|
||||
// Match replay fixture assignment: primary first, then children by creation time, with id as
|
||||
// a deterministic collision tiebreaker.
|
||||
logs.sort((a, b) => {
|
||||
const ap = a.parentSession === undefined ? 0 : 1
|
||||
const bp = b.parentSession === undefined ? 0 : 1
|
||||
|
||||
@@ -1,17 +1,7 @@
|
||||
/**
|
||||
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
|
||||
* tier (`pnpm run test:snapshot`). Three layers, composable per example:
|
||||
* the subprocess scenario harness ({@link runScenario}), the pure golden
|
||||
* normalizers ({@link normalizeStdout} / {@link normalizeSessionLog} /
|
||||
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite factory
|
||||
* ({@link defineAcpSnapshotSuite}) that registers a scenario table as a full
|
||||
* describe/it tree. An example's `*.snapshot.ts` supplies only its
|
||||
* {@link AgentUnderTest} paths, its snapshots directory, and its
|
||||
* {@link Scenario} table.
|
||||
*
|
||||
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
|
||||
* vitest run — a support-tier constraint stated in the README.
|
||||
*
|
||||
* ACP snapshot suite kit: subprocess scenario harness, pure golden normalizers, and the Vitest
|
||||
* suite factory behind `pnpm run test:snapshot`. Because this entry exports `suite.ts`, importing
|
||||
* it requires a Vitest run.
|
||||
* @module @deepseek-ai/dsh-acp-snapshot
|
||||
*/
|
||||
|
||||
|
||||
@@ -1,29 +1,8 @@
|
||||
/**
|
||||
* Pure normalizers for the ACP snapshot goldens. They replace the
|
||||
* non-deterministic values in the two captured surfaces — the stdout JSON-RPC
|
||||
* transcript and the persisted session JSONL — with stable tokens, so a golden
|
||||
* compare reflects behavior, not run-to-run noise. Kept dependency-free and
|
||||
* side-effect-free so they unit-test trivially.
|
||||
*
|
||||
* Scrubbed: `randomUUID()` session ids → `{{sessionId}}`; the temp `mkdtemp`
|
||||
* cwd → `{{cwd}}` (it appears in terminal-card `_meta` and the log header);
|
||||
* JSON-RPC request `id` → a stable per-transcript sequence; the log's per-event
|
||||
* `time` (epoch ms) and header `createdAt` → 0; a `hook/result` event's
|
||||
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
|
||||
* (deterministic — `seq = log.length`, part of the event-log contract).
|
||||
*
|
||||
* Separate, composable normalizers keep bulky request-header content out of
|
||||
* session fixtures. {@link scrubSystemPrompts} replaces the composed system
|
||||
* prompt in EVERY fixture; {@link scrubRequestHeaders} additionally replaces
|
||||
* tool schemas and the session prefix outside each suite's header-pinning
|
||||
* scenario. They are deliberately NOT folded into
|
||||
* {@link normalizeSessionLog}: the suite factory composes the right scrub for
|
||||
* each scenario and snapshots the pin's actual prompt as Markdown (see the
|
||||
* pinned-header RFC,
|
||||
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* Pure ACP transcript and session-log normalizers. They scrub session ids, temp cwd, RPC ids,
|
||||
* timestamps, and hook duration while preserving deterministic event sequence numbers.
|
||||
* Request-header scrubbers stay separate so one scenario per header class can pin tools and a
|
||||
* readable prompt while other fixtures omit duplicated header bulk.
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/normalize
|
||||
*/
|
||||
|
||||
@@ -68,12 +47,10 @@ function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a
|
||||
* stable golden in the SAME shape as the wire: one compact JSON frame per line
|
||||
* (NDJSON), with the JSON-RPC `id` rewritten to a per-transcript sequence
|
||||
* (1, 2, 3, …) and all volatile strings scrubbed. Throws if any non-empty line
|
||||
* is not valid JSON — that doubles as the stdout-purity check (no logger leaked
|
||||
* onto the protocol).
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable golden
|
||||
* in the same shape as the wire: one compact JSON frame per line (NDJSON), with the JSON-RPC
|
||||
* `id` rewritten to a per-transcript sequence (1, 2, 3, …) and all volatile strings scrubbed.
|
||||
* Invalid JSON throws, doubling as a protocol-stdout purity check.
|
||||
*
|
||||
* @param rawStdout The captured stdout bytes, decoded utf8.
|
||||
* @param ctx The run's volatile values to scrub.
|
||||
|
||||
@@ -1,19 +1,7 @@
|
||||
/**
|
||||
* Scripted fake ACP agent bin for `dsh-acp-snapshot`'s unit specs. Speaks
|
||||
* newline-delimited JSON-RPC on stdio like the real `dsh-acp-agent` bin, but
|
||||
* every behavior — how prompts settle, whether session/new rejects, which
|
||||
* session logs get persisted, what filesystem noise to leave — comes from a
|
||||
* `behavior.json` sitting NEXT to the `$DSH_SNAPSHOT_FILE` fixture, so a spec
|
||||
* scripts a whole subprocess run from data. The specs launch it through the
|
||||
* REAL `runScenario` spawn path (tsx loader, temp cwd, env plumbing), so the
|
||||
* harness plumbing is exercised for real; only the agent behind the protocol
|
||||
* is scripted.
|
||||
*
|
||||
* The specs (not the golden tier) own this bin: it asserts nothing, echoes
|
||||
* observable facts into `session/update` text chunks (env probe, permission
|
||||
* outcome, seeded-workspace listing) for the spec to read off `rawStdout`, and
|
||||
* exits 0 on stdin EOF after writing the scripted logs — mirroring the real
|
||||
* bin's dispose-flush-exit shape.
|
||||
* Scripted ACP agent for snapshot-kit tests. A fixture-adjacent `behavior.json` controls the
|
||||
* subprocess reached through the real harness path; the bin reports observations over ACP and
|
||||
* writes scripted logs before exiting on stdin EOF.
|
||||
*/
|
||||
|
||||
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
|
||||
@@ -308,11 +308,8 @@ describe('runScenario', () => {
|
||||
|
||||
it('rejects the run on a scripted permission kind the agent never offered', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ permissionProbe: true })
|
||||
// The fake bin offers allow_once/reject_once; scripting allow_always is a
|
||||
// scenario bug. The agent is answered `cancelled` (it must not be able to
|
||||
// absorb the bug as an error-means-denial), and the RUN fails: a callback
|
||||
// throw would only reach the agent as a JSON-RPC error response, letting
|
||||
// a tolerant agent carry on and the scenario pass — or record.
|
||||
// The fake offers only allow_once/reject_once. The harness must reject an impossible click,
|
||||
// not merely send an RPC error that a tolerant agent could absorb.
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'impossible click' }], permissionAnswers: [{ kind: 'allow_always' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
|
||||
@@ -17,21 +17,14 @@ import {
|
||||
} from '../src/suite.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over
|
||||
* the scripted fake ACP bin (./fixtures/fake-acp-agent.ts) register REAL
|
||||
* describe/it trees at collection time, so every factory path — golden and log
|
||||
* compares, the per-suite header pin and its uniformity guard, record-mode
|
||||
* fixture write-back, skip semantics, and the fixture guard block — executes
|
||||
* as an ordinary green test. The pure helpers get direct cases below.
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over the scripted fake
|
||||
* ACP bin (./fixtures/fake-acp-agent.ts) register real describe/it trees at collection time,
|
||||
* so every factory path — golden and log compares, the per-suite header pin and its uniformity
|
||||
* guard, record-mode fixture write-back, skip semantics, and the fixture guard block —
|
||||
* executes as an ordinary green test.
|
||||
*
|
||||
* The replay suite runs against the committed fixtures in ./fixtures/suite.
|
||||
* The record suite runs against a TEMP COPY of ./fixtures/record-suite
|
||||
* (record mode writes session fixtures back into its snapshots dir; a run must
|
||||
* never touch the committed tree). To re-bootstrap the record tree's goldens
|
||||
* after changing the fake bin's output, run this spec once with
|
||||
* `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` (points the record suite at the committed
|
||||
* tree so vitest creates/updates the goldens and the write-back lands there),
|
||||
* then commit the result.
|
||||
* Record tests use a temp copy. To intentionally rebuild their committed fixtures, run this
|
||||
* spec once with `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1`, then review and commit the resulting tree.
|
||||
*/
|
||||
|
||||
const AGENT = {
|
||||
@@ -43,12 +36,7 @@ const AGENT = {
|
||||
const REPLAY_DIR = fileURLToPath(new URL('./fixtures/suite', import.meta.url))
|
||||
const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.url))
|
||||
|
||||
// The replay suite doubles as the header-CLASS coverage: every scenario names
|
||||
// the same explicit class (the record suite exercises the 'default' fallback),
|
||||
// and plain-turn boots through a per-scenario configPath override (the same
|
||||
// dummy path the agent default carries — the plumbing, not the composition,
|
||||
// is what this suite can exercise; the real overlay boot is the acp-agent
|
||||
// example's code-mode scenarios).
|
||||
// Replay pins explicit header classes; recording covers the default fallback.
|
||||
const REPLAY_SCENARIOS: Scenario[] = [
|
||||
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderChanges: 1, headerClass: 'main' },
|
||||
{ name: 'plain-turn', hasModelTurn: true, recorded: true, headerClass: 'main', configPath: AGENT.configPath },
|
||||
|
||||
Reference in New Issue
Block a user