Merge remote-tracking branch 'origin/master' into codex/ask-user-question
# Conflicts: # docs/module-graph.md
This commit is contained in:
Yichen Jiang
@@ -0,0 +1,73 @@
|
||||
# RFC: Repeat-tool-call guard plugin
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
A model stuck in a loop re-issues the same tool call with byte-identical arguments — re-running a failing grep, re-reading an unchanged file, polling a command that already gave its answer — and each round trip burns tokens, wall-clock, and (for paid APIs) money without adding information. The harness has nothing that notices: the loop has no step budget, no plugin tracks call repetition, and the model only escapes when it happens to vary its own behavior. The failure mode is real and cheap to detect — [pi-repeat-tool-guard](https://github.com/Kingwl/pi-repeat-tool-guard) ships exactly this as a pi coding-agent extension: count consecutive identical calls and, past a threshold, append a `<system-reminder>` telling the model to stop repeating itself and change course.
|
||||
|
||||
The harness already has every seam the pi extension uses, and better ones: [the interception-seams RFC](2026-06-30-interception-seams.md) gives `tools/post-execute` a sanctioned way to attach model-facing context to a finished call, the loop buffers and injects that context with call/result adjacency preserved, and injected context is a logged `context/message` — so a native guard satisfies the model-visible ⟺ logged rule with no new session event. What was missing was only the plugin itself.
|
||||
|
||||
## Decision
|
||||
|
||||
The guard is a loop-hygiene plugin, not a model-facing tool: it never appears in the tool list, never vetoes or rewrites a call, and adds exactly one behavior — it watches each agent's stream of tool calls, counts runs of consecutive calls to the same tool with identical canonicalized arguments, and at configured run lengths injects an escalating advisory reminder telling the model to stop repeating itself, re-read the last result, and either change approach or conclude. The purpose is to break unproductive loops within a few wasted steps instead of letting them run to the turn's natural end — while leaving the decision (retry differently, gather more evidence, or finish) entirely with the model, so a legitimately repeated call is delayed by nothing and blocked by nothing.
|
||||
|
||||
The plugin is `@deepseek-ai/dsh-repeat-tool-guard` at `packages/guard/repeat-tool-guard/`, opening the `guard/` group for loop-hygiene plugins (single-package groups have precedent: [the todo-write RFC](2026-06-29-todo-write-tool.md) shipped `todo/tool-todo`). It registers three listeners and holds all state in plugin-local maps keyed by `AgentId` — the tool registry is a context-level singleton whose waterfalls interleave every agent's calls (subagents run on the same context), so per-agent keying is correctness, not polish.
|
||||
|
||||
- **`tools/post-execute` (waterfall)** — the one detection point. The listener receives `(exec, result)` together, so counting and reminder delivery need no cross-event pending map (the pi extension needs one only because its `tool_call`/`tool_result` hooks are separate events). It always delegates via `next()` and, when a threshold is hit, folds a reminder onto the downstream decision's `additionalContext` — the observe-and-enrich posture [the hooks bridges](2026-06-30-hook-bridges.md) already use, honoring the waterfall contract. Counting happens here rather than in `tools/pre-execute` because post-execute also runs for denied calls (`ToolRegistry.execute` routes a deny through the same pipeline), and a model hammering a denied call is exactly the loop worth breaking.
|
||||
- **`agent/prompt-submit` (waterfall)** — pure reset hook: delegate via `next()`, clear the submitting agent's chain. A user interjection changes the context; repetition across it is not a loop.
|
||||
- **`agent/status` (emit)** — on `disposed`, drop the agent's state, bounding the maps over harness lifetime.
|
||||
|
||||
### Detection semantics
|
||||
|
||||
The chain key is `(tool name, canonical arguments)`; a call identical to the previous tracked call increments the agent's consecutive counter, a different tracked call resets it to 1. Canonicalization is a deep key-sort plus `JSON.stringify`: `ToolExecution.arguments` is by construction the loop's `JSON.parse` output (or the raw string fallback for malformed argument JSON, which is itself a comparable value), so the pi original's bigint/circular/`undefined` handling has no inputs here and is deliberately dropped.
|
||||
|
||||
Two deliberate rules, both documented in [the package README](../../../../packages/guard/repeat-tool-guard/README.md) because they are behavior a reader would otherwise guess at:
|
||||
|
||||
- **Untracked calls are transparent to the chain.** A call excluded by `include`/`exclude` neither increments nor resets the counter, so `grep X → todo_write → grep X` still counts as two consecutive `grep X` when `todo_write` is excluded. This is what makes exclusion useful — bookkeeping tools interleaved into a loop must not launder it — and it is the pi extension's (undocumented) semantics, kept on purpose and written down.
|
||||
- **Calls without an agent are ignored.** A direct `ctx.tools.execute()` caller (tests, non-loop consumers) has no model to remind and no `AgentId` to key on.
|
||||
|
||||
### Reminder delivery
|
||||
|
||||
Reminders ride `additionalContext` (source `{kind: 'plugin', plugin: 'repeat-tool-guard'}` — the label is load-bearing per `HookContext`), never a `content` replacement: the `tool/result` event stays the tool's own output for audit, and the loop appends buffered context as `context/message`(s) after the step's results, which the session renders as the tagged synthetic-user envelope and derived history replays. Thresholds escalate: the first configured threshold gets a short "you are repeating yourself, analyze the previous result" nudge; each later threshold gets the detailed form naming the tool, the repeat count, and the canonical arguments (head-truncated at `argumentsPreviewChars`, default 500 — a looping `write`-sized payload must not ride into the next request unbounded; the chain key always compares the full canonical string), and stating that the calls made no progress. The pi original hardcodes the gentle text to the literal count 3; the guard keys it to `thresholds[0]`, fixing that bug in the port. When the downstream decision already carries `additionalContext` (a hook bridge on the same call), the guard concatenates content under its own `source` — a `HookContext` holds one `MessageSource`, and `source.kind` is what framing depends on.
|
||||
|
||||
### Config
|
||||
|
||||
```yaml
|
||||
- id: repeat-tool-guard
|
||||
name: '@deepseek-ai/dsh-repeat-tool-guard'
|
||||
config:
|
||||
thresholds: [3, 5, 8] # default; consecutive counts that trigger a reminder
|
||||
include: [] # tool-name patterns to track; empty ⇒ all tools
|
||||
exclude: [todo_write] # tool-name patterns transparent to the chain
|
||||
argumentsPreviewChars: 500 # default; cap on arguments quoted in the detailed reminder
|
||||
```
|
||||
|
||||
`thresholds` is validated at load and throws on an empty list, a non-integer, a value below 2, or a duplicate — misconfiguration fails loud, replacing the pi original's silent fall-back to defaults. `include`/`exclude` entries support `*` wildcards. Patterns are predicates over whatever tools exist at call time, not references to a registry entry, so an entry matching no currently registered tool is NOT an error — unlike `toolOrder`'s referent check, `exclude: [mcp_*]` must stay valid in a deployment that loads no MCP tools.
|
||||
|
||||
## Testing
|
||||
|
||||
**Unit** — the suite drives a real agent loop against a scripted mock adapter (no network) and covers, at per-file 100%: counting/reset semantics (identical, different-tracked, untracked-transparent, prompt-submit reset, disposal cleanup, per-agent isolation), canonicalization (deep key-order insensitivity), threshold escalation including the `thresholds[0]` gentle-text rule, denied-call counting, no-agent transparency, wildcard escaping, config fail-loud cases, and both fold-onto-downstream paths (block and accept-with-replacement). **Snapshot** — the `repeat-tool-guard` scenario in the acp-agent example suite scripts five identical `todo_write` calls and pins both reminder tiers (gentle at the third, detailed at the fifth) as `context/message`s in the ACP transcript and the session log; the guard is loaded in the example's live tree (`cordis.yml`), inert for every other scenario (none repeats a call three times). The scenario is authored keyless (like `error-finish`/`cancel`): deterministically forcing a live model to repeat one call three times is not a stable recording. **e2e** — none: the plugin is provider-independent and deterministic, and the seam contracts it relies on are e2e-covered by their owners.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Append the reminder into the tool result** (`accept` with replaced `content` — the pi extension's mechanism, which patches result content because that is the only channel its API offers) — rejected: it makes the logged `tool/result` lie about what the tool returned, and `additionalContext` exists precisely as the separate sanctioned channel for post-execute commentary, with loop-level buffering that preserves call/result adjacency.
|
||||
- **Count in `tools/pre-execute` with a pending-reminder map** (the pi two-phase shape) — rejected: post-execute alone sees `(exec, result)` together and also fires for denied calls, so one listener with no cross-event state covers strictly more attempts with less machinery.
|
||||
- **Escalate to `block` at the highest threshold** — rejected for the initial scope: a blocked call punishes legitimate identical repeats (polling a long-running terminal, re-checking a file the agent expects to change), and an advisory reminder keeps the model in control. Revisit with evidence; the decision shape (`PostToolDecision`) already supports it.
|
||||
- **A per-deployment external hook via the CC/Codex bridges** (a `PostToolUse` script) — rejected as the answer: it works for one deployment, but a shipped, unit-tested, `cordis.yml`-configurable plugin is the harness-native form, without per-call subprocess cost.
|
||||
- **A loop-level step or repetition budget in `agent-loop`** — rejected: "plugins, not loop changes"; a hard step budget is a blunter, orthogonal control that would need its own proposal.
|
||||
- **Fuzzy/near-identical detection** (normalized paths, similar-but-not-equal arguments) — rejected: exact match after canonicalization is cheap, deterministic, and explainable to the model; similarity thresholds invite false positives and need evidence before they earn complexity.
|
||||
- **Placing the package in `core/`** — rejected: core is the product spine; a behavioral guard is an optional leaf plugin, and the `todo/` precedent is a small dedicated group per plugin family.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The reminder is advisory by design: idempotent polling patterns that repeat identical calls on purpose still receive nudges past the thresholds, and the pressure valves are config (`thresholds`, `exclude`) plus reminder text that explicitly allows finishing when enough evidence has been gathered. Each trigger costs reminder tokens on the next request; thresholds bound the frequency.
|
||||
- Chain state is in-memory only: a session resumed from persistence starts with a fresh chain, so a loop spanning a resume draws its reminders later than a live one — accepted, the guard is a heuristic nudge, not a logged invariant, and persisting counter state would buy little for real complexity.
|
||||
- When multiple post-execute producers attach context on one call, the fold concatenates under the guard's `source`; ordering between plugins follows listener registration order. The seam cannot represent mixed provenance — a limit inherited from `HookContext`, not owned by this plugin.
|
||||
- Implementing the snapshot tier surfaced a hidden assumption in the suite kit: the fixture guard equated "authored model scenario" with "override-driven". The `Scenario` table now carries an explicit `overridden` flag, and the sidecar's presence is checked BOTH ways against it (an unregistered stray sidecar would silently replace the derived script) — the suite kit is stricter than it was before this plugin existed.
|
||||
|
||||
## Deferred
|
||||
|
||||
- Compaction does not reset chains: a compacted history changes what the model sees, but the repetition risk usually survives compaction.
|
||||
- Escalating to `block` at a high threshold is not implemented; `PostToolDecision` already supports it if evidence arrives.
|
||||
- Subagent chains stay isolated per agent; no sharing mechanism exists until a concrete case appears.
|
||||
@@ -68,7 +68,7 @@ The replay plugin lives in its own package, `@deepseek-ai/dsh-llm-replay` (`pack
|
||||
|
||||
### Two subcommands, replay in the default gate
|
||||
|
||||
`pnpm run test:snapshot` runs replay (keyless) and is composed into the default `pnpm run test` gate so every PR gets the regression check (the main `vitest.config.ts` include stays narrow; the gate is `test && test:snapshot`). `pnpm run test:snapshot:record` requires `DEEPSEEK_API_KEY` (loaded from repo `.env` first), hits the real API, harvests the produced `session.jsonl` (the replay source AND the expected-log artifact), and `--update`s the stdout golden in one pass. Both forward a scenario filter. A missing fixture in replay **fails loud** with a "record first" message rather than self-skipping (the e2e self-skip rule is a CI-secret accommodation, not appropriate here — a committed-fixture test that silently vanishes is a coverage hole). A no-model scenario's `session.jsonl` simply has no `assistant/chunk` events (empty derived script); fail-loud still applies if a model call happens with no entry. An orphan-fixture guard test fails on a golden/fixture not referenced by any scenario (Vitest does not prune orphaned raw goldens), and a per-kind required-fixture guard asserts each scenario ships exactly the files its kind needs (`input.json` + `stdout.golden.jsonl` + `session.jsonl` for ALL scenarios — the harness passes `<dir>/session.jsonl` to `llm-replay` unconditionally, so even a no-model scenario needs its header-only fixture or `loadReplayScript()` fails; `replay.override.json` additionally for authored model scenarios).
|
||||
`pnpm run test:snapshot` runs replay (keyless) and is composed into the default `pnpm run test` gate so every PR gets the regression check (the main `vitest.config.ts` include stays narrow; the gate is `test && test:snapshot`). `pnpm run test:snapshot:record` requires `DEEPSEEK_API_KEY` (loaded from repo `.env` first), hits the real API, harvests the produced `session.jsonl` (the replay source AND the expected-log artifact), and `--update`s the stdout golden in one pass. Both forward a scenario filter. A missing fixture in replay **fails loud** with a "record first" message rather than self-skipping (the e2e self-skip rule is a CI-secret accommodation, not appropriate here — a committed-fixture test that silently vanishes is a coverage hole). A no-model scenario's `session.jsonl` simply has no `assistant/chunk` events (empty derived script); fail-loud still applies if a model call happens with no entry. An orphan-fixture guard test fails on a golden/fixture not referenced by any scenario (Vitest does not prune orphaned raw goldens), and a per-kind required-fixture guard asserts each scenario ships exactly the files its kind needs (`input.json` + `stdout.golden.jsonl` + `session.jsonl` for ALL scenarios — the harness passes `<dir>/session.jsonl` to `llm-replay` unconditionally, so even a no-model scenario needs its header-only fixture or `loadReplayScript()` fails; `replay.override.json` exactly for the scenarios whose table entry sets `overridden` — required with the flag, forbidden without it, because the harness forwards the sidecar purely on file existence and an unregistered stray would silently replace the derived script).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -32,4 +32,4 @@ Reviewers lose one artifact name that made the expected persisted log visually s
|
||||
|
||||
## Implementation note
|
||||
|
||||
The comparison normalizes BOTH sides, but each against its OWN volatile values, not a shared context. A raw harvested `session.jsonl` bakes in the recording run's session id, cwd, and timestamps; the replay run produces fresh ones. `normalizeSessionLog` scrubs cwd by exact string match, so normalizing the fixture against the *replay* run's cwd would leave the recorded cwd in the header unscrubbed and the compare would fail. The harness therefore derives the fixture's normalize context from its OWN header line (`{ type:'session', id, cwd }`) — `fixtureContext()` in `acp.snapshot.ts` — so both sides scrub to the same `{{sessionId}}`/`{{cwd}}` tokens. An authored fixture copied from the old golden already carries the normalized header (`id:'{{sessionId}}'`, `cwd:'{{cwd}}'`), which yields those tokens as the volatile values and scrubs idempotently. The session-log side uses a plain normalized-string `toEqual`, NOT `toMatchFileSnapshot`, so a run never overwrites the fixture.
|
||||
The comparison normalizes BOTH sides, but each against its OWN volatile values, not a shared context. A raw harvested `session.jsonl` bakes in the recording run's session id, cwd, and timestamps; the replay run produces fresh ones. `normalizeSessionLog` scrubs cwd by exact string match, so normalizing the fixture against the *replay* run's cwd would leave the recorded cwd in the header unscrubbed and the compare would fail. The harness therefore derives the fixture's normalize context from its OWN header line (`{ type:'session', id, cwd }`) — `fixtureContext()` in `dsh-acp-snapshot`'s suite module — so both sides scrub to the same `{{sessionId}}`/`{{cwd}}` tokens. An authored fixture copied from the old golden already carries the normalized header (`id:'{{sessionId}}'`, `cwd:'{{cwd}}'`), which yields those tokens as the volatile values and scrubs idempotently. The session-log side uses a plain normalized-string `toEqual`, NOT `toMatchFileSnapshot`, so a run never overwrites the fixture.
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ Every model-driving ACP snapshot fixture (`session.jsonl`) embedded the full com
|
||||
|
||||
## Decision
|
||||
|
||||
Exactly one scenario — `text-turn`, flagged `pinsHeader` in `acp.snapshot.ts` — commits and compares the full request-header content. Every other fixture stores and compares that content as stable tokens via the pure normalizer `scrubRequestHeaders` in `snapshot-normalize.ts`: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`; a `request/header-delta` keeps its structural facts — the system delta's `keepStart`/`keepEnd` line positions with one `{{system}}` token per inserted line, the tools delta's added/removed/changed tool names — and tokenizes only the bulk (prompt text, schema bodies), so two different deltas still compare different. The scrub is composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap churns every fixture by design (it invalidates the recorded responses), while a prompt or schema edit churns none of them (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
|
||||
Exactly one scenario — `text-turn`, flagged `pinsHeader` in the `acp.snapshot.ts` scenario table — commits and compares the full request-header content; the pin mechanics live in [`dsh-acp-snapshot`](../../../../packages/support/acp-snapshot/README.md), whose suite factory enforces one pin per consuming suite. Every other fixture stores and compares that content as stable tokens via the pure normalizer `scrubRequestHeaders` in that package's `normalize.ts`: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`; a `request/header-delta` keeps its structural facts — the system delta's `keepStart`/`keepEnd` line positions with one `{{system}}` token per inserted line, the tools delta's added/removed/changed tool names — and tokenizes only the bulk (prompt text, schema bodies), so two different deltas still compare different. The scrub is composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap churns every fixture by design (it invalidates the recorded responses), while a prompt or schema edit churns none of them (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
|
||||
|
||||
A system-prompt or tool-schema change therefore lands as exactly one committed-fixture diff — the pinned `text-turn` header line — updated by hand or by re-recording that one scenario (`pnpm run test:snapshot:record` with `-t text-turn`).
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# RFC: Extract the ACP snapshot suite into a support package
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The ACP snapshot tier ([snapshot RFC](2026-06-19-acp-snapshot-tests.md)) was built from three modules living inside one example's test directory: `snapshot-harness.ts` (boot the real bin subprocess, drive it over ACP JSON-RPC, harvest the persisted logs), `snapshot-normalize.ts` (the pure golden normalizers), and the ~150-line scenario body plus fixture guards in `acp.snapshot.ts` (record/replay modes, the stdout-golden and log compares, the pinned-header uniformity guard, the orphan/required-file/single-pin meta-tests).
|
||||
|
||||
A second ACP example wanting snapshot coverage — the sandbox/approval composition is the immediate consumer — could only copy those modules, forking exactly the logic that must not drift: record write-back, header scrubbing, child-session harvest ordering. The spawn/client glue was already triplicated across `acp.e2e.ts`, `hooks.e2e.ts`, and the harness (`TODO(acp-test-harness)`). Location also decided test rigor: the per-file 100% coverage gate measures `packages/*/*/src` only, so none of this machinery was measured — the same gap that had moved `dsh-llm-replay` out of `examples/` into [packages/support](../../../../packages/support/README.md). And the harness's ACP client hardcoded `requestPermission → cancelled`, so an approval round-trip — the headline behavior of the sandbox composition — could not be expressed at the snapshot tier at all.
|
||||
|
||||
## Decision
|
||||
|
||||
The machinery lives in [`packages/support/acp-snapshot`](../../../../packages/support/acp-snapshot/README.md) (`@deepseek-ai/dsh-acp-snapshot`); an example's `*.snapshot.ts` is its scenario table, its agent paths, and one factory call, over its own `snapshots/` fixtures and `cordis.snapshot.yml` overlay ([single-source replay config](2026-07-04-single-source-acp-replay-config.md)). Reading `DSH_SNAPSHOT` stays at that edge — the library takes a resolved `mode`.
|
||||
|
||||
**`src/harness.ts`** — `runScenario` and the input-script/result types, parameterized by an `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath`; absolute paths the consuming suite resolves from its own `import.meta.url`). The client's `session/request_permission` handler consumes an optional `InputScript.permissionAnswers` FIFO queue, each entry selecting by option **kind** (ids are agent-issued randoms a committed script cannot know; kinds are the ACP-stable vocabulary, mapped to the offered `optionId` at answer time); an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run — the agent itself is answered `cancelled`, so the scenario bug fails the harness rather than being absorbed as an agent-side denial. This is what lets an approval suite drive allow/reject round-trips deterministically from `input.json`.
|
||||
|
||||
**`src/normalize.ts`** — the pure normalizers, hook-free by policy: when a future event carries a new volatile field (an approval duration, say), the shared normalizer learns it in the same change, keeping one home for what "normalized" means rather than per-suite scrub extensions.
|
||||
|
||||
**`src/suite.ts`** — the `Scenario` type and `defineAcpSnapshotSuite(options)`, registering the per-scenario compares, record-mode fixture write-back, the header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin, non-pinning fixtures are `scrubRequestHeaders` fixed points). The pinned-header contract ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)) is per-suite: each suite flags exactly one `pinsHeader` scenario (the factory throws on zero, a meta-test rejects more than one; WHICH scenario pins is the table's reviewable choice), and the uniformity guard compares only that suite's sessions. The pure helpers (`childFixturePaths`, `fixtureContext`, `normalizedHeaders`, `headerDeltaCount`) are exported for direct unit coverage.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Copy the modules into each example** — the fork this RFC exists to prevent: the record/guard logic is exactly the code that must stay byte-identical across suites, and examples are outside the coverage gate, so each copy is also unmeasured.
|
||||
- **A shared module directory under `examples/`** — keeps the code outside the coverage gate and forces relative imports across example boundaries, against the package-name import convention; `examples/` leaves stay thin by design.
|
||||
- **A `/testing` subpath export of `dsh-acp-agent`** — couples test infrastructure into a product package's surface and dependency set; `packages/support/` exists precisely for real-but-lower-compatibility dev/test packages, with `dsh-llm-replay` as the precedent this package completes.
|
||||
- **Export raw test-body functions instead of a suite factory** — each example would re-own the `describe`/`it` skeleton (~80 lines of registration boilerplate per suite) for no flexibility gain; the factory keeps consumers to a scenario table plus one call, and the exported pure helpers preserve unit-testability inside the factory design.
|
||||
- **An injectable ACP `Client` factory instead of declarative `permissionAnswers`** — maximally flexible, but it leaks SDK client construction to every consumer and reopens per-example drift in exactly the layer being unified; a declarative queue keeps `input.json` the single scripting surface and stays golden-normalizable.
|
||||
- **Generalize beyond ACP (a transport-agnostic snapshot harness)** — no second transport exists; the harness is ACP-shaped end to end (SDK client, JSON-RPC frames, `session/update` waiters), and a speculative abstraction would be a seam split ahead of any consumer.
|
||||
|
||||
## Testing
|
||||
|
||||
Extraction parity was proven mechanically: after the move, `pnpm run test:snapshot` matched the base commit's result with zero byte changes under `examples/acp-agent/tests/snapshots/`. The package's `src/` holds per-file 100% statements/branches/functions/lines under the gating unit run, driven through the REAL spawn path by a scripted fake ACP bin (`tests/fixtures/fake-acp-agent.ts`, behavior scripted per scenario via a `behavior.json` beside the fixture): `harness.spec.ts` covers every step op, both expect-error arms, the permission queue (selection, fallback, impossible-click), env forwarding, workspace seeding, and the harvest ordering/noise/fallback branches; `suite.spec.ts` runs the factory for real at collection time — a replay suite over committed synthetic fixtures and a record suite over a temp copy (write-back never touches the committed tree; `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` re-bootstraps it) — plus direct cases for the pure helpers. Two structurally unreachable guards carry reasoned `v8 ignore` comments. The fake bin substitutes the `session/new` cwd, not `process.cwd()`, into scripted logs, matching what the real bin's header carries (darwin realpaths `/var/folders/…` to `/private/var/folders/…`).
|
||||
|
||||
## Consequences
|
||||
|
||||
A new example gets the whole snapshot tier from a scenario table plus fixtures — the sandbox branch merges master down and adds its own suite (own pin scenario, own overlay, fixtures via `test:snapshot:record`, approvals via `permissionAnswers`). The costs: `suite.ts` imports vitest, so the package is importable only inside a vitest run — a shape no other package has, stated in its README; each suite pins its own ~8 KB header fixture (a genuinely distinct composition deserves its own pin; an identical one would be caught by that suite's uniformity guard); and the e2e launcher duplication remains (`TODO(acp-test-harness)`) — the harness is the extraction target when that migration lands.
|
||||
Reference in New Issue
Block a user