Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

test(hooks): snapshot the CC + Codex hook matrix end-to-end

Browse Source 
Load both hook bridges in the ACP example (dsh-hooks-claude → ./hooks.json,
dsh-hooks-codex → ./codex-hooks.json) so the full-transcript snapshot tier can
exercise each dialect against the real app. An absent config file is a silent
no-op, so a scenario carries only the file it needs and the other bridge
vanishes — verified byte-identical against every pre-existing snapshot.

Add a scenario per hook point × its headline Decision outcome, both dialects:
UserPromptSubmit block (authored, keyless) + context-fold, PreToolUse deny/ask,
PostToolUse block/context, Stop force-continue. The mid-turn scenarios are
recorded against the real API with the hook active, so the model's reaction to
a denied/blocked/force-continued turn is part of the replayed transcript.

SessionStart and SubagentStart are deliberately excluded (detached best-effort
inject races the log position — a recorded golden fails 10/10 on its own
replay), as is SubagentStop (observe-only, zero transcript footprint — a golden
could never be proven to fail). Both stay on the bridges' unit coverage. See
docs/rfc/implemented/testing/2026-07-04-hook-snapshot-matrix.md.
This commit is contained in:
Tianyi Cui
2026-07-04 00:40:35 +08:00
parent 907c8d3871
commit 74f61c89e3
58 changed files with 3860 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/acp-agent/tests/acp.snapshot.ts
+36 -5
View File
@@ -71,11 +71,42 @@ const SCENARIOS: Scenario[] = [
{ name: 'subagent-multi', hasModelTurn: true, recorded: true, childSessions: 2 },
{ name: 'subagent-fork', hasModelTurn: true, recorded: true, childSessions: 1 },
{ name: 'subagent-mixed', hasModelTurn: true, recorded: true, childSessions: 2 },
// A UserPromptSubmit hook blocks the prompt before any step runs: no model
// call (keyless, authored — its derived script is empty so it needs no
// sidecar), but it persists a `rejected` turn carrying `hook/*` events, so its
// log IS compared. The hooks.json riding in workspace/ drives the bridge.
{ name: 'hook-prompt-block', hasModelTurn: false, comparesLog: true, recorded: false },
// Hook matrix — one scenario per hook point × its headline Decision outcome,
// across BOTH bridges (Claude `hooks.json`, Codex `codex-hooks.json`, seeded in
// workspace/). The block scenarios need no model call: a UserPromptSubmit hook
// blocks the prompt before any step runs (keyless, authored — the derived
// script is empty so no sidecar), yet persists a `rejected` turn carrying
// `hook/*` events, so their logs ARE compared. Every other point fires a real
// seam mid-turn, so its transcript is recorded WITH the hook active.
{ name: 'hook-cc-promptsubmit-block', hasModelTurn: false, comparesLog: true, recorded: false },
{ name: 'hook-codex-promptsubmit-block', hasModelTurn: false, comparesLog: true, recorded: false },
// The mid-turn seams fire during a real model turn, so each is recorded WITH
// its hook active (the model's reaction to a deny/block/force-continue is part
// of the captured transcript). The Codex bridge exercises the same seams in its
// own snake_case dialect.
//
// Two hook points are deliberately NOT snapshotted, and stay on the bridges'
// unit coverage (`bridge.spec.ts` / `coverage.spec.ts`) instead:
// - SessionStart and SubagentStart inject context through a detached,
// best-effort `void runPoint(...).then(agent.inject())` with no turn
// binding, so the resulting `context/message` races the work it precedes
// and lands at a nondeterministic log position — a recorded golden does not
// even reproduce on its own replay.
// - SubagentStop is observe-only with no turn and no injection, so it writes
// NOTHING to the transcript — a golden would be byte-identical to the
// no-hook run and could never be proven to fail.
// See the hook-snapshot-matrix RFC for the full rationale.
{ name: 'hook-cc-promptsubmit-context', hasModelTurn: true, recorded: true },
{ name: 'hook-cc-pretool-deny', hasModelTurn: true, recorded: true },
{ name: 'hook-cc-pretool-ask', hasModelTurn: true, recorded: true },
{ name: 'hook-cc-posttool-block', hasModelTurn: true, recorded: true },
{ name: 'hook-cc-posttool-context', hasModelTurn: true, recorded: true },
{ name: 'hook-cc-stop-continue', hasModelTurn: true, recorded: true },
{ name: 'hook-codex-promptsubmit-context', hasModelTurn: true, recorded: true },
{ name: 'hook-codex-pretool-block', hasModelTurn: true, recorded: true },
{ name: 'hook-codex-posttool-block', hasModelTurn: true, recorded: true },
{ name: 'hook-codex-posttool-context', hasModelTurn: true, recorded: true },
{ name: 'hook-codex-stop-continue', hasModelTurn: true, recorded: true },
]
/** The sibling child-fixture paths for a scenario (`session.1.jsonl` …). */