Tianyi Cui
4dafad4db6
Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
41 lines
1.7 KiB
Markdown
41 lines
1.7 KiB
Markdown
# RFC 003: Deterministic tests, the replay invariant fixture, and race stress
|
|
|
|
Status: proposed
|
|
|
|
## Problem
|
|
|
|
Several loop tests synchronize with `setTimeout(30)` sleeps — flakiness debt
|
|
that wastes agent cycles on retries and can mask ordering bugs. Separately,
|
|
our core architectural promise (any session log replays to identical derived
|
|
history) is asserted in two tests but is cheap to assert *everywhere*. And
|
|
the inbox wakeup race was verified by hand exactly once; nothing re-verifies
|
|
it continuously.
|
|
|
|
## Proposal
|
|
|
|
Three measures:
|
|
|
|
1. **No wall-clock sleeps in tests.** Replace `setTimeout(N)` waits with
|
|
event-driven waits (the existing `waitForIdle` pattern, extended to
|
|
`waitForStatus`, `waitForEvent(n)`) or vitest fake timers where time
|
|
itself is under test. Enforce with a lint rule banning `setTimeout` in
|
|
`packages/*/tests` outside an allowlisted helper module.
|
|
2. **Universal replay fixture.** A shared test helper wraps the loop harness
|
|
so that after every test, the agent's session log is replayed into a fresh
|
|
Session and `deriveMessages()` equality is asserted automatically. The
|
|
invariant then gets checked hundreds of times per CI run across every
|
|
scenario the suite produces, not twice.
|
|
3. **Nightly race stress.** A CI job running the agent-loop and inbox suites
|
|
with `vitest --repeat=200` (and `--shuffle`) to flush scheduling-dependent
|
|
failures; any flake found is a bug to fix, never a retry.
|
|
|
|
## Plan
|
|
|
|
Land 1 and 2 together (they touch the same helpers); add the nightly job
|
|
after the suite is sleep-free so repeats are fast.
|
|
|
|
## Risks
|
|
|
|
Fake timers interact subtly with Promise scheduling in the loop — prefer
|
|
event-driven waits; reserve fake timers for timer-service behavior itself.
|