Add RFCs for the remaining quality-proposal ideas
Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
# RFC 002: Mutation testing as the coverage counterweight
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The per-file 100% coverage gate (ADR 0007) proves every line *executes* under
|
||||
test — not that any assertion would notice if the line were wrong. Under
|
||||
agent-written tests, coverage pressure can produce execution-without-assertion.
|
||||
Mutation testing measures what coverage cannot: whether the suite *kills*
|
||||
deliberately injected bugs.
|
||||
|
||||
## Proposal
|
||||
|
||||
Stryker (`@stryker-mutator/vitest-runner`) over `packages/*/src`:
|
||||
|
||||
- **PR-scoped incremental runs** (changed files only) as a CI job — fast
|
||||
enough to gate merges once tuned.
|
||||
- **Nightly full runs** with a tracked mutation score; start by recording,
|
||||
then set the threshold at the observed baseline and ratchet upward (same
|
||||
policy as coverage: thresholds only ever tighten).
|
||||
- Surviving mutants are work items: an agent picks a survivor, writes the
|
||||
killing test, repeats — a well-shaped autonomous loop.
|
||||
- Equivalent mutants (provably behavior-preserving) get annotated exclusions
|
||||
with reasons, mirroring the `/* v8 ignore */` policy.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Add Stryker config scoped to one package (llm — smallest, most algorithmic)
|
||||
and measure runtime.
|
||||
2. Expand to all packages; record baseline scores in the config.
|
||||
3. Wire the nightly job; add the incremental PR job once runtime is acceptable.
|
||||
|
||||
## Risks
|
||||
|
||||
Runtime: mutation testing is expensive; per-file 100% coverage helps (every
|
||||
mutant is at least reached). If PR-scoped runs stay too slow, keep them
|
||||
nightly-only and rely on the score ratchet.
|
||||
Reference in New Issue
Block a user