Tianyi Cui
605587e79c
Add a second axis to every RFC — its class (feature, bug-fix,
simplification, architecture, process, testing) — encoded in the path
as docs/rfc/{lifecycle}/{class}/file.md. The folder is the label, so
the closed set is enforced by structure rather than a parsed field.
Two new doc-sync gates back it:
- verify-rfc-classification: every RFC sits in a valid class folder and
the README index lists it under the matching lifecycle→class heading.
- verify-doc-refs: every docs/*.md path cited in a packages|examples TS
comment resolves — closes a drift class verify-md-links can't see, and
catches the four comment refs this reorg moved.
The README gains a Classification section explaining the taxonomy and
per-class index sub-sections. A self-referential process RFC records why
the scheme is path-encoded and gated.
3.3 KiB
RFC: Property-based testing for protocol-shaped code
Status: implemented (proposed 2026-06-11, accepted 2026-06-14)
Merges the original proposal and the decision record for one topic. It found a real BlockAssembler duplicate-
block-endbug on first run.
Context
Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion, inbox scheduling — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for. The motivating evidence: a streamBlocks ordering bug once survived 100% line coverage of the happy paths. Per-file 100% coverage proves every line ran, not that every interleaving is correct.
Decision
Adopt fast-check (a root devDependency) with one tests/properties.spec.ts per protocol-shaped package, generators tuned for realistic-but-adversarial inputs (not uniform noise) and numRuns kept so the suite stays well under ~10s locally. Failures print a reproducible seed. (The original proposal also sketched a nightly CI job running 100× the iterations; that was not shipped — the property suite runs only in the normal push/pull_request CI, and a scheduled high-iteration job remains possible future work.)
- dsh-llm / BlockAssembler: arbitrary chunk streams (valid + malformed: duplicate indices, stragglers, missing block-start). Invariants:
flushReady()+flushRemaining() ≡ blocks()in order; the streamed prefix is always a prefix of finalblocks(); partial count ≤ distinct indices; re-assembly idempotent. - dsh-session: arbitrary event logs. Invariants:
deriveMessagesdeterministic; replay-from-seed identical; seq strictly monotonic; non-message events never affect derived history; derived content is decoupled from the log. - dsh-tools: arbitrary
SchemaSpec. Invariants: JSON Schemarequiredequals therequired:truekeys at every level; conversion total; and the composition with runtime arg validation — generated args satisfying a spec passvalidateArgs, and targeted corruptions (dropped required key, non-object top level) are rejected. This closes the validator/InferArgsdrift risk. - dsh-agent-loop: arbitrary send schedules against a never-exhausting adapter, driven through the
agent/statussettle signal (no wall-clock sleeps). Invariants: no message lost; turn numbers strictly increase; status transitions stay on the legal machine.
Consequences
- Generator quality is the value lever — the generators bias toward small index pools and short strings so collisions and interleavings are common.
- It already paid off: the BlockAssembler stream found a real bug — a duplicate
block-endat the same index overwrote an already-flushed block, so the streamed prefix disagreed with finalblocks(). Fixed (first close wins, matching the existing straggler rule) with a dedicated regression test. - A property flake from a timeout is a finding, not something to retry away. The loop properties are deterministic by construction (settle on
agent/status), so a hang is a real defect. - Property tests supplement, not replace, the example tests that pin specific branches for the 100%-coverage gate.