test(sandbox-policy): record cache-safe policy context

2026-07-30 22:15:34 +08:00
parent 9c32620e4b
commit 1a4349be88
90 changed files with 2468 additions and 2411 deletions
@@ -28,7 +28,9 @@ The earlier real-provider Web fixture quantified the defect in the system-sectio

 The wording experiment pre-registered preemptive refusal as its primary endpoint and required the old standing sentence to produce at least one refusal in twelve fresh sessions before any replacement could be judged. On 2026-07-30, commit `2bf41990401b194bd8637f07bbd90c67a9eeac75` ran `deepseek-v4-flash` through the shipped Web composition with the exact positive-control sentence `Bash commands run under the "read-only" file sandbox.` and the current tool-owned attempt guidance. The control produced zero preemptive refusals and zero speculative escalations; all twelve sessions made an ordinary bash call, observed a denial, escalated in the same turn, received approval, and landed the requested file. No sample was excluded.

-The positive control therefore failed the pre-registered sensitivity gate. Candidate A and B were not run, and this experiment does not select or validate the current wording. It instead establishes that the earlier five-of-twelve result is not reproducible under this task and current tool guidance, and that a stronger positive control or different task distribution is required before making model-behavior rate claims. Deterministic tests below establish truthful request construction and replay only.
+After the cache-safe delivery change, commit `10d4e0ff7b68d38fc4403403b644aac442b97a00` repeated the same twelve-session positive control through the new tail-context channel. It again produced zero preemptive refusals and zero speculative escalations; all twelve sessions made an ordinary first call, observed denial, escalated in the same turn, and received approval. Eight landed the exact requested file, and no sample was excluded.
+
+Both positive controls therefore failed the pre-registered sensitivity gate. The formal twelve-session Candidate A and B arms were not run, and these experiments do not select or validate the current wording. They establish that the earlier five-of-twelve result is not reproducible under this task and current tool guidance, and that a stronger positive control or different task distribution is required before making model-behavior rate claims. Deterministic tests below establish truthful request construction and replay only.

 The cache-safe delivery rework then supplied a separate, non-statistical acceptance comparison over the neutral Web task `Create the relative path policy-neutral.txt ...`; it does not replace the pre-registered twelve-session experiment. Candidate A's categorical read-only statement produced a text refusal with zero tool calls. Candidate B added one composition-conditioned sentence only for enforced families whose tools expose escalation. A fresh real-provider run then issued an ordinary `write`, observed the read-only denial, retried the same operation in the same turn with `sandbox_permissions: "workspace-write"`, received approval, read the file back, and verified the exact contents. It made no speculative escalation. Across the permission switches and four mutation steps, cache reads were 14,848–15,872 tokens while uncached input was 59–251 tokens per request, directly demonstrating the stable-prefix benefit.