Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

docs: record reduced-fanout validation

Browse Source
This commit is contained in:
Tianyi Cui
2026-07-22 15:57:39 +08:00
parent cd19f87c3f
commit d7945da13e
3 changed files with 44 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md
+21 -17
View File
@@ -14,13 +14,13 @@ Larger runners make it possible to pay setup once and parallelize inside the rep
The organization keeps twelve x64 larger-runner pools in the repo-restricted `dsh-larger-ci` group: Ubuntu 24.04 and Windows 2025 at 4, 8, 16, 32, 64, and 96 cores. Public IPs are disabled. Each pool has an autoscaling ceiling of 256; the ceiling does not allocate idle machines or remove the need to bound workflow demand.
Production CI uses three larger-runner jobs and keeps Node compatibility, Python, and the final aggregator on standard runners:
Production CI uses three coarse larger-runner jobs and keeps Node compatibility, Python, and the final aggregator on standard runners:
- `node 24 / complete` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds the complete unsharded 40-gate primary inventory. `run-gates` starts up to 32 independent gates, ESLint uses 32 workers, snapshots use up to 32 subprocesses, and coverage uses 16 forks. Build starts beside typecheck; snapshot and artifact consumers still wait for emitted output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage.
- `windows / blocking builds` uses the 16-core Windows pool. Build and the production VitePress site run concurrently after one setup.
- `windows node 24 / observational` uses the 32-core Windows pool. Its complete unsharded 37-gate static, lint, and artifact inventory runs with 32 outer scheduler slots and remains non-blocking. ESLint itself stays single-threaded because 16 ESLint worker threads increased full-lint time to 174.54 seconds; with outer concurrency and no ESLint workers, the same full lint took 31.67 seconds.
- `node 24 / core` uses the 96-core Linux pool. One checkout, setup, cache restore, and install feeds the unsharded 39-gate primary inventory other than the production site build. `run-gates` starts up to 32 independent gates, ESLint uses 32 workers, snapshots use up to 32 subprocesses, and coverage uses 16 forks. Build starts beside typecheck; snapshot and artifact consumers still wait for emitted output. Coverage stays below 32 forks because that setting twice caused Node 24's CJS lexer to terminate a Vitest worker and invalidate coverage.
- `node 24 / production site` uses the 16-core Linux pool for the longest independent primary gate. This is one coarse split, not a shard matrix: the job performs one setup and one production VitePress build.
- `windows node 24 / complete` uses the 32-core Windows pool. One setup feeds the required package build, the required production site build, and the complete observational portability inventory. The outer scheduler has 32 slots. Required failures fail the job; observational failures are printed as non-blocking and preserve their former advisory status. ESLint itself stays single-threaded because 16 ESLint worker threads increased full-lint time to 174.54 seconds; outer gate concurrency uses the runner without multiplying Windows worker startup and TypeScript project loading.
The two Windows jobs deliberately use different pools. A first candidate put both on the 16-core pool and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. Using the 16-core and 32-core pools let every production job begin within two seconds in the final validation run.
The final shape followed two cold-path observations. A first candidate used two 16-core Windows jobs, and GitHub took 93 seconds to provision the second same-label runner despite the configured autoscaling ceiling. A later [documentation-head validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29900502413) took 64 seconds on the combined Linux job because its production site gate lasted 29.05 seconds, while a separate Windows blocking job took 266 seconds after spending 138 seconds restoring a 153 MB pnpm cache. Moving that independent Linux gate to one coarse 16-core lane and combining all Windows work on one 32-core box removed both duplicate setup waves.
The workflow retains four manual diagnostics. `suite=larger-runner-benchmark` compares isolated critical lanes across every size, `suite=consolidated-runner-benchmark` compares whole aggregates, `suite=sharded-reference` preserves the former production shard topology, and `suite=serial-reference` remains the unsharded cross-platform completeness oracle. `suite=optimized-larger-runners` runs the exact production topology against a branch ref when a pull request cannot form a merge commit.
@@ -33,23 +33,23 @@ The first [twelve-size critical-lane benchmark](https://github.com/deepseek-harn
Those isolated results showed that setup dominated but did not identify the production size. A [whole-aggregate benchmark without native ESLint concurrency](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29897826082) found a 69-second single-threaded Linux lint gate. After enabling native Linux ESLint concurrency, the [second whole-aggregate benchmark](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29898331705) produced these active job times:
| Aggregate job | 4 cores | 8 cores | 16 cores | 32 cores | 64 cores | 96 cores |
| Job | 4 cores | 8 cores | 16 cores | 32 cores | 64 cores | 96 cores |
|---|---:|---:|---:|---:|---:|---:|
| Linux complete primary | 147 s | 104 s | 95 s | failed at 57 s | 66 s | 60 s |
| Windows blocking builds | 137 s | 127 s | 113 s | 107 s | 105 s | 131 s |
The Linux 32-core failure was the first CJS-lexer worker crash. The 96-core aggregate was the only successful all-size result at the one-minute boundary. Windows gained little above 16 cores, so the blocking job uses 16 cores; the observational job uses a separate 32-core pool to avoid same-label provisioning delay and to start all outer gates together.
The Linux 32-core failure was the first CJS-lexer worker crash. The 96-core aggregate was the only successful all-size result at the one-minute boundary. Although Windows repository work gained little above 16 cores, the 32-core pool can start the complete outer inventory together and, more importantly, removes an entire paid setup from production.
The exact production [validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29899733584) passed every job at the tested branch head:
The exact [reduced-fanout validation run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29901539360) passed every job at the tested branch head:
| Production job | Active time | Repository work | Result |
|---|---:|---:|---:|
| Linux complete primary | 50 s | 40 gates in 23.23 s | passed |
| Slowest standard non-Windows job | 40 s | Node 26 compatibility | passed |
| Windows blocking builds | 91 s | 2 gates in 28.69 s | passed |
| Windows observational | 153 s | 37 gates in 31.83 s | passed |
| Linux core | 52 s | 39 gates in 25.84 s | passed |
| Linux production site | 44 s | VitePress in 22.42 s | passed |
| Slowest standard non-Windows job | 43 s | Node 26 compatibility | passed |
| Windows complete | 96 s | 37 gates in 31.13 s | passed |
The Windows observational job spent 57 seconds restoring its pnpm cache, so its remaining margin measures hosted setup variance as well as repository work. The final run still stays below one minute for every non-Windows job and below three minutes for both Windows jobs.
All three paid jobs began together. The Windows job spent 21 seconds restoring its pnpm cache and 25 seconds installing dependencies, so its margin measures hosted setup variance as well as repository work. Every non-Windows job stays below one minute and the sole Windows job stays below three minutes.
## Alternatives considered
@@ -57,16 +57,20 @@ The Windows observational job spent 57 seconds restoring its pnpm cache, so its
**Select a production size from the critical-lane benchmark.** Four cores looked cost-effective for isolated typecheck and site builds, but the full aggregate found repository-wide lint and dependent artifact work that those commands did not represent.
**Prebuild before starting the Linux aggregate.** This moved build onto the setup path and produced a 66-second candidate. Starting build eagerly inside `run-gates` preserves artifact dependencies while overlapping it with unrelated checks; the final aggregate completed in 23.23 seconds.
**Run the production site inside the Linux core aggregate.** This reached 50 seconds with warm hosted setup, then crossed the threshold at 64 seconds when the site gate took 29.05 seconds. One coarse independent site job protects the target without returning to gate-level sharding.
**Keep required and observational Windows checks in separate jobs.** The split preserved status semantics at the workflow level but paid setup twice, and a cold cache pushed the required job to 266 seconds. `run-gates` now preserves those semantics inside one process: build and production site are required, while the remaining inventory is explicitly non-blocking.
**Prebuild before starting the Linux aggregate.** This moved build onto the setup path and produced a 66-second candidate. Starting build eagerly inside `run-gates` preserves artifact dependencies while overlapping it with unrelated checks.
**Use native ESLint worker concurrency on Windows.** Sixteen workers made lint more than five times slower than the final single-threaded result. Outer gate parallelism uses the 32-core runner without multiplying ESLint's Windows worker startup and TypeScript project loading.
**Move compatibility, Python, and aggregation to larger runners.** These standard-runner jobs all completed in 40 seconds or less. Paid capacity would not shorten the critical path.
**Move compatibility, Python, and aggregation to larger runners.** These standard-runner jobs completed in 43 seconds or less. Paid capacity would not shorten the critical path.
## Consequences
The final production validation consumed one billed 96-core Linux minute, two billed 16-core Windows minutes, and three billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.902. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to three matters as much as shortening repository work.
The reduced-fanout validation consumed one billed 96-core Linux minute, one billed 16-core Linux minute, and two billed 32-core Windows minutes. At the configured larger-runner rates, its larger-runner cost was $0.618. The all-size critical benchmark cost $2.936. GitHub rounds each larger-runner job up to a whole minute, so reducing paid job count from 49 to three matters as much as shortening repository work.
The existing zero-dollar Actions budget did not block larger-runner jobs. The repo-only runner group, bounded workflow topology, manual benchmark triggers, and job timeouts are the observed cost controls; the budget is not treated as an execution guard.
Production CI depends on the organization-owned runner names in this note and in `.github/workflows/ci.yml`. Missing or renamed pools leave jobs queued instead of falling back to standard capacity. Manual all-size and former-shard suites remain available so image, dependency, scheduler, or pricing changes can be remeasured before changing production labels.
Production CI depends on the organization-owned runner names in this note and in `.github/workflows/ci.yml`. Missing or renamed pools leave jobs queued instead of falling back to standard capacity. Manual all-size, consolidated, former-shard, and serial suites remain available so image, dependency, scheduler, or pricing changes can be remeasured before changing production labels.