ci: retune consolidated jobs for stacked workload
This commit is contained in:
+5
-3
@@ -16,9 +16,9 @@ The organization keeps twelve x64 larger-runner pools in the repo-restricted `ds
|
||||
|
||||
Production CI uses five larger-runner executions and one standard-runner aggregator. The primary Node inventory is not sharded:
|
||||
|
||||
- `node 24 / complete` uses one 96-core Linux runner. One checkout, setup, cache restore, and install feeds all 42 primary gates. `run-gates` starts up to 32 independent gates; ESLint uses 32 workers, coverage uses at most 12, and snapshot replay uses at most 16. Build starts immediately beside typecheck, coverage, lint, and documentation work, while snapshot replay and publication consumers retain explicit dependencies on emitted `lib/` output.
|
||||
- `node 24 / complete` uses one 96-core Linux runner. One checkout, setup, cache restore, and install feeds all 42 primary gates. `run-gates` starts up to 16 independent gates; ESLint uses 16 workers, coverage uses at most 12, and snapshot replay uses at most 8. Build starts as soon as the first short gates release scheduler slots, while snapshot replay and publication consumers retain explicit dependencies on emitted `lib/` output. Pull requests restore pnpm and ESLint caches without saving them, so cache compression and upload do not extend the required job; the master serial references refresh those caches outside the pull-request critical path.
|
||||
- Node 22.19 and Node 26 use the 4- and 32-core Linux pools for their runtime compatibility smokes. Python 3.10 uses the 8-core Linux pool for the complete keyless SDK suite. These are environment contracts, not slices of the primary Node gate inventory.
|
||||
- `windows node 24 / complete` uses one 32-core Windows runner. One setup feeds the required package build, required production site build, and complete observational portability inventory. Required failures fail the job; observational failures are reported as non-blocking. ESLint stays single-threaded because 16 ESLint workers took 174.54 seconds, while the outer scheduler retains 32 slots.
|
||||
- `windows node 24 / complete` uses one 32-core Windows runner. One setup feeds the required package build, required production site build, and complete observational portability inventory. Required failures fail the job; observational failures are reported as non-blocking. ESLint stays single-threaded because 16 ESLint workers took 174.54 seconds, coverage uses at most 12 workers, and the outer scheduler retains 16 slots. The job restores only the small master-refreshed ESLint cache and performs a clean pnpm install instead of restoring or saving the many-file package store.
|
||||
|
||||
The former gate-level and coarse primary shard jobs are absent from the workflow. Their static, lint, coverage, snapshot, and scenario shard selectors are also absent from the repository, so an unused diagnostic path cannot preserve a second CI architecture.
|
||||
|
||||
@@ -38,11 +38,13 @@ The same benchmark measured the required Windows build surfaces across every pro
|
||||
|
||||
Repository work gains little above 16 Windows cores, but the 32-core pool can start the complete outer inventory together. A [retargeted production validation](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29907581119/attempts/2) completed the full one-box Windows inventory in 173 seconds, including coverage and snapshot replay, so Windows remains consolidated.
|
||||
|
||||
The larger client package graph makes cache mechanics and scheduler pressure part of the measured workload. In [one exact-head production run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29912577681), Linux spent 39 seconds in repository gates but 69 seconds in the complete job, while Windows spent 117 seconds in repository gates and 228 seconds in the complete job. The Windows pnpm cache downloaded its 154 MB archive in about two seconds but spent 27 seconds extracting it, followed by a 23-second install and a 14-second post-job save. A [cacheless all-size trace](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29913033155) completed the same 32-core Windows install in 27 seconds. Production therefore avoids the Windows package-store cache, uses restore-only caches on latency-critical pull-request jobs, and bounds outer concurrency so typecheck, lint, coverage, and build do not oversubscribe one host.
|
||||
|
||||
Two host effects remain part of the decision. A standard Node 26 job once spent 36 of its 67 seconds in `Set up job`, which is why environment contracts use distinct larger-runner pools instead of standard capacity. A Linux candidate spent 18 seconds registering a 50 KB Bubblewrap package because the hosted image scanned 202,507 package-database files. [`scripts/prepare-ci-bubblewrap.sh`](../../../../scripts/prepare-ci-bubblewrap.sh) instead verifies and extracts the pinned payload into the ephemeral runner directory, runs a functional confinement probe, and overlaps that preparation with dependency installation.
|
||||
|
||||
Coverage remains capped at 12 forks. Thirty-two forks crashed Node 24's CJS lexer twice, and a later 16-fork run reproduced the worker loss and invalid coverage result. Twelve forks preserve process headroom without becoming the single-host critical path.
|
||||
|
||||
The workflow retains three manual measurement suites. `suite=larger-runner-benchmark` compares isolated critical lanes across every size, `suite=consolidated-runner-benchmark` compares whole aggregates, and `suite=serial-reference` remains the unsharded cross-platform completeness oracle. `suite=optimized-larger-runners` dispatches the exact production topology against a branch ref when a pull request cannot form a merge commit.
|
||||
The workflow retains two manual measurement suites. `suite=larger-runner-benchmark` compares isolated critical lanes across every size, and `suite=consolidated-runner-benchmark` compares whole aggregates. Complete serial Linux, macOS, and Windows references run only when `master` moves; pull requests run only the optimized jobs.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
Reference in New Issue
Block a user