Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

docs(ci): runbook — glob migrated identity files; scope the promise to enterprise-pool outages

Browse Source 
- The clone exclusion is now '.runner*' (catches .runner_migrated /
  .credentials_migrated, which GitHub writes on migrated runners and
  which equally trip the already-configured refusal — hit live during
  the 2026-07-27 org-group migration).
- The problem statement now states the verdict runs on ubuntu-latest,
  scopes the switch to enterprise Linux-pool outages, and names the
  residual dependency: node-compat/python-sdk/windows stay standard-
  hosted, so a broader hosted-capacity failure still blocks the
  verdict, with Windows the only leg lacking any in-house substitute.
Static gate green locally: 32 passed, 0 failed.
This commit is contained in:
Chinesezjc
2026-07-27 16:56:02 +08:00
parent 2bacd4814e
commit 7b49f4a4ee
3 changed files with 6 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
+2 -2
View File
@@ -6,7 +6,7 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md)
## Problem
The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch any responder with repository write access can throw without merging anything.
The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. **Scope: this switch recovers an enterprise Linux-pool outage.** The verdict's other required dependencies (`node-compat`, `python-sdk`, `windows`) stay on standard hosted runners by design (the portable boundary); in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block `all checks passed`, and only the Windows leg has no in-house substitute at all — during the 2026-07-27 outage the standard pools recovered first, which is the ordering this design bets on. An outage therefore needs a switch any responder with repository write access can throw without merging anything.
## Decision
@@ -28,7 +28,7 @@ Each of the three required Linux worker jobs — and the `all checks passed` ver
## Capacity during failover
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/` — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance.
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance.
### Switch back