Files

T

Chinesezjc 68e280ce4f docs(ci): make the failover runbook a conforming dated Agent Note

The failover runbook landed as .agents/notes/implemented/process/ci-failover-runbook.md,
which fails three doc-sync gates: the classification/format gates require a
yyyy-mm-dd-topic.md filename and the implemented Agent Note skeleton
(Problem/Decision/Alternatives/Consequences), and the bilingual pairing gate
requires cross-note link targets to match between the two language sides.

Rename to 2026-07-26-ci-failover-runbook.md/.zh.md, reshape both sides into the
implemented skeleton (the runbook steps live in bespoke sections under Decision),
point the sibling topology note and the ci.yml comment at the dated filename, and
make both sides link the canonical .md per the bilingual convention. Re-recorded
the i18n pairing records.

2026-07-26 05:22:28 +08:00

4.2 KiB

Raw Blame History

Agent Note: CI failover runbook — hosted pools → in-house pool

Status: implemented

English | 中文

Problem

The three required Linux jobs in CI (node 24 / static, node 24 / coverage, node 24 / snapshots and artifacts) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything.

Decision

Each of the three required Linux jobs resolves its runner pool through the DSH_CI_FAILOVER repository variable. Unset (normal), they run on the hosted enterprise pools. Set to selfhosted by a repository admin, all three retarget onto the in-house self-hosted vm-backup pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the serial / linux (self-hosted standby) lane, which runs the complete unsharded aggregate on every master push.

What the in-house pool is

vm-backup: one 64-core VM, four always-on systemd-managed runner instances, four registered spares. Check the latest serial / linux (self-hosted standby) run before switching: a green standby is verified-yesterday capacity.

Switch (repo admin, ~1 minute, no merge)

Repository Settings → Secrets and variables → Actions → Variables → New repository variable: name DSH_CI_FAILOVER, value selfhosted.
Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up).
That is the entire switch. Under failover the workflow also, automatically: halves DSH_COVERAGE_MAX_WORKERS to 12 and DSH_SNAPSHOT_MAX_CONCURRENCY to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).

Capacity during failover

Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered):

for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done

Switch back

Delete the DSH_CI_FAILOVER variable (or set it to anything other than selfhosted). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started.

Trust boundary

The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.

Alternatives considered

Merge a workflow change to switch pools. Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is admin-controlled state that takes effect on re-run without a merge.

Keep the self-hosted pool always in the required path. Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variable keeps the hosted pools primary and the self-hosted pool a proven, one-action standby.

Consequences

Recovering from a hosted-pool outage is a single admin variable plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in ci.yml carry a selfhosted leg that must stay in step with the hosted leg.

4.2 KiB Raw Blame History