← all branches

fix/cdk-timeout

needs attention
f484979 · fullpre-PRreviewed 2026-07-23 06:02 UTC1H · 2M · 2L · 2I
The branch
Purpose
Fix a production incident where the 30-min CDK job ceiling fired mid-deploy (41/43 stacks on 2026-07-23), yielding 'cancelled' instead of 'failure' and silently bypassing the rollback gate
Goal
Ensure CDK timeouts surface as job FAILURE (which triggers rollback) rather than CANCELLED (which the rollback gate ignores), and provide deploy-duration telemetry for evidence-based future tuning
Sub-goals
  • SG-1: Raise job ceiling from 30 → 75 min with a documented recovery runbook for mid-deploy kills
  • SG-2: Add step-level timeouts (diff 15 min, deploy 45 min) so overruns surface as 'failure' not 'cancelled'
  • SG-3: Add $SECONDS deploy-duration telemetry to the step summary for evidence-based calibration
The changes (whole branch)
What
Single file changed (.github/workflows/release-promote.yml): job timeout-minutes 30→75 with recovery comment; step-level timeout-minutes: 15 added to CDK diff step; step-level timeout-minutes: 45 added to CDK deploy step; deploy step run block expanded to echo $SECONDS timing to $GITHUB_STEP_SUMMARY
Why
The 30-min job-level timeout fired at 41/43 stacks during a large deploy (accrued week of changes + CFE layer rebuilds). GitHub marks a job-level timeout as 'cancelled', which the rollback job's if-condition explicitly ignores — meaning the rollback gate did not fire. Step-level timeouts fire as step 'failure', which propagates to job 'failure', which the rollback gate does catch.
Areas
.github/workflows/release-promote.yml+212
Blast
1 file, +21/-2 lines; CI-only change, no application code or infrastructure modified; affects only the prod CDK deploy phase of release-promote.yml
ci-only prod-incident-fix no-app-code-changed
CI· no open PR — push-triggered review of pre-PR branchCodeRabbit· no .coderabbit.yaml in repo

Findings · 7

correctness2

high

Setup steps have no timeout — hung layer build yields 'cancelled', not 'failure'

.github/workflows/release-promote.yml:226

The rollback gate checks 'needs.cdk-deploy.result == failure'. The step-level timeouts on CDK diff (15 min) and CDK deploy (45 min) correctly produce 'failure' when they fire. But the setup steps before them — checkout, pnpm install, AWS creds, CFE captcha layer build (onnxruntime+ONNX from S3), CFE browser layer build (chromium) — have no individual timeout-minutes. If any hangs indefinitely, the 75-min job ceiling fires as 'cancelled' (not 'failure'), and the rollback gate is silently bypassed — exactly the scenario the step-timeout design was meant to prevent. The two layer build steps are the highest-risk candidates (external S3 fetch, native binary install).

info

Deploy timing telemetry absent on failure paths — calibration data only from green runs

.github/workflows/release-promote.yml:322

'echo CDK deploy took ${SECONDS}s' only runs if CDK exits 0. A mid-deploy timeout or error means no timing is logged. This means the evidence base for future timeout calibration accumulates only from successful (fast) deploys, missing the near-timeout cases that motivated the change.

tests1

info

No actionlint in CI — workflow file changes get no static validation

.github/workflows/

pr-checks.yml runs typecheck, eslint, and migration-lint but has no actionlint step. Workflow YAML changes receive no static validation beyond YAML parse. Actionlint would catch expression syntax errors and invalid references. Low urgency for this change (pure numeric values), but worth wiring once to guard future workflow edits.

improvement2

medium

$SECONDS measures CDK deploy step shell lifetime — correct but misleading if the run block gains a preamble

.github/workflows/release-promote.yml:322

$SECONDS is a bash builtin counting seconds since the current shell started. Since each 'run:' step spawns a fresh shell, $SECONDS at the end of the deploy step effectively measures that step's wall time — which is correct for the stated intent. However, if the run block ever gains a preamble (env exports, pre-checks) above 'pnpm run deploy', $SECONDS will silently over-count the deploy. A more explicit and resilient form: 'START=$SECONDS; pnpm run deploy …; echo "CDK deploy took $((SECONDS - START))s"'. Also note: $SECONDS is absent on failure (the echo runs only if CDK exits 0), so timing data won't accumulate for near-timeout failures — the cases most relevant for future calibration.

low

CDK diff step lacks timing telemetry symmetric with the deploy step

.github/workflows/release-promote.yml:308

The deploy step emits 'CDK deploy took ${SECONDS}s' for evidence-based tuning. The diff step — which can take several minutes on a large stack set (synthesis + asset hashing for all Batu*-prod stacks) — has no equivalent. If the diff step eventually hits its 15-min bound, there will be no duration history to inform whether 15 min is tight or generous. Adding 'echo CDK diff took ${SECONDS}s | tee -a $GITHUB_STEP_SUMMARY' at the end of the diff step's run block would give symmetric telemetry.

correctness+conventions+improvement1

medium

Stale comment: 'diff 10 / deploy 45' and '60-min ceiling' contradict actual values (15 and 75)

.github/workflows/release-promote.yml:297

The comment added to the CDK diff step reads: 'Step timeouts (diff 10 / deploy 45) keep a stuck CDK inside the job's 60-min ceiling'. The diff step timeout (line 301) is 15 (raised from 10 in the final r2 commit), and the job ceiling (line 224) is 75 (not 60). The job-level comment two dozen lines above correctly states 'diff (15)' and 75, so the two comments directly contradict each other. Anyone triaging a future timeout against this comment will see the wrong arithmetic. Fix: update the step comment to 'diff 15 / deploy 45' and '75-min ceiling'.

correctness+improvement1

low

3-minute buffer between step-bound sum (72 min) and job ceiling (75 min) is thin

.github/workflows/release-promote.yml:224

The comment's own arithmetic: ~12 min setup + 15 min diff + 45 min deploy = ~72 min vs. 75-min ceiling. The ~12 min estimate is one data point from the incident, not a measured p99. A cold pnpm cache, a slow OIDC endpoint, or S3 throttle during a layer build can push setup past 15 minutes. If setup takes 16+ min, the remaining budget (75 - 16 = 59 min) is still above the step bounds (15 + 45 = 60) — which actually means the deploy step timeout fires at minute 76, exceeding the job ceiling and yielding 'cancelled'. Consider raising the job ceiling to 90 min for adequate headroom, or adding step-level timeouts to the two layer-build steps.

History · 2 commits

  1. f484979needs attentionfull1H · 2M · 2L2026-07-23 06:02current
  2. 17dfa91needs attentionincremental0H · 3M · 3L2026-07-23 05:56