fix/cdk-timeout
needs attentionviewing older commit17dfa91 · incrementalPR #342reviewed 2026-07-23 05:56 UTC0H · 3M · 3L · 2I- Purpose
- Fix a prod CDK deploy job timeout that fired at 30 min during the 2026-07-23 promote (41/43 stacks), leaving in-flight CloudFormation stacks and silently skipping the rollback gate
- Goal
- Raise the CDK job timeout to 60 min and add step-level timeouts so overruns surface as FAILURE (rollback-catchable) not CANCELLED (ignored by the rollback gate)
- Sub-goals
- SG-1: Raise job-level timeout-minutes from 30 to 60
- SG-2: Add step-level timeouts (diff: 10 min, deploy: 45 min) so hangs surface as FAILURE
- SG-3: Add deploy duration telemetry for evidence-based future calibration
- SG-4: Document CloudFormation mid-deploy recovery procedure
- What
- Single file: `.github/workflows/release-promote.yml`. Job-level CDK timeout 30→60 min; step-level timeout-minutes added to CDK diff (10 min) and CDK deploy (45 min); recovery note comment; `echo "CDK deploy took ${SECONDS}s"` appended to deploy step for telemetry.
- Why
- A legitimate 30-min ceiling breach during normal prod promote. A job-level timeout fires as `cancelled` which the rollback gate's `if:` deliberately ignores — rollback silently doesn't run. Step-level timeouts fire as `failure`, which the rollback gate catches.
- Areas
- .github/workflows/release-promote.yml+18−2
- Blast
- 1 file, +18/-2 lines. CI/CD only — no application code, no schema, no infrastructure changes.
Findings · 8
correctness2
Step timeout budget (55 min) + pre-CDK steps can still exceed 60-min job ceiling
.github/workflows/release-promote.yml:298
CDK diff (10 min) + CDK deploy (45 min) = 55 min of step maximums. Checkout, pnpm/node setup, dep install, AWS configure, SSM seed, CFE captcha + browser layer builds all run before CDK. If pre-CDK work consumes >5 min, the 60-min ceiling fires as `cancelled`, bypassing the rollback gate. Consider raising the job ceiling to 75 min or reducing the diff step timeout to 5 min.
`$SECONDS` measures step-local time on GHA (correct), but non-obvious
.github/workflows/release-promote.yml:319
Each GHA `run:` block starts a fresh shell, so `$SECONDS` resets to 0 at step start — the telemetry is accurate. But this is non-obvious. An explicit `START=$(date +%s)` + elapsed computation is self-documenting.
security1
CDK diff output (prod ARNs, account IDs) written to GitHub Step Summary — pre-existing
.github/workflows/release-promote.yml:305
Pre-existing pattern explicitly retained. CDK diff routinely includes AWS account IDs, resource ARNs, and IAM policy statements. Step Summaries are visible to all repo members. Worth a follow-up to sanitize or scope the summary output.
conventions2
Operational recovery runbook embedded in workflow YAML comments
.github/workflows/release-promote.yml:217
The RECOVERY block describes CloudFormation incident recovery steps. This is procedure, not code rationale — belongs in a runbook or wiki, not inline YAML that no one reads during a live incident.
Step-timeout comment over-explains GHA cancelled vs failure semantics
.github/workflows/release-promote.yml:294
The difference between `cancelled` and `failure` exit codes is GHA documentation. One line is sufficient; the rest is mechanics.
tests1
No automated test coverage for timeout values — expected for CI config
.github/workflows/release-promote.yml:212
GHA timeout thresholds are not testable with standard CI tooling. No action required.
improvement2
10-min CDK diff timeout may be tight for a 43-stack prod fleet
.github/workflows/release-promote.yml:298
Synthesis + DescribeStacks for 43 stacks with any throttling can approach 7-8 min on a cold runner. Consider 15 min for safer margin.
No timing telemetry on pre-CDK layer build steps — the root-cause hotspot
.github/workflows/release-promote.yml:276
The incident mentioned CFE layer rebuilds as a contributor. Adding step-level timing to layer build steps would make future ceiling recalibrations evidence-based.