fix/cdk-deprov
needs attentionviewing older commit9814d4d · incrementalPR #193reviewed 2026-07-05 05:43 UTC1H · 3M · 5L · 1I- Purpose
- Fix IAM role quota exhaustion in the dev AWS account caused by orphaned CloudFormation stacks from deleted preview branches (was at 999/1000 roles, blocking new previews).
- Goal
- Make preview-deprovision.yml reliably tear down all branch-scoped CDK stacks on branch delete, with good observability when teardown fails.
- Sub-goals
- SG-1: OIDC-authenticated AWS teardown + slug derivation matching provision logic
- SG-2: Parallel stack deletion with concurrent waiters
- SG-3: Protected-slug guard + shared-stack exclusion safety
- SG-4 (this commit): Improve failure diagnostics — capture waiter stderr, log exit code, two-phase event lookup
- What
- The waiter's 2>/dev/null is replaced with 2>&1 variable capture, exit code is surfaced in the warning, and a two-phase event lookup (DELETE_FAILED then DELETE_IN_PROGRESS fallback) replaces the single DELETE_FAILED query.
- Why
- Round 3 review feedback: previous waiter suppressed stderr making failures completely opaque in CI logs.
- Areas
- .github/workflows/preview-deprovision.yml+210−1
- Blast
- 1 file, +210/-1 (branch total); CI-only change, no application code affected.
Findings · 11
correctness3
werr captures stdout + stderr — success stdout silently dropped
.github/workflows/preview-deprovision.yml:294
2>&1 merge is cosmetic-only; if-condition driven by exit code.
DELETE_IN_PROGRESS fallback may return unbounded output on timeout
.github/workflows/preview-deprovision.yml:307
Add --max-items 20 or | head -20 to keep diagnostics compact.
Timed-out stack may complete before describe-stack-events is called
.github/workflows/preview-deprovision.yml:304
Inherent to polling model; stack may finish after waiter gives up, leaving $ev empty.
security2
ResourceStatusReason logged unconditionally — may contain secrets
.github/workflows/preview-deprovision.yml:304
CFN failure reasons can include plaintext env vars, passwords, policy docs. Drop the field or accept risk.
Waiter stderr capture may include transient credential diagnostics
.github/workflows/preview-deprovision.yml:295
Low risk on OIDC runners; mid-wait role expiry could expose session ARN.
conventions3
describe-stack-events calls have no timeout guard
.github/workflows/preview-deprovision.yml:307
A throttled or hung AWS API call will block the subshell's wait slot indefinitely; wrap both calls with timeout 30.
::warning:: annotation used for failure that escalates to ::error::
.github/workflows/preview-deprovision.yml:299
Creates inconsistent signals in GHA annotations; use ::error:: instead.
Comment claims rc distinguishes failure types but no code branches on it
.github/workflows/preview-deprovision.yml:293
Soften comment or add branching on rc=255 vs rc=124.
improvement3
Two describe-stack-events calls could merge into one with OR query
.github/workflows/preview-deprovision.yml:304
Single API call with StackEvents[?ResourceStatus=='DELETE_FAILED' || ResourceStatus=='DELETE_IN_PROGRESS'] reduces latency.
rc=$? inside else block — fragile if a statement is inserted before it
.github/workflows/preview-deprovision.yml:298
Restructure as: timeout … ; rc=$?; if [ $rc -eq 0 ]
Waiter stderr not prefixed with stack name — hard to attribute in concurrent output
.github/workflows/preview-deprovision.yml:300
Prefix with [$s] for attribution in concurrent multi-stack teardowns.
History · 6 commits
- d159446needs attentionincremental1H · 5M · 7L2026-07-07 03:47
- 5ec41e1needs attentionincremental3H · 4M · 6L2026-07-07 03:23
- 9814d4dneeds attentionincremental1H · 3M · 5L2026-07-05 05:43current
- 9e87f75needs attentionincremental0H · 1M · 2L2026-07-05 05:17
- 02936b3blockedincremental1H · 0M · 4L2026-07-05 04:53
- 4407258needs attentionfull0H · 3M · 4L2026-07-05 04:27