fix/cdk-deprov
needs attentionviewing older commit4407258 · fullPR #193reviewed 2026-07-05 04:27 UTC0H · 3M · 4L · 2I- Purpose
- The dev AWS account's 1000-IAM-roles-per-account quota was exhausted (999/1000) because preview-provision.yml deployed 8–14 Batu*-{slug}-dev CloudFormation stacks per branch but preview-deprovision.yml had no AWS teardown — leaving every stack and its IAM roles orphaned on branch delete.
- Goal
- Add OIDC-authenticated AWS teardown to preview-deprovision.yml: enumerate the branch's Batu*-{slug}-dev stacks, delete each, and best-effort clean up the /batu/{slug}/dev SSM namespace.
- Sub-goals
- SG-1: Derive slug from deleted branch using exact same rule as preview-provision.yml
- SG-2: Enumerate and delete matching CloudFormation stacks (not cdk destroy — avoids BatuTerraformState and layer-asset build requirement)
- SG-3: Add protected-slug guard and exact-match filter to prevent accidental shared-stack deletion
- SG-4: Best-effort SSM parameter cleanup under /batu/{slug}/dev
- SG-5: OIDC credentials + if:always() so teardown runs even if Supabase/Vercel steps fail
- What
- preview-deprovision.yml gains OIDC permissions block, timeout bump 10→20 min, and two new steps: AWS credential config + a bash teardown script.
- Why
- IAM role quota at 999/1000 was blocking all new preview deploys. 169 stacks across 23 deleted branches were manually swept to recover quota; this PR prevents recurrence.
- Areas
- .github/workflows/preview-deprovision.yml+145−1
- Blast
- 1 file, +145/-1 lines. CI/CD only — no application code, no schema, no CDK constructs changed.
Findings · 9
correctness1
Shared-stack guard relies on undocumented no-hyphen invariant for CDK capability names
.github/workflows/preview-deprovision.yml
The shared-stack detection `[ "$slug" = "$base" ] && continue` works because PascalCase capability names (e.g. `BatuCfeLambda`) contain no hyphens — `${base#*-}` returns the full string unchanged when there is no `-`, and the equality check correctly skips the stack. This invariant is not documented in the code. If any future CDK stack were named with a hyphenated prefix (e.g. `Batu-Cfe-Bills-dev`), `${base#*-}` would yield `Cfe-Bills` (not equal to base), the guard would fail to skip it, and `CfeBills` would be compared against SLUG. The CDK naming convention prevents this today, but a comment noting the dependency (`# assumes Batu{PascalCase}-dev — no hyphens in capability names`) would make the invariant explicit.
security1
`id-token: write` scoped at workflow level instead of job level
.github/workflows/preview-deprovision.yml:7
Declaring `id-token: write` at the workflow level grants the OIDC token capability to every step in every job. The provision workflow uses the same pattern, so this is consistent, but the least-privilege best practice is to scope it at the job level (`jobs.cleanup.permissions`). Given the workflow has only one job and no untrusted third-party actions in the critical path, practical risk is low — flagged for defense-in-depth.
conventions1
Step name casing inconsistent with `preview-provision.yml`
.github/workflows/preview-deprovision.yml
New step: `Configure AWS credentials (OIDC, dev account)`. Provision uses: `Configure AWS Credentials (OIDC)` (title-case 'Credentials'). Minor, but log scanning is easier with consistent naming.
tests1
No shellcheck/actionlint gate — slug and stack-matching logic are untested
.github/workflows/preview-deprovision.yml
The slug computation and slug-extraction logic are pure string transformations testable with ~10 bats or inline bash assertions (normal branch, uppercase, double-slash, protected-name collision, shared-stack pattern). No shellcheck or actionlint runs in CI for this repo. `actionlint` in particular catches GHA expression issues and shell quoting bugs that reviewers miss. Adding it as a simple CI step would surface regressions on future edits to workflow scripts.
improvement3
Teardown summary always says 'complete' regardless of failures
.github/workflows/preview-deprovision.yml
The final `echo "Branch-scoped AWS teardown complete for slug '$SLUG'."` is always emitted, even when one or more `delete-stack` requests failed and some stacks did not reach DELETE_COMPLETE. Log scanning for 'complete' gives a false green. Track a counter (`failed=0; … failed=$((failed+1))`) and emit a structured summary: `Deleted: N Failed: M`. Exit with a non-zero code when M > 0 so the GHA step itself is marked failed and CI turns red.
SSM delete errors completely silenced — should emit `::warning::` on failure
.github/workflows/preview-deprovision.yml
Each `aws ssm delete-parameter` failure is caught by `>/dev/null 2>&1 || true`, producing no signal. A permissions error (the deploy role may lack `ssm:DeleteParameter`), a throttle, or a malformed path is indistinguishable from success. Change to `aws ssm delete-parameter --name "$p" && echo " removed SSM $p" || echo "::warning::failed to remove SSM $p"` — consistent with how `delete-stack` failures are handled above.
No `$GITHUB_STEP_SUMMARY` for teardown results
.github/workflows/preview-deprovision.yml
The teardown outcome (slug, stacks deleted/failed, SSM params removed) is only in the raw log. GitHub Actions supports `$GITHUB_STEP_SUMMARY` (Markdown, persistent per-run). A small table written there (`| Stack | Outcome |`) makes post-delete audits fast without log diving — especially useful for the IAM quota scenario that motivated this fix.
correctness+improvement2
DELETE_FAILED reason silenced — operator has no diagnostic on stack deletion failure
.github/workflows/preview-deprovision.yml
`aws cloudformation wait stack-delete-complete --stack-name "$s" 2>/dev/null` discards all stderr. When a stack lands in DELETE_FAILED (a retained resource, a dependency, or a non-empty S3 bucket the autoDeleteObjects custom resource failed to drain), the only log line is `::warning:: … inspect manually` with zero context. After failure, emit the blocking events: `aws cloudformation describe-stack-events --stack-name "$s" --query "StackEvents[?contains(ResourceStatus,'FAILED')].[LogicalResourceId,ResourceStatusReason]" --output table`. This one line gives operators the resource name and reason without requiring a console login.
Sequential `wait` calls can exhaust the 20-min job timeout before all stacks are confirmed
.github/workflows/preview-deprovision.yml
Deletions are fired in parallel server-side, then confirmed sequentially. The AWS CLI `stack-delete-complete` waiter defaults to 120 retries × 30 s = 60 min per stack. If the first stack in the loop is slow (e.g., emptying a large S3 bucket), its wait can consume most or all of the 20-min job budget, leaving the remaining waits unrun and their stacks in an unconfirmed state. The stacks continue deleting server-side, but the job exits without confirming or logging them. Bounding the waiter (`--delay 15 --max-attempts 75` = 18.75 min) or parallelising the waits (background `&` each + `wait -n`) prevents silent timeout-truncation.
History · 6 commits
- d159446needs attentionincremental1H · 5M · 7L2026-07-07 03:47
- 5ec41e1needs attentionincremental3H · 4M · 6L2026-07-07 03:23
- 9814d4dneeds attentionincremental1H · 3M · 5L2026-07-05 05:43
- 9e87f75needs attentionincremental0H · 1M · 2L2026-07-05 05:17
- 02936b3blockedincremental1H · 0M · 4L2026-07-05 04:53
- 4407258needs attentionfull0H · 3M · 4L2026-07-05 04:27current