← all branches

fix/cdk-deprov

needs attentionviewing older commit
4407258 · fullPR #193reviewed 2026-07-05 04:27 UTC0H · 3M · 4L · 2I
The branch
Purpose
The dev AWS account's 1000-IAM-roles-per-account quota was exhausted (999/1000) because preview-provision.yml deployed 8–14 Batu*-{slug}-dev CloudFormation stacks per branch but preview-deprovision.yml had no AWS teardown — leaving every stack and its IAM roles orphaned on branch delete.
Goal
Add OIDC-authenticated AWS teardown to preview-deprovision.yml: enumerate the branch's Batu*-{slug}-dev stacks, delete each, and best-effort clean up the /batu/{slug}/dev SSM namespace.
Sub-goals
  • SG-1: Derive slug from deleted branch using exact same rule as preview-provision.yml
  • SG-2: Enumerate and delete matching CloudFormation stacks (not cdk destroy — avoids BatuTerraformState and layer-asset build requirement)
  • SG-3: Add protected-slug guard and exact-match filter to prevent accidental shared-stack deletion
  • SG-4: Best-effort SSM parameter cleanup under /batu/{slug}/dev
  • SG-5: OIDC credentials + if:always() so teardown runs even if Supabase/Vercel steps fail
The changes (whole branch)
What
preview-deprovision.yml gains OIDC permissions block, timeout bump 10→20 min, and two new steps: AWS credential config + a bash teardown script.
Why
IAM role quota at 999/1000 was blocking all new preview deploys. 169 stacks across 23 deleted branches were manually swept to recover quota; this PR prevents recurrence.
Areas
.github/workflows/preview-deprovision.yml+1451
Blast
1 file, +145/-1 lines. CI/CD only — no application code, no schema, no CDK constructs changed.
ci-only no-app-code
CI checks· gh pr checks not accessible via this tokenCodeRabbit· no .coderabbit.yaml in repo

Findings · 9

correctness1

info

Shared-stack guard relies on undocumented no-hyphen invariant for CDK capability names

.github/workflows/preview-deprovision.yml

The shared-stack detection `[ "$slug" = "$base" ] && continue` works because PascalCase capability names (e.g. `BatuCfeLambda`) contain no hyphens — `${base#*-}` returns the full string unchanged when there is no `-`, and the equality check correctly skips the stack. This invariant is not documented in the code. If any future CDK stack were named with a hyphenated prefix (e.g. `Batu-Cfe-Bills-dev`), `${base#*-}` would yield `Cfe-Bills` (not equal to base), the guard would fail to skip it, and `CfeBills` would be compared against SLUG. The CDK naming convention prevents this today, but a comment noting the dependency (`# assumes Batu{PascalCase}-dev — no hyphens in capability names`) would make the invariant explicit.

security1

low

`id-token: write` scoped at workflow level instead of job level

.github/workflows/preview-deprovision.yml:7

Declaring `id-token: write` at the workflow level grants the OIDC token capability to every step in every job. The provision workflow uses the same pattern, so this is consistent, but the least-privilege best practice is to scope it at the job level (`jobs.cleanup.permissions`). Given the workflow has only one job and no untrusted third-party actions in the critical path, practical risk is low — flagged for defense-in-depth.

conventions1

low

Step name casing inconsistent with `preview-provision.yml`

.github/workflows/preview-deprovision.yml

New step: `Configure AWS credentials (OIDC, dev account)`. Provision uses: `Configure AWS Credentials (OIDC)` (title-case 'Credentials'). Minor, but log scanning is easier with consistent naming.

tests1

info

No shellcheck/actionlint gate — slug and stack-matching logic are untested

.github/workflows/preview-deprovision.yml

The slug computation and slug-extraction logic are pure string transformations testable with ~10 bats or inline bash assertions (normal branch, uppercase, double-slash, protected-name collision, shared-stack pattern). No shellcheck or actionlint runs in CI for this repo. `actionlint` in particular catches GHA expression issues and shell quoting bugs that reviewers miss. Adding it as a simple CI step would surface regressions on future edits to workflow scripts.

improvement3

medium

Teardown summary always says 'complete' regardless of failures

.github/workflows/preview-deprovision.yml

The final `echo "Branch-scoped AWS teardown complete for slug '$SLUG'."` is always emitted, even when one or more `delete-stack` requests failed and some stacks did not reach DELETE_COMPLETE. Log scanning for 'complete' gives a false green. Track a counter (`failed=0; … failed=$((failed+1))`) and emit a structured summary: `Deleted: N Failed: M`. Exit with a non-zero code when M > 0 so the GHA step itself is marked failed and CI turns red.

low

SSM delete errors completely silenced — should emit `::warning::` on failure

.github/workflows/preview-deprovision.yml

Each `aws ssm delete-parameter` failure is caught by `>/dev/null 2>&1 || true`, producing no signal. A permissions error (the deploy role may lack `ssm:DeleteParameter`), a throttle, or a malformed path is indistinguishable from success. Change to `aws ssm delete-parameter --name "$p" && echo " removed SSM $p" || echo "::warning::failed to remove SSM $p"` — consistent with how `delete-stack` failures are handled above.

low

No `$GITHUB_STEP_SUMMARY` for teardown results

.github/workflows/preview-deprovision.yml

The teardown outcome (slug, stacks deleted/failed, SSM params removed) is only in the raw log. GitHub Actions supports `$GITHUB_STEP_SUMMARY` (Markdown, persistent per-run). A small table written there (`| Stack | Outcome |`) makes post-delete audits fast without log diving — especially useful for the IAM quota scenario that motivated this fix.

correctness+improvement2

medium

DELETE_FAILED reason silenced — operator has no diagnostic on stack deletion failure

.github/workflows/preview-deprovision.yml

`aws cloudformation wait stack-delete-complete --stack-name "$s" 2>/dev/null` discards all stderr. When a stack lands in DELETE_FAILED (a retained resource, a dependency, or a non-empty S3 bucket the autoDeleteObjects custom resource failed to drain), the only log line is `::warning:: … inspect manually` with zero context. After failure, emit the blocking events: `aws cloudformation describe-stack-events --stack-name "$s" --query "StackEvents[?contains(ResourceStatus,'FAILED')].[LogicalResourceId,ResourceStatusReason]" --output table`. This one line gives operators the resource name and reason without requiring a console login.

medium

Sequential `wait` calls can exhaust the 20-min job timeout before all stacks are confirmed

.github/workflows/preview-deprovision.yml

Deletions are fired in parallel server-side, then confirmed sequentially. The AWS CLI `stack-delete-complete` waiter defaults to 120 retries × 30 s = 60 min per stack. If the first stack in the loop is slow (e.g., emptying a large S3 bucket), its wait can consume most or all of the 20-min job budget, leaving the remaining waits unrun and their stacks in an unconfirmed state. The stacks continue deleting server-side, but the job exits without confirming or logging them. Bounding the waiter (`--delay 15 --max-attempts 75` = 18.75 min) or parallelising the waits (background `&` each + `wait -n`) prevents silent timeout-truncation.

History · 6 commits

  1. d159446needs attentionincremental1H · 5M · 7L2026-07-07 03:47
  2. 5ec41e1needs attentionincremental3H · 4M · 6L2026-07-07 03:23
  3. 9814d4dneeds attentionincremental1H · 3M · 5L2026-07-05 05:43
  4. 9e87f75needs attentionincremental0H · 1M · 2L2026-07-05 05:17
  5. 02936b3blockedincremental1H · 0M · 4L2026-07-05 04:53
  6. 4407258needs attentionfull0H · 3M · 4L2026-07-05 04:27current