fix/cdk-deprov
needs attentionviewing older commit5ec41e1 · incrementalPR #193reviewed 2026-07-07 03:23 UTC3H · 4M · 6L · 5I- Purpose
- Fix critical IAM role accumulation in the dev AWS account — every preview branch deployed 8–14 CloudFormation stacks with 1–15 IAM roles each, but branch deletion had no AWS teardown. Account hit 999/1000 IAM roles, blocking new preview deploys with ServiceLimitExceeded.
- Goal
- Add OIDC-authenticated AWS teardown to preview-deprovision.yml and harden preview-provision.yml against flaky transient failures.
- Sub-goals
- SG-1: Derive slug from deleted branch using exact same slugification rule as provision, enumerate Batu*-{slug}-dev stacks, delete each and wait for completion
- SG-2: Best-effort remove /batu/{slug}/dev/** SSM parameter namespace on branch delete
- SG-3: Add curl retry hardening to preview-provision.yml for flaky network calls (this commit)
- What
- This incremental commit (5ec41e13) adds `--retry 5 --retry-all-errors --retry-delay 2 --connect-timeout 15 --max-time N` to 6 curl calls in preview-provision.yml: the Supabase CLI version-fetch and binary download, the Supabase auth PATCH, and 4 Vercel API POSTs.
- Why
- Flaky transient network failures in the provision workflow were causing CI runs to fail non-deterministically. The main PR adds a full CDK stack teardown on branch delete (preview-deprovision.yml, +214/-1 lines); this commit hardens the provision side against the same class of failures.
- Areas
- .github/workflows/preview-deprovision.yml+214−1.github/workflows/preview-provision.yml+7−7
- Blast
- 2 files, +221/−8 across .github/workflows. CI-only change — no application code, no database, no CDK stacks, no API contracts.
Findings · 18
correctness4
--retry-all-errors on Supabase PATCH retries 4xx auth failures — up to 370s wasted
.github/workflows/preview-provision.yml:272
--retry-all-errors retries on every HTTP error including 401 Unauthorized and 403 Forbidden. If SUPABASE_ACCESS_TOKEN is missing or expired, the PATCH /config/auth will be retried 5 times with --max-time 60 per attempt (worst case 6×60 + 5×2 = 370s) before the step fails. The actual error (bad token vs. transient network failure) is masked — only the last retry's response is visible. For an authenticated PATCH, --retry-connrefused or --retry with explicit HTTP code filtering (500,502,503,504) would be more appropriate than retrying on all errors including 4xx.
--retry-all-errors on Vercel deployments POST may create duplicate deployments on 5xx
.github/workflows/preview-provision.yml:475
POST /v13/deployments is retried on 5xx by --retry-all-errors. If Vercel returns a 5xx after the deployment was actually created server-side (a common pattern with creation APIs), a retry triggers a second deployment with the same gitSource.sha. Vercel's deduplication behavior is not documented as a hard guarantee. The same pattern appears at line 1058. A safer approach: retry only on connection errors (--retry-connrefused) or check for an existing deployment before creating.
Worst-case retry timing can exhaust the 45-minute job timeout with CDK
.github/workflows/preview-provision.yml:440
--max-time is per attempt, not total. With --retry 5 and --retry-delay 2, each --max-time 90 call can take up to 5×(90+2)=460s (~7.7 min). There are 5 such calls in the provision workflow (lines 272, 440, 475, ~1023, 1058). In the adversarial case where multiple calls each hit all retries, total curl time can exceed 45 min — especially combined with CDK deploy (~25 min). Consider --max-time 30 for API calls (not binary downloads) or reducing to --retry 3.
GitHub API version check lacks User-Agent — rate-limited IPs retry 5× before failing
.github/workflows/preview-provision.yml:70
GitHub's REST API requires a User-Agent header and rate-limits unauthenticated requests to 60/hour per IP. GitHub Actions runners share IP pools. If the runner IP is rate-limited, the API returns 403. With --retry-all-errors and --max-time 30, this retries 5× (up to 190s) before failing. If LATEST_VERSION ends up empty, the download URL becomes malformed (v//supabase...) and tar fails. Pre-existing issue; retry addition widens the failure window. Fix: add -H 'User-Agent: batu-preview-provision' and authenticate with the $GITHUB_TOKEN.
security4
Supabase CLI binary downloaded without checksum or signature verification
.github/workflows/preview-provision.yml:71
The binary is downloaded from GitHub Releases and extracted directly with no integrity check (sha256sum, cosign, or GPG). The retry logic introduces a subtle new risk: LATEST_VERSION is resolved in one curl call; the tarball is downloaded in a second. If GitHub releases a new version between the two requests, the installed binary silently diverges from the resolved version string. There is no checksum gate to detect this. Mitigation: pin to a fixed version or verify the tarball against the release's .sha256 file before extraction.
--retry-all-errors re-sends Bearer tokens on every 4xx retry
.github/workflows/preview-provision.yml:272
All three authenticated curl calls carry --retry-all-errors, causing the full Authorization: Bearer header to be sent on each retry attempt including 401/403 responses. This increases the observable footprint in server-side access logs and delays detection of a misconfigured token. Not an exploitable vulnerability (the token is already in memory and used on the first attempt), but --retry-http-codes 429,500,502,503,504 would be a more precise and secure alternative.
LATEST_VERSION extracted via grep/sed from unvalidated API response — no format guard
.github/workflows/preview-provision.yml:70
The version string from the GitHub API is used directly in a URL path segment without a format validation. The sed regex ([^"]+) restricts to non-quote characters but doesn't enforce semver format. A [[ $LATEST_VERSION =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] guard before the download would fail fast on a malformed response. Pre-existing issue; flagging as defense-in-depth.
No new credential exposure introduced by this diff
.github/workflows/preview-provision.yml:272
All secrets (SUPABASE_ACCESS_TOKEN, VERCEL_TOKEN) are sourced from secrets.* context and passed as HTTP headers, not as environment variables visible in process listings. The retry flags do not change how credentials are stored or logged within GitHub Actions. The -sS flag suppresses progress but retains error messages — response bodies are not echoed. No new secrets introduced.
conventions5
preview-deprovision.yml has 6 bare curl calls not hardened
.github/workflows/preview-deprovision.yml:43
The companion deprovisioner (the main PR work — AWS teardown) has 6 bare curl calls (lines 43, 44, 76, 99, 128, 145) with no --retry, --connect-timeout, or --max-time. Lines 43–44 mirror the exact Supabase CLI install pattern hardened in provision.yml. Lines 76, 99, 128, 145 are Vercel API calls using bare `curl -s` (silent — swallows errors). Since deprovision runs on branch deletion, transient failures leave orphaned Vercel resources. The hardening applied to provision.yml should be applied symmetrically.
Inconsistent flag ordering across curl calls
.github/workflows/preview-provision.yml:272
On line 272, retry flags are inserted between -sS and -o/-w output flags. On lines 440/475/1023/1058, they appear before -w and -X. No codified convention, but the de-facto grouping (mode flags → retry/timeout → output/method flags → URL) is not consistently followed. Minor but worth standardising if a CURL_RETRY env var approach is adopted.
--retry-all-errors behavior on 4xx is non-intuitive for readers
.github/workflows/preview-provision.yml:440
When used with -w '%{http_code}', a 4xx response does NOT set curl's error exit code — so retry won't actually fire on 409/404/422 from Vercel. The flag is therefore safer than it looks, but a comment explaining 'retries on connection errors and 5xx only (4xx don't trigger curl error exit code with -w)' would prevent future confusion when someone reads --retry-all-errors and assumes all HTTP errors are retried.
--max-time values are appropriate per use-case
.github/workflows/preview-provision.yml:70
30s for GitHub API JSON (version check), 120s for a ~50MB binary download, 60s for Supabase management PATCH, 90s for Vercel API POSTs — all appropriate. No anomalies.
--retry-all-errors is safe on current GHA runners (curl 7.81+)
.github/workflows/preview-provision.yml:70
--retry-all-errors requires curl >= 7.71.0 (June 2020). ubuntu-latest ships curl 7.81+ (Ubuntu 22.04) / 8.5+ (Ubuntu 24.04). The self-hosted runner has curl 8.17.0. No compatibility risk.
tests1
No unit tests — acceptable for CI workflow hardening
.github/workflows/preview-provision.yml:1
GHA workflows are not unit-testable in a meaningful way. The retry behavior is a curl built-in and trust in it is reasonable. No act-based test infrastructure exists in the repo (normal for this scale). Correct validation is observing flaky-failure rates drop in production runs.
improvement4
DRY: identical curl retry flags repeated 7 times
.github/workflows/preview-provision.yml:70
The string `--retry 5 --retry-all-errors --retry-delay 2 --connect-timeout 15` appears identically on all 7 curl invocations (lines 70, 71, 272, 440, 475, 1023, 1058). Maintenance debt: changing the retry count or timeout requires updating every line. Idiomatic GHA fix: a job-level env var `CURL_RETRY: "--retry 5 --retry-all-errors --retry-delay 2 --connect-timeout 15"` used as `curl $CURL_RETRY ...`. Centralises all instances and makes the intent explicit.
Missing --retry-max-time to bound total retry window
.github/workflows/preview-provision.yml:440
--max-time caps a single attempt but the total retry sequence is unbounded except by the job timeout. Adding --retry-max-time 300 would stop retrying after 5 minutes total regardless of remaining attempts. Useful for surfacing stuck steps sooner without changing per-attempt behavior.
Fixed --retry-delay 2 suboptimal for API rate-limit (429) scenarios
.github/workflows/preview-provision.yml:272
A 2-second fixed delay is fine for transient TCP errors but too short for API rate-limit (HTTP 429) scenarios where the server's retry-after may be 60s+. curl doesn't natively support exponential backoff, but a shell wrapper loop with sleep $((2 ** attempt)) would handle 429s more gracefully for the Vercel/Supabase calls. Minor — 429 is not the primary failure mode being fixed here.
--retry-all-errors compatibility confirmed — no action needed
.github/workflows/preview-provision.yml:70
GHA ubuntu-latest and the self-hosted runner both have curl 8.x, well above the 7.71.0 minimum for --retry-all-errors. Document if runner images are ever pinned to older Ubuntu versions.
History · 6 commits
- d159446needs attentionincremental1H · 5M · 7L2026-07-07 03:47
- 5ec41e1needs attentionincremental3H · 4M · 6L2026-07-07 03:23current
- 9814d4dneeds attentionincremental1H · 3M · 5L2026-07-05 05:43
- 9e87f75needs attentionincremental0H · 1M · 2L2026-07-05 05:17
- 02936b3blockedincremental1H · 0M · 4L2026-07-05 04:53
- 4407258needs attentionfull0H · 3M · 4L2026-07-05 04:27