feat/tb-deploy
needs attentionviewing older commit19bac85 · incrementalPR #261reviewed 2026-07-08 22:46 UTC0H · 5M · 6L · 2I- Purpose
- Wire Batu's Tinybird deployment strategy: three long-lived workspaces (batu_dev / batu_stg / batu_prod), ephemeral Forward branches per PR inside dev, per-environment token split, and schema-CD lane.
- Goal
- Automate what was done by hand (workspace bootstrap, runtime tokens, staging/prod Vercel env vars) and add CDK stack teardown on branch-delete to prevent IAM role quota exhaustion.
- Sub-goals
- SG-1: Strategy rule doc + README de-stale; scoped runtime vs deploy token split
- SG-2: Per-env metrics-SSM seed (preview/staging/prod)
- SG-3: Vercel per-env tokens (operator action + docs)
- SG-4: Ephemeral per-PR Tinybird branch + cleanup
- SG-5: Schema-CD — tinybird-staging / tinybird-prod jobs
- SG-6: Fold per-env tokens + schema-CD into infrastructure.md / migration-pipeline.md
- What
- This commit merges origin/main into feat/tb-deploy. The only branch-own changes in the incremental window are to the 2 CI provisioning workflows: (1) preview-deprovision.yml gains OIDC permission, CURL_RETRY hardening, a 20-min timeout, and a major new CDK stack teardown step (enumerate + concurrently delete Batu*-{slug}-dev stacks + clean SSM params); (2) preview-provision.yml gains the same CURL_RETRY env var applied to 5 existing curl calls.
- Why
- IAM role quota exhaustion observed in the dev account (1000-role limit hit during branch CDK deploys). Branches weren't cleaning up their CloudFormation stacks on delete, leaving 1–15 IAM roles per stack orphaned. The new teardown step is the primary remediation.
- Areas
- .github/workflows/preview-deprovision.yml+33−1.github/workflows/preview-provision.yml+169−36.claude/rules/+280−2.github/actions/tb-install+33−0infra/tinybird/+128−58
- Blast
- CI/infra only — 15 files total (+895/-104 lines across the full branch). No application code, no TypeScript, no DB migrations. Risk is confined to CI workflows and CDK config.
Findings · 14
correctness3
Silent false-positive success when `list-stacks` fails
.github/workflows/preview-deprovision.yml
Under `set -uo pipefail` (no -e), if `aws cloudformation list-stacks` fails (throttle, OIDC expiry, network error) the assignment `ALL=$(...)` fails silently: ALL is empty, the for-loop is a no-op, DELETED=0 FAILED=0, and the step exits green. The teardown summary reads as success when nothing was actually deleted. Fix: `ALL=$(aws cloudformation list-stacks ... | tr '\t' '\n') || { echo '::error::list-stacks failed'; exit 1; }`.
Missing in-progress stack statuses from enumeration filter
.github/workflows/preview-deprovision.yml
The `--stack-status-filter` list omits `UPDATE_IN_PROGRESS`, `ROLLBACK_IN_PROGRESS`, and related transient states. A branch deleted while a CDK deploy/update is still running (a realistic race with a manual `/deploy-cdk`) will have its in-flight stacks silently skipped, leaving IAM roles orphaned — the exact failure this teardown was introduced to prevent. Include those statuses in the filter (the subsequent delete-stack request will fail for an in-progress stack, which increments FAILED and surfaces the issue) or document explicitly why they're excluded.
Slug extraction `${base#*-}` is fragile if any CDK capability name ever acquires a hyphen
.github/workflows/preview-deprovision.yml
The extraction relies on the invariant that CDK capability prefixes are hyphen-free PascalCase. All current stacks satisfy this, but it's undocumented at the CDK naming layer. If a future capability like `Batu-Metrics-Api` is introduced, `${base#*-}` would return `Metrics-Api-{slug}-dev` → wrong slug match → silently orphaned stacks. Consider asserting the invariant in `infra/cdk/src/stacks/naming.ts` (throw if capability name contains a hyphen).
security3
`id-token: write` at workflow level rather than job level
.github/workflows/preview-deprovision.yml
The `id-token: write` permission is declared at workflow scope, propagating OIDC token-generation capability to every job. This matches the existing repo pattern (release-promote.yml, staging-update.yml, etc.), so no immediate action needed — but it's worth noting that moving it to the specific job that runs `aws-actions/configure-aws-credentials` would be marginally tighter. Low-blast-radius today; note it as a convention improvement if the workflow grows more jobs.
Protected slug deny-list may miss plausible shared-stack slugs
.github/workflows/preview-deprovision.yml
The guard `case "$SLUG" in main|master|staging|stg|prod|production|dev|management)` is good but misses slugs like `preview`, `infra`, `shared`, `common`. A branch named `preview` would pass the guard and enumerate `Batu*-preview-dev` stacks. Consider adding a minimum slug length check (e.g. require len ≥ 4) or extending the denylist.
`--retry-all-errors` retries on 401/403 — hammers auth endpoints on misconfiguration
.github/workflows/preview-provision.yml
`curl --retry-all-errors` retries on all non-zero exit codes AND all HTTP errors. With `-f`, a 401/403 exits non-zero and triggers retries — 3 attempts with 2-second delay, consuming up to 180 s and generating repeated failed auth attempts. This can trigger rate-limiting or lockout on services with strict brute-force policies. Omit `-f` where you're already checking `$HTTP_CODE`, or limit retries to 5xx only with a wrapper.
conventions1
Slug trim regex diverges from provision despite 'EXACTLY mirrors' comment
.github/workflows/preview-deprovision.yml:248
`preview-provision.yml` (line 715) uses `s|^-+||; s|-\\+$||` (GNU BRE `\\+` = one-or-more) while `preview-deprovision.yml` uses `s|^-*||; s|-*$||` (zero-or-more). Functionally identical for all real branch names, but not textually identical. The inline comment says 'Mirror preview-provision.yml's slug rule EXACTLY' — the comment's claim is stronger than the code. Unify to the same sed expression in both files to prevent silent divergence if a runner uses a stricter BRE dialect.
tests2
No dry-run path for CDK teardown — only testable on a real branch delete
.github/workflows/preview-deprovision.yml
The teardown has no dry-run mode or `workflow_dispatch` trigger. The slug-matching and delete logic can only be exercised by actually deleting a branch. Consider adding a `workflow_dispatch` input (e.g. `dry_run: true`) that logs TARGETS without calling `delete-stack`, giving a safe way to validate enumeration before trusting it against the live dev account.
Manual custom-slug orphan gap is documented inline but not tracked as an open test gap
.github/workflows/preview-deprovision.yml
The inline comment acknowledges the custom-slug gap (BAT-245) and the periodic-sweep plan. The gap is known and tracked. No action needed here beyond what's already documented.
improvement5
`--retry-max-time 180` conflicts with `--max-time 120` on the Supabase binary download
.github/workflows/preview-deprovision.yml
Both files set `CURL_RETRY="--retry-max-time 180 ..."` then call `curl -fsSL $CURL_RETRY --max-time 120` for the Supabase CLI tarball. `--retry-max-time` is the cumulative retry budget; `--max-time` is per-transfer cap. A 120-second download leaves only 60 s of retry budget — at most one retry. Raise `--retry-max-time` to ≥360 (3 × 120) for the binary download, or inline a higher value for that specific call.
SSM cleanup may worsen DELETE_FAILED stacks by removing CFN-managed params externally
.github/workflows/preview-deprovision.yml
The SSM cleanup deletes everything under `/batu/${SLUG}/dev/**` regardless of stack state. CDK stacks own `AWS::SSM::Parameter` resources; when a stack is in DELETE_FAILED, its CFN-managed SSM params still exist. Deleting them outside CloudFormation leaves the stack permanently stuck — the next delete attempt may fail because CloudFormation still lists those params as stack resources. Consider skipping the SSM cleanup when `FAILED > 0`, or scoping it to externally-seeded param names only.
CURL_RETRY defined identically in two workflow files with no sync note
.github/workflows/preview-provision.yml
The CURL_RETRY env string is byte-identical in both `preview-provision.yml` and `preview-deprovision.yml`. GitHub Actions has no cross-workflow env inheritance, so the duplication is inherent. Add a comment in each file noting 'keep in sync with preview-{provision,deprovision}.yml' so a future editor knows to update both.
Manual custom-slug CDK deploys orphan stacks with no alerting or sweep
.github/workflows/preview-deprovision.yml
The script itself documents (inline comment) that stacks deployed with a manually-shortened slug via `/deploy-cdk` won't be matched by the auto-slug teardown (BAT-245). This is the exact quota-exhaustion failure this teardown was introduced to prevent. Without a periodic sweep job or IAM role count alert, the quota edge could be reached silently. Consider a nightly cron querying `aws iam list-roles | wc -l` against a warning threshold.
`IMPORT_COMPLETE` in status filter is likely dead weight for branch preview stacks
.github/workflows/preview-deprovision.yml
Branch preview stacks are created entirely by CDK and never imported via `aws cloudformation import-stacks-to-changeset`. Including `IMPORT_COMPLETE` in the filter adds no coverage and could confuse a future reader into thinking branch imports are expected. Removing it would make the filter's intent clearer.
History · 10 commits
- 5d30429safeincremental0H · 0M · 1L2026-07-08 23:04
- 19bac85needs attentionincremental0H · 5M · 6L2026-07-08 22:46current
- 92e890aneeds attentionincremental0H · 1M · 3L2026-07-08 21:46
- e81a781needs attentionincremental0H · 4M · 4L2026-07-08 21:31
- 2eca91cneeds attentionfull5H · 13M · 12L2026-07-08 19:18
- cf63bb1needs attentionincremental4H · 4M · 7L2026-07-07 19:47
- f43d034needs attentionincremental2H · 3M · 3L2026-07-07 18:35
- b7800c0needs attentionincremental1H · 2M · 7L2026-07-07 03:31
- 5aef105needs attentionfull6H · 9M · 2L2026-07-06 17:52
- 95b4484blockedfull3H · 8M · 4L2026-07-06 05:36