fix/ph-tf-alert

safe

efa720b · fullPR #368reviewed 2026-07-31 18:36 UTC0H · 0M · 2L · 4I

The branch

Purpose

Fix two defects that caused a failed Terraform apply to roll back a successful Vercel deployment to production on 2026-07-31 (BAT-289).

Goal

Unblock future terraform applies and prevent analytics IaC failures from reverting healthy prod app deployments.

Sub-goals

Fix A: Add check_ongoing_interval=true to the only hourly PostHog alert so provider response validation agrees with the plan
Fix B: Remove posthog-apply from postflight needs and rollback trigger so a PostHog IaC failure cannot revert the app

The changes (whole branch)

What: alerts.tf gets one new attribute on job_failure_spike; release workflow removes posthog-apply from two dependency lists and their rollback condition.
Why: Provider bug: Terraform planned null, API returned true, provider rejected its own response — causing a failed apply that tripped rollback and reverted two PRs from production.
Areas: .github/workflows/release-promote.yml+19−4infra/posthog/alerts.tf+30−8
Blast: 2 files, +49/-12; CI workflow + PostHog Terraform only — no application code, no domain logic, no schema changes.

➖ci· Check runs API not accessible via PAT➖coderabbit· No .coderabbit.yaml in repo

Findings · 6

correctness1

low

posthog-apply failure is now silent outside the run UI

.github/workflows/release-promote.yml

posthog-apply still runs but is excluded from rollback. A Terraform failure surfaces only in the workflow UI — no automated recovery. Intentional per PR rationale (analytics-only blast radius), but worth documenting in on-call runbook.

conventions2

info

Decoupling approach is idiomatic — stronger than continue-on-error

.github/workflows/release-promote.yml

Removing posthog-apply from postflight/rollback needs means no downstream job waits on it. Strictly stronger than continue-on-error. Consistent with tinybird-prod pattern.

info

alerts.tf field addition is canonical — additive and forward-compatible

infra/posthog/alerts.tf

Adding check_ongoing_interval follows canonical-form pattern. Comment documents the provider bug, real incident, why ignore_changes does not help, and flags future hourly alerts.

tests1

info

No application logic changed — no test coverage gap

.github/workflows/release-promote.yml

Only CI workflow and Terraform config changed. Vitest/Playwright do not apply. terraform validate + plan in CI is the appropriate mechanism.

improvement2

low

posthog-apply needs list has pre-existing inconsistency (not introduced here)

.github/workflows/release-promote.yml

posthog-apply lists needs upstream jobs but its comment says it runs independently. With skip_* inputs, upstream skips can cascade to posthog-apply. Pre-existing drift, not introduced here.

info

alerts.tf comment length proportionate given incident severity

infra/posthog/alerts.tf

The ~18-line comment for a one-line fix documents a non-obvious provider bug, the exact error, the real incident, why ignore_changes cannot help, and flags future hourly alerts. Each sentence earns its place.