← all branches

feat/tb-deploy

needs attentionviewing older commit
f43d034 · incrementalPR #261reviewed 2026-07-07 18:35 UTC2H · 3M · 3L · 4I
The branch
Purpose
Define and wire Batu's Tinybird deployment strategy: three long-lived workspaces (batu_dev/batu_stg/batu_prod) for hard environment isolation, ephemeral Forward branches per PR inside dev, and a per-environment token split replacing the single shared TINYBIRD_TOKEN.
Goal
Automate what was done manually during this branch: per-env token split, schema-CD via tb deploy in staging/prod pipelines, and a durable rule doc capturing the full model.
Sub-goals
  • SG-1: Durable strategy rule .claude/rules/tinybird-deployment.md — workspace topology, token model, three-lane seeding pattern
  • SG-2: Per-env token split in preview-provision.yml — TINYBIRD_TOKEN_DEV for runtime, TINYBIRD_DEPLOY_TOKEN_DEV for CI branch ops
  • SG-3: staging-update.yml + release-promote.yml seed TINYBIRD_TOKEN_STG/PROD and run tb deploy --cloud
  • SG-4: SHA-pin dorny/paths-filter in all occurrences across workflows
  • SG-5: Fix portable minute format (%i not %M) in resolve_stream_window.pipe and document quality field sync in metric.datasource
The changes (whole branch)
What
This incremental commit adds: SHA-pin for dorny/paths-filter in all 3 workflow occurrences; expands the token-precedence documentation in tinybird-deployment.md; adds a quality field sync comment to metric.datasource; fixes %M→%i minute format in resolve_stream_window.pipe; deletes the now-invalid stub test.
Why
The %M format was a portability bug (newer ClickHouse reads %M as month name); the SHA pin is supply-chain hardening; the doc expansion clarifies the TINYBIRD_BRANCH_TOKEN ?? TINYBIRD_TOKEN precedence contract that was previously underspecified.
Areas
.claude/rules+2782.github/workflows+37650infra/tinybird+8358
Blast
12 files, +737/-110 across rules, workflows, and tinybird infra. Workflows touch the critical preview/staging/prod deploy lanes. Tinybird schema changes affect the production metric read path.
Workspaces batu_stg and batu_prod were bootstrapped out-of-band during this branch — PR merges the automation, not the initial setup tb deploy --cloud jobs skip cleanly until TINYBIRD_DEPLOY_TOKEN_STG/PROD secrets are set Pre-existing TINYBIRD_TOKEN secret is being phased out but still present as fallback
CI checks· gh pr checks returned no output — CI status unavailable at review timeCodeRabbit· No .coderabbit.yaml in repo

Findings · 13

correctness2

low

README still documents the old (now-wrong) %M minute format

infra/tinybird/README.md:52

The README's bucket_ts explanation still shows '%Y-%m-%dT%H:%M:%SZ' — the format that was just fixed to '%Y-%m-%dT%H:%i:%SZ'. A future reader following the README example for a new pipe will reintroduce the portability bug the pipe comment explicitly warns against. Update the README format string and add the one-line %M/%i note that lives in the pipe.

info

Stub test deletion is justified — test was for placeholder pipe, not real SQL

infra/tinybird/tests/resolve_stream_window.test.yaml

The deleted test expected a status='stub' column only valid for the placeholder pipe. The real pipe has completely different params and output shape. Keeping the stub test would cause tb test failures on every run. Deletion is correct; the gap is the absence of a replacement fixture for the real pipe.

security3

medium

Workspace Tinybird token written to GITHUB_OUTPUT without explicit masking

.github/workflows/preview-provision.yml:282

The tb_workspace_token output is written via shell variable expansion without a preceding '::add-mask::' call. Contrast with the branch token path (line 340) which explicitly masks before writing output. GitHub auto-masks values that flow through ${{ secrets.X }} expressions, but shell-expanded values may appear unmasked in workflow summaries or step references. Pre-existing pattern, but the diff's new documentation increases awareness of this token's role. Recommend adding echo '::add-mask::${TINYBIRD_TOKEN}' before line 282.

low

Other action tags in the same workflow files remain mutable (not introduced by this diff)

.github/workflows/staging-update.yml

The diff correctly SHA-pins dorny/paths-filter in preview-provision.yml and staging-update.yml. However, actions/checkout@v4, pnpm/action-setup@v3, aws-actions/configure-aws-credentials@v4, and others remain on mutable tags in the same files. The hardening is directionally correct but partial. A follow-up to pin the remaining third-party actions would complete it.

info

%i format fix is a correctness improvement that also reduces data-integrity risk

infra/tinybird/pipes/resolve_stream_window.pipe:107

On newer ClickHouse, %M renders the month name rather than the minute number, silently producing malformed ISO-8601 timestamps. parseTinybirdUtc provides a belt-and-suspenders filter, but the fix at the source is the right approach. No security concern; noted for completeness.

conventions2

info

dorny/paths-filter SHA pinning is complete — pr-checks and release-promote don't use it

.github/workflows/preview-provision.yml

All three occurrences of dorny/paths-filter across the repo (2 in preview-provision.yml, 1 in staging-update.yml) are now SHA-pinned. pr-checks.yml and release-promote.yml do not use this action, so no additional pinning is needed for it.

info

quality field comments are in sync between datasource and pipe

infra/tinybird/datasources/metric.datasource:26

metric.datasource DESCRIPTION now matches resolve_stream_window.pipe raw_dedup DESCRIPTION: both document that quality is stored on ingest but not yet surfaced in the read path. The cross-reference and 'keep both ends in sync' instruction are clear and actionable.

tests5

high

No Tinybird pipe-level tests for the real resolve_stream_window pipe

infra/tinybird/tests/

After deleting the stub test, infra/tinybird/tests/ is empty. The real pipe covers three non-trivial nodes: raw_dedup (argMax dedup), priority_splice (argMin+indexOf source selection), and downsample (multiIf aggregator dispatch). None are covered by tb test fixtures. Schema changes or SQL regressions will only surface in production. At minimum, a happy-path fixture asserting the bucket_ts ISO-8601 shape and a two-source priority splice scenario should exist.

high

The %i format fix has no regression test at the pipe level

infra/tinybird/pipes/resolve_stream_window.pipe:107

The %M→%i change is a real portability bug fix (on newer ClickHouse %M renders the month name 'May' instead of the minute). The fix is correct, but there is no fixture asserting the bucket_ts output format (e.g. '2026-05-01T14:34:00Z'). If %i were ever reverted, no automated check would catch it before parseTinybirdUtc consumers receive malformed timestamps.

medium

No deduplication test for argMax(value, ingested_at) in raw_dedup node

infra/tinybird/pipes/resolve_stream_window.pipe:36

The argMax deduplication is the mechanism that makes billing-grade late-arriving corrections self-heal. There is no fixture inserting two rows with the same (metric_source_id, ts) but different ingested_at values and asserting the later one wins. ReplacingMergeTree may or may not have collapsed duplicates at query time, making the explicit argMax layer especially important to test.

medium

No test for priority_splice multi-source winner selection

infra/tinybird/pipes/resolve_stream_window.pipe:60

priority_splice picks the highest-precedence source at each timestamp via argMin(value, indexOf(ids[], metric_source_id)). This is the core business logic of priority splicing and has zero test coverage. A fixture with two sources sharing a timestamp should assert the first-array-position source wins.

low

No test for multiIf aggregator dispatch, especially the 'last' branch

infra/tinybird/pipes/resolve_stream_window.pipe:109

The downsample node dispatches five aggregators (sum/min/max/last/mean) via multiIf. The 'last' branch uses argMax(value, ts) which is semantically different from all others. No fixture validates that aggregator='last' returns the chronologically last value in a bucket rather than mean or sum.

improvement1

info

quality sync note could add a cross-reference pointer from pipe back to datasource

infra/tinybird/pipes/resolve_stream_window.pipe:33

The pipe's raw_dedup says 'not exposed yet' without pointing to the datasource DESCRIPTION's more detailed sync note. Adding a brief pointer (e.g. 'see metric.datasource DESCRIPTION') would help future readers find the intent when adding the quality read path.

History · 10 commits

  1. 5d30429safeincremental0H · 0M · 1L2026-07-08 23:04
  2. 19bac85needs attentionincremental0H · 5M · 6L2026-07-08 22:46
  3. 92e890aneeds attentionincremental0H · 1M · 3L2026-07-08 21:46
  4. e81a781needs attentionincremental0H · 4M · 4L2026-07-08 21:31
  5. 2eca91cneeds attentionfull5H · 13M · 12L2026-07-08 19:18
  6. cf63bb1needs attentionincremental4H · 4M · 7L2026-07-07 19:47
  7. f43d034needs attentionincremental2H · 3M · 3L2026-07-07 18:35current
  8. b7800c0needs attentionincremental1H · 2M · 7L2026-07-07 03:31
  9. 5aef105needs attentionfull6H · 9M · 2L2026-07-06 17:52
  10. 95b4484blockedfull3H · 8M · 4L2026-07-06 05:36