← all branches

feat/powerradar

needs attentionviewing older commit
ec4847f · fullpre-PRreviewed 2026-07-08 03:32 UTC2H · 11M · 8L · 15I
The branch
Purpose
Port the PowerRadar (Centrica/Smarter) integration — 31 legacy sites on hourly latin-1 CSVs — into the v2 metrics engine as a CSV-first intake worker, then provision the CDK stack.
Goal
Unblock SG-3 (live end-to-end intake validation): worker + CDK must be deployed and the powerradar ARN seeded before a real CSV can be uploaded and processed.
Sub-goals
  • SG-1: CSV worker (latin-1 parse, DST-safe local→UTC, site-name channel map, kW→W) + 37 unit tests
  • SG-2: IntegrationPowerradarLambdaStack (intake bucket + metrics Lambda, SSM, coordinator invoke grant, addDependency)
  • SG-3: Preview data plane + powerradarChannel D3 unblock + live CSV validation
  • SG-4: End-to-end Tinybird + legacy parity
The changes (whole branch)
What
New powerradar worker package (engine/csv.ts, translation/rows-to-batu.ts, metrics.lambda.ts + 3 test files), CDK stack (IntegrationPowerradarLambdaStack: S3 intake bucket + Lambda), foundation seeds for 12 integration manifests + catalog rows + ARN registry --only filter, site-context-aggregator csvS3Key + powerradarChannel passthrough, migration scripts (D2/D3) for legacy-registry cleanup.
Why
PowerRadar has 31 production sites emitting hourly CSVs; the legacy Python intake (prod-lambda-batu-centrica-historic-intake) is manual-cadence. Porting to v2 engine puts these sites on the automated collection schedule and makes their data available to billing/savings calculations.
Areas
services/metrics/integrations/powerradar+8450infra/cdk/src+4042packages/integration-manifests/src/manifests+7576packages/database/src+25714scripts/metrics+16020services/metrics/engine/src+573domains/core/src+151
Blast
~55 files, +4,231/−26 lines. Core engine type widened (MetricChannelSpec). Foundation seeds touch all 12 integration manifests. No schema migrations — secret_configurations provider enum is TS-only (no DB CHECK constraint). New CDK stack, no changes to existing stacks.
SG-1-complete SG-2-complete no-pr-yet full-review
typecheck· not run in this review — CDK synth clean per scope.mdtests· 37 unit tests in scope.md intent log; not re-run in this reviewcoderabbit· no .coderabbit.yaml in repoci· no open PR — pre-PR branch review

Findings · 29

correctness2

info

localToUtc called twice per row via isNonExistentLocalTime — performance only

services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:130

isNonExistentLocalTime internally calls localToUtc, then line 135 calls localToUtc again. For ~8,760 rows/year this doubles the date-fns-tz parse cost. No correctness impact.

info

formatLocal seconds comparison is safe for integer-minute CSV rows but brittle

services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:144

The fall-back guard compares formatLocal(shifted, timezone) === localStr (which ends in ':00'). This holds because PowerRadar CSVs are always on minute boundaries. A future sub-minute variant would silently drop fall-back second occurrences.

security7

medium

csvS3Key passed unvalidated to S3 GetObject — within-bucket path freeloading

services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:82

csvS3Key is trimmed but not structurally validated before the S3 GetObject call. A crafted key (malicious SFN input or operator error) can read any object in the intake bucket since the IAM resource is arn:aws:s3:::${intakeBucketName}/* (no prefix scope). S3 does not interpret '..' as directory traversal, but any key the Lambda's role can read is accessible. Fix: validate csvS3Key against an expected prefix regex (e.g. /^powerradar\/uploads\//); scope the IAM resource to that prefix.

medium

IAM GetObject not scoped to upload prefix — full bucket read

infra/cdk/src/stacks/services/metrics/integrations/powerradar/iam.ts:27

The IAM resource is `arn:aws:s3:::${intakeBucketName}/*` covering the entire bucket. If any future object in the bucket carries higher sensitivity, the Lambda can read it. Scope the grant to `${intakeBucketName}/powerradar/uploads/*` and enforce the same prefix in the upload path.

low

CORS includes http://localhost:3000 (plain HTTP) in dev/stg

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:99

Practical risk is low (bucket is blockPublicAccess, requires IAM auth), but allowing http:// origins is a hygiene issue. Use https://localhost:3000 or remove it when presigned browser uploads are wired.

low

CORS wildcard *.vercel.app covers all Vercel tenants, not just Batu deployments

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:99

Any Vercel-hosted site could issue CORS-permitted requests to the bucket. Combined with PUT and GET methods, this is a confused-deputy risk if presigned URLs are ever wired without origin binding. Consider scoping to *.batuenergy.vercel.app.

low

removalPolicy DESTROY + autoDeleteObjects in dev/stg — uploaded CSVs silently deleted

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:105

A stg stack redeploy that drops and recreates the intake bucket permanently destroys any unprocessed uploads. Consider RETAIN for stg, or document that the stg bucket is ephemeral.

info

No S3 server access logging or object lifecycle on intake bucket

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:87

No audit trail of which CSV objects were fetched; unprocessed uploads accumulate. Not a vulnerability but limits forensic capability. Recommend adding access logging to a central log bucket.

info

No size cap before loading CSV into memory

services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:113

Lambda downloads the entire S3 object into a Buffer before parsing. For the operational envelope (~9k rows × ~30 columns = a few MB), 512MB RAM is fine. A HeadObject size-check before download would be a future-proofing improvement.

conventions5

high

ArnEnvVarMissing misused for bucket-name env var POWERRADAR_INTAKE_BUCKET_NAME

services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:186

ArnEnvVarMissing is semantically scoped to missing Lambda ARN env vars (its field is `envVar`, all existing uses reference ARNs). A bucket name is not an ARN. Use translationFailed() or a new tag for configuration-missing errors. Using ArnEnvVarMissing misleads operators and any downstream error-routing that branches on _tag.

medium

Vendor-specific fields accumulating on shared MetricChannelSpec type

services/metrics/engine/src/types/invocation.types.ts:166

MetricChannelSpec now has egaugeColumn (feat/egauge) and powerradarChannel (this branch) as optional vendor-specific fields. A third vendor-specific field would be a high violation. The canonical alternative is a generic `sourceConfig?: Record<string,unknown>` passthrough field. Not blocking given egaugeColumn precedent, but should not continue to accumulate.

info

CDK mandatory tags: 3 missing at stack level are supplied by applyBatuTags at app level

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:68

batu:env, batu:owner, batu:costCenter are not set at stack level. They are provided by applyBatuTags() on the CDK App in main.ts, matching the Growatt stack pattern. All 8 mandatory tags are present on deployed resources via CDK tag inheritance. No violation.

info

addDependency for powerradar SSM read correctly placed

infra/cdk/src/app/main.ts:419

coordinatorLambdaStack.addDependency(powerradarLambdaStack) enforces coordinator deploys after powerradar writes metrics-lambda-arn SSM. Correct per infrastructure.md.

info

Manifest variable keys confirmed seeded in seed-metrics-catalog.ts

packages/database/src/seed-metrics-catalog.ts:279

consumption, demand, grid_import, grid_export are all seeded as MetricType rows. No gap.

tests9

medium

resolveChannels scalar-params fallback path untested

services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:35

The backward-compat scalar path in resolveChannels (when params.sources is absent) is never exercised. A regression here would silently break direct-invoke compatibility (zero channels → fail.translate).

medium

Empty sources array early-exit guard untested

services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:112

The handler returns TranslationFailed when resolveChannels returns []. No test covers both missing/empty sources AND missing scalars. The 'all channels skipped' test reaches the later rowsToBatu check, not this early guard.

medium

Lambda entrypoint POWERRADAR_INTAKE_BUCKET_NAME guard untested

services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:180

Only runPowerradarMetrics (injected-storage) is tested. The exported handler entrypoint — which guards on the env var — is never exercised by tests.

medium

Whitespace-only cell handling undocumented and untested

services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:96

Production code trims cells and checks cell === ''. A whitespace-only cell (' ') is silently skipped, but not counted in invalidValueCells. This is probably desired but is untested — a CSV with space-padded empty cells would produce fewer points than expected with no diagnostic counter.

low

kWh suffix normalization not tested

services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:39

normalizeChannelName strips both (kW) and (kWh) (regex: /\(k\s*w\s*h?\)/i). Tests only cover the (kW) and ' - (kW)' forms.

low

Timestamp column variant tests don't assert row extraction

services/metrics/integrations/powerradar/src/__tests__/csv.test.ts:75

Tests confirm 'Timestamp (America/Mexico_City)' headers are recognised (parsed.ok), but don't assert rows are correctly extracted under those header variants.

info

DST coverage is comprehensive across all four required cases

services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:136

Spring-forward gap drop (Tijuana), fall-back single row (first DST instant), fall-back doubled row (two distinct UTC instants), non-DST timezone (Mexico City) — all four cases covered.

info

Channel mapping coverage is complete

services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:60

Exact match, normalized match, HM→H&M, CA→C&A, Chillers* → canonical, channel-not-found skip, channel-not-configured skip — all covered.

info

CSV parse coverage is complete for all stated quirks

services/metrics/integrations/powerradar/src/__tests__/csv.test.ts:33

UTF-8 BOM as latin-1, '?' variant, YYYY-MM-DD HH:MM, seconds stripped, empty CSV, no-timestamp-column, no-parseable-rows, no-data-columns — all covered.

improvement6

high

Outer catch maps parse/logic bugs to UpstreamUnavailable — wrong retry semantics

services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:167

The outer try/catch maps any unhandled exception to `{ _tag: 'UpstreamUnavailable', statusCode: 502 }`. The inner S3 catch (lines 114-126) already handles the only genuine upstream-unavailable case. Anything that escapes the inner logic (a bug in rowsToBatu, a corrupt params.window value, an unexpected type error) is incorrectly tagged as 502 — coordinators use _tag for retry strategy, so a logic bug gets retried as if the S3 service were down. Fix: the outer catch should use translationFailed() (500 TranslationFailed) instead of UpstreamUnavailable.

medium

--only validation is case-sensitive; wrong case silently seeds nothing

packages/database/src/seed-integration-arns.ts:118

--only=Powerradar (capital P) passes the unknown-code check (it IS unknown, so an error is thrown)... actually wait — let me re-read: `only.filter(c => !REGISTRY.some(({code}) => code === c))` — if 'Powerradar' is passed, REGISTRY has no entry with code 'Powerradar' so it IS in `unknown` and the error IS thrown. But if the operator passes the correct code with different casing like 'powerRadar', same thing — error thrown. So the validation IS correct. The edge case is only if you pass an exact code match. Re-flagged as INFO actually — the validation is correct and safe.

medium

Chillers regex over-matches any channel starting with 'chillers'

services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:62

Line 62: /^chillers/i.test(s) collapses any channel whose normalized name starts with 'chillers' to 'Chillers - Total'. A future CSV with 'Chillers North' or 'Chillers Bloque 2' columns would silently resolve to the same canonical name, making both channels indistinguishable. Add a comment documenting the single-Chillers-per-site invariant, or tighten the regex.

medium

Intake bucket has no object lifecycle/expiration policy — confidential data accumulates

infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:87

The bucket is tagged dataClass:confidential but has no S3 lifecycle rule. Uploaded CSVs accumulate indefinitely. Add lifecycleRules: [{ expiration: Duration.days(90), enabled: true }] consistent with the data-minimisation principle and other intake buckets in the platform.

low

Window clamping near DST boundary — caller concern, but worth a comment

services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:160

fromUtc/toUtc must be genuine UTC instants; callers who convert local time naively near a DST gap may silently clip boundary rows. A code comment noting this invariant would prevent future misuse.

low

sources array cast without type validation in resolveChannels

services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:49

Line 49: `sources as MetricChannelSpec[]` is an unchecked cast. A malformed sources array surfaces as a confusing undefined access inside rowsToBatu rather than a clear TranslationFailed at the entry point.

History · 15 commits

  1. f34e4ceneeds attentionincremental0H · 1M · 3L2026-07-23 00:32
  2. 1bc8aedneeds attentionincremental0H · 2M · 3L2026-07-22 21:34
  3. 31e63bbneeds attentionincremental0H · 2M · 5L2026-07-22 20:53
  4. 482cb88safeincremental0H · 0M · 1L2026-07-22 17:56
  5. 9a73b79safeincremental0H · 0M · 0L2026-07-22 17:34
  6. 87f8298needs attentionincremental0H · 1M · 3L2026-07-22 16:59
  7. 1e80096safeincremental0H · 0M · 1L2026-07-22 16:42
  8. 6ed65b4safeincremental0H · 1M · 3L2026-07-22 00:00
  9. 74a1131needs attentionfull0H · 7M · 12L2026-07-21 19:06
  10. 563252bneeds attentionincremental4H · 9M · 8L2026-07-21 18:32
  11. beced58needs attentionincremental0H · 3M · 3L2026-07-21 01:13
  12. 7fa8684needs attentionincremental4H · 9M · 7L2026-07-20 22:56
  13. 230784fneeds attentionfull2H · 11M · 14L2026-07-10 00:15
  14. b876b54needs attentionincremental2H · 2M · 5L2026-07-08 04:44
  15. ec4847fneeds attentionfull2H · 11M · 8L2026-07-08 03:32current