feat/powerradar
needs attentionviewing older commitec4847f · fullpre-PRreviewed 2026-07-08 03:32 UTC2H · 11M · 8L · 15I- Purpose
- Port the PowerRadar (Centrica/Smarter) integration — 31 legacy sites on hourly latin-1 CSVs — into the v2 metrics engine as a CSV-first intake worker, then provision the CDK stack.
- Goal
- Unblock SG-3 (live end-to-end intake validation): worker + CDK must be deployed and the powerradar ARN seeded before a real CSV can be uploaded and processed.
- Sub-goals
- SG-1: CSV worker (latin-1 parse, DST-safe local→UTC, site-name channel map, kW→W) + 37 unit tests
- SG-2: IntegrationPowerradarLambdaStack (intake bucket + metrics Lambda, SSM, coordinator invoke grant, addDependency)
- SG-3: Preview data plane + powerradarChannel D3 unblock + live CSV validation
- SG-4: End-to-end Tinybird + legacy parity
- What
- New powerradar worker package (engine/csv.ts, translation/rows-to-batu.ts, metrics.lambda.ts + 3 test files), CDK stack (IntegrationPowerradarLambdaStack: S3 intake bucket + Lambda), foundation seeds for 12 integration manifests + catalog rows + ARN registry --only filter, site-context-aggregator csvS3Key + powerradarChannel passthrough, migration scripts (D2/D3) for legacy-registry cleanup.
- Why
- PowerRadar has 31 production sites emitting hourly CSVs; the legacy Python intake (prod-lambda-batu-centrica-historic-intake) is manual-cadence. Porting to v2 engine puts these sites on the automated collection schedule and makes their data available to billing/savings calculations.
- Areas
- services/metrics/integrations/powerradar+845−0infra/cdk/src+404−2packages/integration-manifests/src/manifests+757−6packages/database/src+257−14scripts/metrics+1602−0services/metrics/engine/src+57−3domains/core/src+15−1
- Blast
- ~55 files, +4,231/−26 lines. Core engine type widened (MetricChannelSpec). Foundation seeds touch all 12 integration manifests. No schema migrations — secret_configurations provider enum is TS-only (no DB CHECK constraint). New CDK stack, no changes to existing stacks.
Findings · 29
correctness2
localToUtc called twice per row via isNonExistentLocalTime — performance only
services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:130
isNonExistentLocalTime internally calls localToUtc, then line 135 calls localToUtc again. For ~8,760 rows/year this doubles the date-fns-tz parse cost. No correctness impact.
formatLocal seconds comparison is safe for integer-minute CSV rows but brittle
services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:144
The fall-back guard compares formatLocal(shifted, timezone) === localStr (which ends in ':00'). This holds because PowerRadar CSVs are always on minute boundaries. A future sub-minute variant would silently drop fall-back second occurrences.
security7
csvS3Key passed unvalidated to S3 GetObject — within-bucket path freeloading
services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:82
csvS3Key is trimmed but not structurally validated before the S3 GetObject call. A crafted key (malicious SFN input or operator error) can read any object in the intake bucket since the IAM resource is arn:aws:s3:::${intakeBucketName}/* (no prefix scope). S3 does not interpret '..' as directory traversal, but any key the Lambda's role can read is accessible. Fix: validate csvS3Key against an expected prefix regex (e.g. /^powerradar\/uploads\//); scope the IAM resource to that prefix.
IAM GetObject not scoped to upload prefix — full bucket read
infra/cdk/src/stacks/services/metrics/integrations/powerradar/iam.ts:27
The IAM resource is `arn:aws:s3:::${intakeBucketName}/*` covering the entire bucket. If any future object in the bucket carries higher sensitivity, the Lambda can read it. Scope the grant to `${intakeBucketName}/powerradar/uploads/*` and enforce the same prefix in the upload path.
CORS includes http://localhost:3000 (plain HTTP) in dev/stg
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:99
Practical risk is low (bucket is blockPublicAccess, requires IAM auth), but allowing http:// origins is a hygiene issue. Use https://localhost:3000 or remove it when presigned browser uploads are wired.
CORS wildcard *.vercel.app covers all Vercel tenants, not just Batu deployments
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:99
Any Vercel-hosted site could issue CORS-permitted requests to the bucket. Combined with PUT and GET methods, this is a confused-deputy risk if presigned URLs are ever wired without origin binding. Consider scoping to *.batuenergy.vercel.app.
removalPolicy DESTROY + autoDeleteObjects in dev/stg — uploaded CSVs silently deleted
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:105
A stg stack redeploy that drops and recreates the intake bucket permanently destroys any unprocessed uploads. Consider RETAIN for stg, or document that the stg bucket is ephemeral.
No S3 server access logging or object lifecycle on intake bucket
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:87
No audit trail of which CSV objects were fetched; unprocessed uploads accumulate. Not a vulnerability but limits forensic capability. Recommend adding access logging to a central log bucket.
No size cap before loading CSV into memory
services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:113
Lambda downloads the entire S3 object into a Buffer before parsing. For the operational envelope (~9k rows × ~30 columns = a few MB), 512MB RAM is fine. A HeadObject size-check before download would be a future-proofing improvement.
conventions5
ArnEnvVarMissing misused for bucket-name env var POWERRADAR_INTAKE_BUCKET_NAME
services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:186
ArnEnvVarMissing is semantically scoped to missing Lambda ARN env vars (its field is `envVar`, all existing uses reference ARNs). A bucket name is not an ARN. Use translationFailed() or a new tag for configuration-missing errors. Using ArnEnvVarMissing misleads operators and any downstream error-routing that branches on _tag.
Vendor-specific fields accumulating on shared MetricChannelSpec type
services/metrics/engine/src/types/invocation.types.ts:166
MetricChannelSpec now has egaugeColumn (feat/egauge) and powerradarChannel (this branch) as optional vendor-specific fields. A third vendor-specific field would be a high violation. The canonical alternative is a generic `sourceConfig?: Record<string,unknown>` passthrough field. Not blocking given egaugeColumn precedent, but should not continue to accumulate.
CDK mandatory tags: 3 missing at stack level are supplied by applyBatuTags at app level
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:68
batu:env, batu:owner, batu:costCenter are not set at stack level. They are provided by applyBatuTags() on the CDK App in main.ts, matching the Growatt stack pattern. All 8 mandatory tags are present on deployed resources via CDK tag inheritance. No violation.
addDependency for powerradar SSM read correctly placed
infra/cdk/src/app/main.ts:419
coordinatorLambdaStack.addDependency(powerradarLambdaStack) enforces coordinator deploys after powerradar writes metrics-lambda-arn SSM. Correct per infrastructure.md.
Manifest variable keys confirmed seeded in seed-metrics-catalog.ts
packages/database/src/seed-metrics-catalog.ts:279
consumption, demand, grid_import, grid_export are all seeded as MetricType rows. No gap.
tests9
resolveChannels scalar-params fallback path untested
services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:35
The backward-compat scalar path in resolveChannels (when params.sources is absent) is never exercised. A regression here would silently break direct-invoke compatibility (zero channels → fail.translate).
Empty sources array early-exit guard untested
services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:112
The handler returns TranslationFailed when resolveChannels returns []. No test covers both missing/empty sources AND missing scalars. The 'all channels skipped' test reaches the later rowsToBatu check, not this early guard.
Lambda entrypoint POWERRADAR_INTAKE_BUCKET_NAME guard untested
services/metrics/integrations/powerradar/src/__tests__/metrics-handler.test.ts:180
Only runPowerradarMetrics (injected-storage) is tested. The exported handler entrypoint — which guards on the env var — is never exercised by tests.
Whitespace-only cell handling undocumented and untested
services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:96
Production code trims cells and checks cell === ''. A whitespace-only cell (' ') is silently skipped, but not counted in invalidValueCells. This is probably desired but is untested — a CSV with space-padded empty cells would produce fewer points than expected with no diagnostic counter.
kWh suffix normalization not tested
services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:39
normalizeChannelName strips both (kW) and (kWh) (regex: /\(k\s*w\s*h?\)/i). Tests only cover the (kW) and ' - (kW)' forms.
Timestamp column variant tests don't assert row extraction
services/metrics/integrations/powerradar/src/__tests__/csv.test.ts:75
Tests confirm 'Timestamp (America/Mexico_City)' headers are recognised (parsed.ok), but don't assert rows are correctly extracted under those header variants.
DST coverage is comprehensive across all four required cases
services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:136
Spring-forward gap drop (Tijuana), fall-back single row (first DST instant), fall-back doubled row (two distinct UTC instants), non-DST timezone (Mexico City) — all four cases covered.
Channel mapping coverage is complete
services/metrics/integrations/powerradar/src/__tests__/rows-to-batu.test.ts:60
Exact match, normalized match, HM→H&M, CA→C&A, Chillers* → canonical, channel-not-found skip, channel-not-configured skip — all covered.
CSV parse coverage is complete for all stated quirks
services/metrics/integrations/powerradar/src/__tests__/csv.test.ts:33
UTF-8 BOM as latin-1, '?' variant, YYYY-MM-DD HH:MM, seconds stripped, empty CSV, no-timestamp-column, no-parseable-rows, no-data-columns — all covered.
improvement6
Outer catch maps parse/logic bugs to UpstreamUnavailable — wrong retry semantics
services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:167
The outer try/catch maps any unhandled exception to `{ _tag: 'UpstreamUnavailable', statusCode: 502 }`. The inner S3 catch (lines 114-126) already handles the only genuine upstream-unavailable case. Anything that escapes the inner logic (a bug in rowsToBatu, a corrupt params.window value, an unexpected type error) is incorrectly tagged as 502 — coordinators use _tag for retry strategy, so a logic bug gets retried as if the S3 service were down. Fix: the outer catch should use translationFailed() (500 TranslationFailed) instead of UpstreamUnavailable.
--only validation is case-sensitive; wrong case silently seeds nothing
packages/database/src/seed-integration-arns.ts:118
--only=Powerradar (capital P) passes the unknown-code check (it IS unknown, so an error is thrown)... actually wait — let me re-read: `only.filter(c => !REGISTRY.some(({code}) => code === c))` — if 'Powerradar' is passed, REGISTRY has no entry with code 'Powerradar' so it IS in `unknown` and the error IS thrown. But if the operator passes the correct code with different casing like 'powerRadar', same thing — error thrown. So the validation IS correct. The edge case is only if you pass an exact code match. Re-flagged as INFO actually — the validation is correct and safe.
Chillers regex over-matches any channel starting with 'chillers'
services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:62
Line 62: /^chillers/i.test(s) collapses any channel whose normalized name starts with 'chillers' to 'Chillers - Total'. A future CSV with 'Chillers North' or 'Chillers Bloque 2' columns would silently resolve to the same canonical name, making both channels indistinguishable. Add a comment documenting the single-Chillers-per-site invariant, or tighten the regex.
Intake bucket has no object lifecycle/expiration policy — confidential data accumulates
infra/cdk/src/stacks/services/metrics/integrations/powerradar/lambda.stack.ts:87
The bucket is tagged dataClass:confidential but has no S3 lifecycle rule. Uploaded CSVs accumulate indefinitely. Add lifecycleRules: [{ expiration: Duration.days(90), enabled: true }] consistent with the data-minimisation principle and other intake buckets in the platform.
Window clamping near DST boundary — caller concern, but worth a comment
services/metrics/integrations/powerradar/src/translation/rows-to-batu.ts:160
fromUtc/toUtc must be genuine UTC instants; callers who convert local time naively near a DST gap may silently clip boundary rows. A code comment noting this invariant would prevent future misuse.
sources array cast without type validation in resolveChannels
services/metrics/integrations/powerradar/src/handlers/metrics.lambda.ts:49
Line 49: `sources as MetricChannelSpec[]` is an unchecked cast. A malformed sources array surfaces as a confusing undefined access inside rowsToBatu rather than a clear TranslationFailed at the entry point.
History · 15 commits
- f34e4ceneeds attentionincremental0H · 1M · 3L2026-07-23 00:32
- 1bc8aedneeds attentionincremental0H · 2M · 3L2026-07-22 21:34
- 31e63bbneeds attentionincremental0H · 2M · 5L2026-07-22 20:53
- 482cb88safeincremental0H · 0M · 1L2026-07-22 17:56
- 9a73b79safeincremental0H · 0M · 0L2026-07-22 17:34
- 87f8298needs attentionincremental0H · 1M · 3L2026-07-22 16:59
- 1e80096safeincremental0H · 0M · 1L2026-07-22 16:42
- 6ed65b4safeincremental0H · 1M · 3L2026-07-22 00:00
- 74a1131needs attentionfull0H · 7M · 12L2026-07-21 19:06
- 563252bneeds attentionincremental4H · 9M · 8L2026-07-21 18:32
- beced58needs attentionincremental0H · 3M · 3L2026-07-21 01:13
- 7fa8684needs attentionincremental4H · 9M · 7L2026-07-20 22:56
- 230784fneeds attentionfull2H · 11M · 14L2026-07-10 00:15
- b876b54needs attentionincremental2H · 2M · 5L2026-07-08 04:44
- ec4847fneeds attentionfull2H · 11M · 8L2026-07-08 03:32current