feat/energy-api
needs attentionviewing older commitc24738e · incrementalPR #207reviewed 2026-07-04 05:12 UTC3H · 10M · 9L · 4I- Purpose
- Port the legacy electricity-data-api (billing + savings + metrics + webhooks + PDFs, Pulumi/DDB/Athena stack in batu-monorepo) into the platform. Domain logic and contracts port; AWS plumbing dies.
- Goal
- Deliver the functional core of the end-to-end port (Phases 0–8 + pure cores of 4/5/6/7): metrics adapter → tariff/TOU reference data → CFE bill calculators → calculated-bill persistence → PDF. Inc-2 infra (cron/SFN estate) and the legacy A/B are follow-ups.
- Sub-goals
- SG-1: Phase 0 — Port spec + decisions (D1–D5)
- SG-2: Phase 1 — Tinybird→SiteMetrics adapter, DQ, KK validated to 1e-6
- SG-3: Phase 2 — tariff_rate_values storage + CFE TOU classifier (97 live schedules, conservation 1e-9)
- SG-4: Phase 3 — CFE bill calculators + calculated bill source
- SG-5: Phase 4 — HelioScope CSV-upload intake + API-fetch + materialize-persist
- SG-6: Phase 5 — HelioScope daily rolling top-up job
- SG-7: Phase 6 — QuickSight-parity synthesizers (powerFactorAdjustment, billingFrequency)
- SG-8: Phase 7/8 — CDK stacks (DevOpsStack, IntegrationHelioscopeLambdaStack) + preview-provision wiring
- What
- This incremental window (94b95a05→c24738e9) adds: (1) HelioScope intake contract + handler routing in the ts-rest router; (2) QuickSight-parity bill synthesizers (powerFactorAdjustment + billingFrequency) in bill.decisions + queries + shells; (3) egaugeColumn field threading through asset-management-validation and the provision shell; (4) Expect<AssertEqual> drift-guard hardening across all 25+ type-check files in core + utility domains; (5) Phase 7 CI hardening (catch-all no-op detection) + HelioScope SSM params in preview-provision; (6) DevOpsStack + IntegrationHelioscopeLambdaStack CDK registration; (7) CollectOptionsFields shared UI component replacing duplicated advanced-options code in AddContractDrawer and InlineContractForm.
- Why
- Completes the HelioScope integration wire-up (Phases 3–5 infra glue), adds QuickSight analytics parity for CFE billing columns, and hardens type-safety tooling that was previously a silent no-op.
- Areas
- domains/utility/src+77−0apps/platform/src/api+31−0apps/platform/src/app+30−0domains/cross-domain/src+25−0domains/core/src+23−0packages/api/src + packages/database+24−0domains/metrics/src+12−0.github/workflows + infra/cdk+8−0
- Blast
- 66 files in incremental window, +49760/−4697 across the full branch (482 files). Core areas: utility-domain bill calculators (billing-critical), metrics-domain queries (daily-job discovery), cross-domain provision shell (onboarding path), CDK infra (preview deploy).
Findings · 26
correctness5
findHelioscopeSourcesDueForTopUp: duplicate rows when org has multiple active secrets
domains/metrics/src/metric-source/metric-source.queries.ts
LEFT JOIN to integrationSecrets uses (orgId, integrationId, status='active') with no DISTINCT or LIMIT 1. An org with multiple active IntegrationSecret rows for the same integration (multiple secretConfigIds) produces one row per secret, so the same MetricSource appears multiple times. The daily top-up job dispatches each candidate independently — this causes double materialize-persist invocations for the same source, racing to overwrite coverage state. Fix: add DISTINCT ON metricSources.publicId or a LIMIT 1 subquery on the credential join.
synthesizePowerFactorAdjustmentFromCommand: zero-valued charge emits numericValue 0 instead of blank
domains/utility/src/bill/bill.decisions.ts
The null-guard `if (charge == null && bonus == null) return null` fires only when both are strictly null. A CFE lineItem with numericValue=0 (zero charge present in XML) makes charge=0, so the condition is false and adjustment=0+0=0 emits a concrete line item contradicting the 'blank, not misleading $0' contract.
synthesizeBillingFrequencyFromCommand: new Date(dateString) UTC parsing reliance
domains/utility/src/bill/bill.decisions.ts
new Date('YYYY-MM-DD').getTime() is parsed as UTC midnight per ECMA-262. This is correct in V8/Node.js but was implementation-defined in ES5. A non-ISO date string (e.g. from a CFE data anomaly) returns NaN (guarded) — but making UTC intent explicit with 'YYYY-MM-DDT00:00:00.000Z' removes the environment dependency. Low blast radius, but worth a one-liner fix.
mergeSourceConfigS3JsonRefByPublicId: unlocked updatedAt bumps without version — breaks optimistic-lock invariant
domains/metrics/src/metric-source/metric-source.queries.ts
The self-heal update bumps updatedAt without bumping version. If a concurrent optimistic-locked update (operator editing sourceConfig via API) commits between this write's SELECT and UPDATE, the operator's next save sees a stale version but still succeeds because only updatedAt advanced. Minor inconsistency, not data loss.
Migration ordering 0053→0054→0055 is correct — no DROP-before-backfill risk
packages/database/drizzle
Journal confirms: 0053 adds sites.operation_start_date, 0054 backfills from site_locations, 0055 drops site_locations.operation_start_date. Sequential order is sound.
security7
Lambda ARNs echoed to GITHUB_OUTPUT without ::add-mask:: masking
.github/workflows/preview-provision.yml
HS_INTAKE_ARN, HS_API_FETCH_ARN, HS_MATERIALIZE_ARN are written to GITHUB_OUTPUT via bare echo statements. GHA does not auto-mask step outputs; ARNs appear in plain text in workflow logs, exposing AWS account ID, region, and function names to everyone with read access to the repo. Add '::add-mask::$VAR' immediately after each SSM read. Pre-existing pattern for CFE ARNs — this extends its scope.
No file-size cap on HelioScope presigned PUT URL
apps/platform/src/api/handlers/helioscope-intake.handler.ts
The presigned PUT has no ContentLengthRange condition. An authenticated member can upload arbitrarily large files. The intake Lambda reads the entire CSV into memory before parsing — a very large upload triggers OOM/timeout. Authenticated-member-only but worth adding a max-size check in the Lambda before full parse.
simulationId: no length or format validation
apps/platform/src/api/handlers/helioscope-intake.handler.ts
parseIntakeBody accepts any non-null string as simulationId with no maxLength or numeric format check. The value is stored in source_config.simulation_id (JSONB) and forwarded to the intake Lambda. A very long string inflates the Lambda payload. Low risk (authenticated member, JSONB tolerates large values), but a 256-char guard closes the attack surface.
HELIOSCOPE_INTAKE_BUCKET_NAME set as plain type in Vercel env
.github/workflows/preview-provision.yml
The bucket name is type:plain (visible in Vercel UI/logs) while Lambda ARNs are type:encrypted. Bucket names in presigned URLs already reach the browser so this is not a new exposure, but it's inconsistent with defense-in-depth. Matches CFE_BILLS_BUCKET_NAME precedent — worth noting for future hardening.
tariff_jobs SELECT USING(true) exposes step_function_execution_arn to all authenticated users
packages/database/drizzle/0053_third_hammerhead.sql
All authenticated Supabase sessions can SELECT every tariff_jobs row, including step_function_execution_arn (reveals AWS account ID + region). Likely intentional since tariff rates are global reference data, but the execution ARN feels internal. Consider excluding it from the API response mapper even if the RLS stays open.
savings_configs/savings_reports DELETE granted to all authenticated members
packages/database/drizzle/0053_third_hammerhead.sql
Both tables allow DELETE by any authenticated org member via RLS, even though no application handler currently exposes DELETE. If deletion should be admin/service-role only, restrict the policy now before a handler accidentally exposes it.
site_exception_states RLS join-based pattern is correct but creates indirect trust dependency
packages/database/drizzle/0053_third_hammerhead.sql
RLS uses `site_id IN (SELECT id FROM sites WHERE org_id IN (SELECT get_user_org_ids()))`. Functionally correct and consistent with the established pattern for other site-linked tables. Worth flagging in future audits of get_user_org_ids().
conventions3
Domain constants + 6-table JOIN coordinator logic embedded in metric-source.queries.ts
domains/metrics/src/metric-source/metric-source.queries.ts
Two violations: (1) HELIOSCOPE_INTEGRATION_NAMES constant array and re-exports of HELIOSCOPE_API_INTEGRATION_ID/HELIOSCOPE_CSV_INTEGRATION_ID from @batu/integration-manifests are authored in queries.ts — a thin-DB-wrapper file that should not relay constants. (2) findHelioscopeSourcesDueForTopUp performs a 6-table JOIN spanning core and metrics domains (metric_sources→integrations→metric_streams→sites→integration_secrets→secret_configurations) and embeds a business-rule cadence predicate. Both belong in a coordinator shell or a dedicated read-model query, not queries.ts.
Executable statement interleaved between import declarations
domains/cross-domain/src/site-with-devices-provisioning.shells.ts
const { resolveStartDates } = MetricSourceFCIS; appears at line ~59, between two import blocks. While syntactically valid, it violates the ESM convention that all imports precede executable statements and confuses linters and readers. Move it to the module-level destructure block after all imports.
@batu/metrics-domain in devDependencies makes the seam drift guard invisible in stripped CI passes
packages/api/package.json
SiteMetricsFCIS.SiteMetrics is used type-only in energy-metrics.schemas.ts for the Expect<AssertEqual<...>> drift guard. Correct as devDep — but if a CI step skips devDependencies during typecheck, the guard compiles away silently with no error. Consider keeping it as a regular dependency to make the constraint durable, or document the CI gate that ensures devDeps are always available.
tests8
mergeSourceConfigS3JsonRefByPublicId: zero test coverage — jsonb_set clobber safety unverified
domains/metrics/src/metric-source/metric-source.queries.ts
The function's doc comment claims it does not clobber other sourceConfig keys (simulation_id, start, max_years_coverage), but there is no unit or integration test verifying this. A regression in the jsonb_set path or a future refactor could silently wipe billing-critical sourceConfig fields.
ProvisionSiteWithDevicesInvalidConfigError: source-config (egaugeColumn) invalid path has no integration test
domains/cross-domain/src/site-with-devices-provisioning.shells.ts
The shell validates both assetConfig and sourceConfig before the transaction. The assetConfig branch has integration test coverage; the sourceConfig (egaugeColumn) invalidConfig branch does not. This new error path can silently regress.
No test exercises all three synths (averageFee + powerFactor + billingFrequency) together
domains/utility/src/bill/__tests__/bill.decisions.test.ts
Each synth is tested in isolation. No test constructs a BatchCreateBillsState with all six conceptId fields populated. Interaction bugs (e.g. one synth stomping another's output via the appendSynth chain) would not be caught.
synthesizePowerFactorAdjustmentFromCommand: bonus with positive numericValue (data error) untested
domains/utility/src/bill/__tests__/bill.decisions.test.ts
PF-2 only tests a correctly-negative bonus. A malformed CFE lineItem where the bonus has a positive numericValue sums as a penalty contribution. Behavior is undocumented and untested.
loadDerivedSynthConceptIds: concept name→id mapping not directly tested
domains/utility/src/bill/bill.queries.ts
The function is only reachable through persistBatchBillsShell. No unit test stubs the DB and asserts the four concept names (powerFactorCharge, powerFactorBonus, powerFactorAdjustment, billingFrequency) map to the correct IDs. A mistyped concept name would surface only at runtime.
synthesizeBillingFrequencyFromCommand: invalid date string (NaN guard) untested
domains/utility/src/bill/__tests__/bill.decisions.test.ts
FREQ-5 covers null period dates but not invalid date strings (e.g. 'not-a-date'). The Number.isNaN guard is present but its branch is untested; a future refactor could silently break null-safety.
Non-string egauge_column input type (number, null) untested in parseProvisionBody
apps/platform/src/api/utils/__tests__/asset-management-validation.test.ts
Existing tests cover string acceptance and empty-string rejection but not { egauge_column: 42 } or { egauge_column: null } from a JSON body. The type guard `typeof s.egauge_column !== 'string'` handles these but the branch is unexercised.
findHelioscopeSourcesDueForTopUp: no real DB integration test for complex JOIN
domains/metrics/src/metric-source/metric-source.queries.ts
The 6-table JOIN with LEFT JOINs and cadence cutoff predicates is only covered by mock-chain builder tests. Schema-name mismatches or incorrect join conditions only surface against a real DB. An integration test with seeded data would provide stronger guarantees for this billing-critical query.
improvement3
validateSourceConfig pre-check fires only when egaugeColumn is truthy — guard too narrow
domains/cross-domain/src/site-with-devices-provisioning.shells.ts
The pre-transaction sourceConfig validation is gated on `if (metricStream.egaugeColumn)`. If the manifest's sourceConfigSchema evolves to require additional keys beyond egaugeColumn, those keys will bypass this check when egaugeColumn is absent. Passing the full prospective sourceConfig to validateSourceConfig unconditionally is more robust.
CDK Phase 7 no-op detection grep pattern is fragile to upstream wording changes
.github/workflows/preview-provision.yml
The grep pattern (no stacks match|no stacks to deploy|matched no stacks) is correct for CDK v2's current CLI output but CDK output messages are unversioned. A CDK CLI upgrade that rewrites these strings silently converts the benign no-op path back into a failing job. Consider pinning a comment with the CDK version verified against, or using a dedicated CDK exit code when available.
HELIOSCOPE_INTEGRATION_NAMES exported from queries.ts but has no external consumer yet
domains/metrics/src/metric-source/metric-source.queries.ts
The array simplifies the WHERE clause internally but is exported from the module barrel with no current external consumer. The daily-job Lambda branches on HELIOSCOPE_API_INTEGRATION_ID directly, not this array. Keep it unexported until there is an actual external caller.
History · 8 commits
- d6b4a35needs attentionincremental4H · 4M · 5L2026-07-05 20:29
- 678cb93safeincremental0H · 0M · 1L2026-07-05 17:40
- d219403needs attentionincremental1H · 3M · 4L2026-07-05 05:37
- 5ecd80cneeds attentionincremental1H · 2M · 1L2026-07-05 04:59
- 3fd1d03needs attentionincremental0H · 3M · 5L2026-07-05 04:36
- 1446f7eneeds attentionincremental2H · 4M · 7L2026-07-05 04:08
- ca73a1bneeds attentionfull4H · 10M · 14L2026-07-05 02:46
- c24738eneeds attentionincremental3H · 10M · 9L2026-07-04 05:12current