← all branches

feat/victron

needs attention
d5b82d2 · fullpre-PRreviewed 2026-07-07 20:52 UTC4H · 7M · 10L · 4I
The branch
Purpose
Port the Victron VRM integration into the v2 metrics engine as the spike brand — the FIRST of 12 device-integration ports — proving the full 10-step port checklist end-to-end on the smallest fleet (4 devices).
Goal
Establish the canonical pattern for token-auth device integrations: worker package (metrics + connection), CDK stack, seeds, live Site-SFN validation, and framework learnings for the remaining 11 ports.
Sub-goals
  • SG-1: VRM API probe + worker package (engine + translation + both handlers) with unit tests — 15m native confirmed
  • SG-2: CDK IntegrationVictronLambdaStack + main.ts wiring + coordinator SSM/invoke grants; synth verified
  • SG-3: Preview deployed; catalog+sites seeded; D2 secrets copied+emitted; D3 4 devices/12 streams; ARNs seeded post-deploy
  • SG-4: 3 Site-SFN runs + 2 Connection-SFN probes; 740 Tinybird points; 9/9 day-sum parity ≤0.02% vs legacy DDB; 15m native achieved
  • SG-N: 11 framework learnings → scope.md; Victron Tier-3 caveats → integrations CLAUDE.md
The changes (whole branch)
What
New services/metrics/integrations/victron/ package (metrics + connection Lambda workers, VRM HTTP client, pure translation layer, 546 unit tests); CDK IntegrationVictronLambdaStack; coordinator wiring; seed scripts; integration manifests for 12 brands (feat/int-base batch); SecretProvider type extensions; legacy-registry/secrets migration tooling; integration CLAUDE.md with caveats for all 12 ports.
Why
Victron is the spike brand that validates the full porting checklist before batch-porting the remaining 11 brands. Live validation at a real Mexican solar site (9/9 day-sum parity vs legacy DDB at ≤0.02%) proves the translation mechanics, DST-safe local-midnight anchoring, and the eGauge-shaped CDK pattern.
Areas
services/metrics/integrations/victron+17220infra/cdk+3982packages/integration-manifests+10712scripts/metrics+17190packages/database+25714domains/core+161.branch (docs/tooling)+311163services/metrics/integrations (CLAUDE.md+STANDARDS)+626domains/metrics+54
Blast
58 files, +5961/-192 across 9 areas. Core blast: new Lambda package + CDK stack (isolated to victron namespace). Secondary blast: 12 integration manifests added (all additive), SecretProvider type union extended (additive), coordinator stack gets 2 new SSM reads + invoke grants. No existing functionality modified — pure additions with coordinator wiring.
First of 12 device-integration ports — patterns established here will replicate Live validated against real Victron site: 9/9 day parity ≤0.02% vs legacy DDB 5.9k net additions, ~546 unit tests across 3 test files 11 framework learnings documented in scope.md for batch-porting remainder
CI· No PR open for this branch; CI status not availableCodeRabbit· No .coderabbit.yaml in repo

Findings · 25

correctness3

medium

Window cap check uses `alignedFrom`, which can reject valid at-cap requests

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

windowDays is computed as (toUtc − alignedFrom) / 86_400_000. Since alignWindowStart always floors the start backward (by up to 14m59s for 15m, 59m59s for 1h), a request of exactly 31.0 days can become 31.01+ days after alignment and get rejected with 'exceeds VRM cap' even though the original window was within bounds. A correct check would compare against the original fromUtc span, or subtract the alignment delta from the cap. Most impactful for the 1h granularity where up to 59 minutes are added.

low

Future `last_timestamp` from VRM produces negative secsAgo and falsely reports device online

services/metrics/integrations/victron/src/handlers/connection.lambda.ts

secsAgo = nowEpochSec − last. If VRM returns a last_timestamp in the future (device clock skew), secsAgo is negative, making `secsAgo <= 900` true. The device would be reported reachable:true with a misleading 'last update -N seconds ago' detail. Given that VRM is a cloud intermediary that normalizes timestamps, this is unlikely but possible under device clock drift.

info

Partial channel success emits no signal to coordinator that some channels were skipped

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

When some channels project successfully (payloads.length > 0) and some fail (channel-not-found), the handler returns InvocationSuccess with only the successful payloads — logs the skipped channels but the coordinator has no structured way to know grid_import data is missing from the collection window. Consistent with the documented design ('Only fail if ALL fail'), but a consistently-absent channel produces no alert — only log noise. An observability gap that could mask a misconfigured MetricSource.

security4

medium

IAM wildcard allows worker to read any org's Victron secret

infra/cdk/src/stacks/services/metrics/integrations/victron/iam.ts

The policy grants GetSecretValue on `{env}/victron/*` and `batu/{env}/metrics/victron/*`. Both are per-env wildcards covering every organisation's Victron credential. A compromised worker Lambda could retrieve another org's VRM token by passing any matching ARN. In the normal SFN path this cannot be triggered by end-user input (secretConfigArn is DB-sourced), but defence-in-depth is absent at the IAM layer. Consider per-org path segments (e.g. `batu/{env}/metrics/victron/{orgId}/*`) or explicitly document the intentional cross-org trust boundary.

low

`secretConfigArn` is not validated against the expected prefix before use

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

inv.secretConfigArn is passed directly to loadCredentialFromArn → GetSecretValueCommand without a code-level check that it matches the expected victron/* prefix. IAM mitigates this (the Lambda role rejects reads outside those paths), but if the IAM policy were ever widened, a crafted Invocation could cause the worker to fetch an unrelated secret. Defence-in-depth runtime prefix check would be independent of IAM policy correctness. Same in connection.lambda.ts.

low

Silent `catch {}` in `loadCredentialFromArn` masks IAM/permission failures as `MissingSecret`

services/metrics/integrations/victron/src/engine/vrm-client.ts

The catch block swallows all Secrets Manager errors — including AccessDeniedException (misconfigured/revoked IAM policy) and ResourceNotFoundException — surfacing them identically as MissingSecret. An IAM regression stripping GetSecretValue from the Lambda role would be indistinguishable from a genuinely absent secret, delaying diagnosis. Consider distinguishing AccessDeniedException (log a distinct warning) from ResourceNotFoundException (current null treatment is correct).

info

`fail.exception` log may echo fetch-stack error messages

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

The outer catch logs e.message from any unhandled exception. Currently this will be runtime errors, not VRM network errors (caught inside vrmGet). However, if a future refactor moves credential handling outside the try block or if Node.js fetch surfaces the request URL in an exception message, the token in x-authorization could theoretically appear in logs. The explicit comment 'NEVER log the payload' in loadCredentialFromArn is good hygiene and should extend to the catch-all handler.

conventions2

medium

`TranslationFailed` always maps to statusCode:500 — input validation errors classified as infrastructure errors

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

translationFailed() hard-codes statusCode:500, so validation failures like 'invalid window', 'window exceeds VRM cap', and 'missing timezone' are emitted with status:500 and become InvocationFailure.status='error' (not 'fail'). These are logical precondition failures, which the JSend contract classifies as 'fail' (4xx). The established ecosystem does this the same way (eGauge identical) — so this is a contract-level inherited semantic choice, not a Victron deviation. However, it means the coordinator's retry/alerting logic cannot distinguish a misconfigured invocation from a genuine infrastructure breakdown. This semantic mismatch should be addressed upstream before 11 more ports inherit it.

low

Manifest `externalVariableId: 'solarGeneration'` is a legacy vestige — never used in routing

packages/integration-manifests/src/manifests/victron.ts

The victron manifest declares solar_generation.externalVariableId as 'solarGeneration' (the legacy VRM channel name). Per metrics-pipeline.md gotcha #1 and resolveExternalVariableId(), what gets stored in MetricSource.source_config.externalVariableId is the manifest VARIABLE KEY ('solar_generation'), not this inner field. The worker routes on that key against CHANNEL_ATTRIBUTE_CODES. The inner externalVariableId field is structurally unused in routing. A future maintainer could misinterpret it as the routing key. A clarifying comment or normalization to the canonical key would prevent confusion.

tests10

high

Mixed-granularity invocation (15m + 1d channels) — two VRM fetches not tested

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

metrics.lambda.ts groups channels by granularity and calls fetchStats ONCE PER DISTINCT GRANULARITY (the for-loop over `granularities`). Every test uses only same-granularity channels, so the multi-fetch path is never exercised. A bug in how `recordsByGranularity` is populated or keyed for the second granularity would go undetected. Add a test with sources: [{…, granularity:'15m'}, {…, granularity:'1d'}] and assert fetchStats is called TWICE with the correct `interval` each time and both payloads returned.

high

`resolveChannels` scalar-params fallback path is untested

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

When `params.sources` is absent or empty, `resolveChannels` constructs a single-channel spec from scalar params (params.metricSourcePublicId, params.externalVariableId, params.granularity), falling back to DEFAULT_CHANNEL ('solar_generation') and DEFAULT_GRANULARITY ('15m'). No test exercises this branch. If the fallback defaults or the type cast to MetricChannelSpec are wrong, older or direct-invoke callers silently get the wrong channel. This path will be copied into 11 more ports.

high

Zero-length window (`fromUtc === toUtc`) not tested — boundary in guard

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

The handler guards `fromUtc >= toUtc`, which rejects both the equal case and reversed windows. No test directly covers `from === to`. A direct test with an equal window should confirm the TranslationFailed path. The equal case is most likely to appear from a misconfigured aggregator emitting a zero-width window.

high

`loadCredentialFromArn` returning null not tested in the connection handler

services/metrics/integrations/victron/src/__tests__/connection-handler.test.ts

The metrics handler tests the null-credential path (MissingSecret). The connection handler has identical logic but the connection-handler test file has no corresponding case. A regression where the connection handler throws instead of returning a clean MissingSecret on null credential would go undetected.

medium

`fetchOwnUserId` failure path not tested in connection handler

services/metrics/integrations/victron/src/__tests__/connection-handler.test.ts

The connection handler calls fetchOwnUserId when credential.userId is absent and, on failure, returns a success-probe with probeFailurePayload. The failure path (fetchOwnUserId returns {ok:false}) is never tested. Test needed: loadCredentialFromArn returns {token:'tok'} (no userId) AND fetchOwnUserId returns {ok:false, reason:'auth-failed', …} → result should be success (probe) with credentialValid:false.

medium

`probeFailurePayload` — only 2 of 7 VRM failure reasons exercised

services/metrics/integrations/victron/src/__tests__/connection-handler.test.ts

probeFailurePayload has branching logic: reason==='auth-failed' produces a specific detail string; all others use the details field verbatim. The remaining VRM failure reasons (rate-limited, timeout, http-error, vendor-rejected, unparseable) are neither tested via the handler mock nor directly as a unit test of the exported pure function. Direct unit tests for each reason would pin the auth-failed branch and guard against regressions.

medium

`kwhToMeanWatts` 1h test uses trivially-satisfied value

services/metrics/integrations/victron/src/__tests__/points-to-batu.test.ts:37

The 1h test passes `1 kWh` and expects `1000 W` — trivially satisfied by almost any formula that divides by slotSeconds (1×1000×3600/3600=1000). The 15m test uses `0.25 kWh → 1000 W` (non-trivial) and the 1d test adds `48.897 kWh → …`. The 1h case should get a non-trivial value (e.g. 0.5 kWh → 500 W) to differentiate a correct formula.

medium

1d window cap (180 days) not tested — only the 31-day 15m cap is covered

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

MAX_WINDOW_DAYS defines different caps: 15m/1h → 31 days, 1d → 180 days. Only the 15m 31-day breach is tested. There is no test confirming (a) a 1d request within 180 days succeeds; (b) a 1d request > 180 days is rejected. The 180-day cap path is unexercised.

low

`vendor-rejected` fetch reason not tested via metrics handler end-to-end

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

mapFetchError unit tests cover vendor-rejected → UpstreamUnavailable, but no test mocks fetchStats returning {ok:false, reason:'vendor-rejected'} and asserts on the handler's overall status/errors. The unit test pins the error mapping but not the handler-level wiring for this reason.

low

`grid_export` channel not exercised end-to-end through the metrics handler

services/metrics/integrations/victron/src/__tests__/metrics-handler.test.ts

The handler tests use only solar_generation and grid_import. The pointsToBatu unit tests confirm grid_export shares Pg with solar_generation by reference but do not test a full projection for grid_export (codes: Pg+Bg). Through the handler, grid_export has never been requested — a typo in CHANNEL_ATTRIBUTE_CODES.grid_export would not be caught.

improvement6

low

Envelope helpers (`success`/`failure`) duplicated 6 ways, heading for 28 with remaining 11 ports

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

success<T>() and failure() are byte-for-byte identical across growatt/metrics, growatt/explore, egauge/metrics, egauge/connection, victron/metrics, victron/connection (6 files). With 11 more ports, this reaches 28 copies. @batu/metrics-engine already exports InvocationSuccess, InvocationFailure, Invocation — the building blocks. Exporting invocationSuccess()/invocationFailure() helpers from services/metrics/engine/src/lib/ eliminates this before it replicates. Concrete divergence risk: TraceFields has a workerRequestId? field absent from all 6 current helpers — they can diverge independently.

low

`resolveChannels` duplicated across 3 metrics handlers with no shared home

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

resolveChannels(params, fallbackSourceId) is verbatim in growatt, egauge, and victron metrics handlers. This BAT-189 boilerplate will appear in every metrics worker. A shared resolveChannelsFallback() in @batu/metrics-engine/lib means one deletion point when the scalar-params legacy path is retired, instead of 14. The eGauge extension (egaugeColumn spread) can be handled via an options object.

low

`alignWindowStart` called twice with identical arguments in the channel projection loop

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

alignWindowStart(fromUtc, granularity, timezone ?? 'UTC') is computed at line 145 (stored as alignedFrom for the cap check and fetchStats call) and recomputed inside the for (const ch of channels) loop for every channel projection. All channels sharing a granularity call it with the same three inputs. Fix: populate an alignedFromByGranularity: Map alongside recordsByGranularity in the fetch loop and read it in the projection loop. alignWindowStart uses Intl.DateTimeFormat on the 1d path — non-trivial cost. This pattern will be copied into 11 more ports.

low

Sequential granularity fetches; 90s timeout sized exactly to worst case with no margin

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

The fetch loop awaits each granularity serially. For a mixed 15m+1d collection, two serial worst-case fetches consume 60s of the 90s budget. Promise.all(granularities.map(...fetchStats)) halves worst-case latency. If sequential is intentional (e.g. VRM per-org rate-limit), document it explicitly — otherwise each of the 11 remaining ports will independently debate and decide.

info

`resolveChannels` comment says 'older callers' but Victron has none in the TypeScript system

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

The JSDoc copies 'keeps direct invokes / older callers working' from eGauge/Growatt, where it's accurate (live legacy paths being migrated). Victron's prior callers are in the Python monorepo — there are no TypeScript 'older callers'. As the template for 11 ports, copying this comment as-is makes it impossible to audit which integrations genuinely have migration debt vs. which adopted the pattern speculatively.

info

Window cap validation mixed into fetch loop — partial VRM calls before validation failure

services/metrics/integrations/victron/src/handlers/metrics.lambda.ts

The per-granularity window cap check sits inside the fetch loop. For a 2-granularity request where 15m is valid and 1d is oversized, the 15m VRM call fires before the 1d cap error is returned. Extracting all cap checks into a validation pass before the try block (alongside jobType/asset/window/timezone checks) makes 'validate all before any I/O' the explicit contract. For 11 ports copying this pattern, structural separation reduces the risk of partial-network-call validation failures becoming entrenched.