← all branches

feat/multi-gran

needs attentionviewing older commit
9312968 · incrementalPR #289reviewed 2026-07-09 04:04 UTC2H · 3M · 5L · 4I
The branch
Purpose
Fix a correctness bug where the eGauge worker fetched the vendor CSV once at sources[0].granularity and mislabeled all channels with that cadence, even when the same variable was provisioned at two different native granularities (e.g. 5m recent / 15m aged for the eGauge retention split).
Goal
Let a single metrics collection run collect multiple granularities per device — each variable may declare several cadences — so the worker returns one MetricsPayload per (variable × granularity), keeping granularity keyed at the MetricSource row level with no schema or wire-shape change.
Sub-goals
  • SG-1: New pure groupChannelsByGranularity helper + contract doc clarification
  • SG-2: eGauge worker — fetch per distinct granularity, project each group from its own CSV
  • SG-3: Growatt worker — regression-test multi-granularity (no code change — already correct)
  • SG-4: Aggregator — prove per-(variable,granularity) channel emission (no code change)
  • SG-5: End-to-end SFN validation — DEFERRED (needs live AWS + seeded device)
  • SG-6: Framework docs fold-back (metrics-pipeline.md, integrations CLAUDE.md, engine CLAUDE.md)
The changes (whole branch)
What
New pure `groupChannelsByGranularity` helper exported from `@batu/metrics-engine`. eGauge worker refactored to group channels by granularity, fetch the vendor CSV once per distinct cadence, project each group's channels from its own response. New Growatt handler test (package had none). New aggregator test proving two-source-row multi-gran grouping. Doc clarifications to invocation.types.ts (no field changes). Documentation fold-back: metrics-pipeline.md new section, integrations/CLAUDE.md new worker rule, engine/CLAUDE.md file map update.
Why
eGauge's minute DB uses a different decimation stride per granularity (5m→s=4, 15m→s=14), so a 15m channel MUST come from its own fetch — projecting it from a 5m CSV produced mislabeled cadence data. No schema or wire-shape change was needed: granularity is already part of the MetricSource uniqueness key and params.sources[] / MetricsPayload[] are already element-granular (BAT-189).
Areas
services/metrics/engine+1759services/metrics/integrations/egauge+15540services/metrics/integrations/growatt+1130.claude/rules/metrics-pipeline.md+234.branch+1500
Blast
13 files, +625/-53 lines. Touches eGauge worker (production), engine lib + types (shared), new Growatt + aggregator tests, and docs. No schema changes, no API surface changes, no read-path changes. Claim-check / persist / Tinybird untouched.
SG-5 end-to-end validation deferred — needs supervised session with live AWS + multi-gran seeded device
ci· No CI checks registered on this PR yetcoderabbit· No .coderabbit.yaml present

Findings · 14

security2

low

params.sources[] cast bypasses runtime validation at the Lambda boundary

services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:53

resolveChannels returns `sources as MetricChannelSpec[]` — a TypeScript-only assertion on a `Record<string,unknown>`. The downstream `fetchHistoricalData` defends against unknown granularity strings via the `SLOT_SECONDS_BY_GRANULARITY` lookup, so no injection into the URL's `?s=` parameter is possible. However, other fields (`metricSourcePublicId`, `externalVariableId`, `egaugeColumn`) flow unvalidated into log lines and error strings. Risk is low because the invocation is AWS-internal (SFN→Lambda), but a Zod parse of `sources[]` would harden the boundary.

info

Multi-granularity fan-out fetch count is bounded by SLOT_SECONDS_BY_GRANULARITY lookup

services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:149

The loop issues one HTTP call per distinct granularity. Any unrecognized granularity string hits the SLOT_SECONDS_BY_GRANULARITY undefined check in fetchHistoricalData and returns immediately without a network call, bounding real fetch count to ≤2 (5m and 15m). Informational — no action required.

conventions3

medium

Multi-paragraph docstring on a pure helper violates CLAUDE.md

services/metrics/engine/src/lib/group-channels-by-granularity.ts:1

CLAUDE.md prohibits multi-paragraph docstrings and comments that explain WHAT the code does. The 20-line file-level JSDoc has three paragraphs; paragraphs 1–2 describe the channel fan-out and sources[] layout — information the function name, parameter type, and @example already express. Only the FCIS justification ('Pure, no I/O → plain Map') and the ordering guarantee are non-obvious WHY content. Trim to one short comment covering those two points.

low

Barrel comment restates what the export name already says

services/metrics/engine/src/index.ts:64

The two-line comment ('Buckets a device's flat channel list… Pure; safe for workers.') explains WHAT `groupChannelsByGranularity` does, which its name already communicates. CLAUDE.md: 'don't explain WHAT the code does.' The section header '-- Multi-granularity collection helper --' is sufficient.

low

ChannelGranularity re-export has zero external consumers

services/metrics/engine/src/lib/group-channels-by-granularity.ts:26

ChannelGranularity is MetricChannelSpec['granularity'] under a new name, exported from the package barrel. Codebase grep finds no callers importing it — not the eGauge worker, not the tests (which use MetricChannelSpec['granularity'] inline). Adding it as a public surface without a consumer adds indirection without benefit. Keep as file-local or drop the export.

tests5

high

Growatt '1d' unit rule is dead code — test proves an unreachable branch

services/metrics/integrations/growatt/src/__tests__/metrics-handler.test.ts:98

The test asserts `p1d.units['solar_generation'] === 'kWh'` by mocking `fetchHistoricalData` to return `response.granularity = '1d'`. In production the real Growatt `fetchHistoricalData` rejects any `granularity !== '5m'` with `{ ok: false, reason: 'unsupported-granularity' }` before issuing any HTTP call, and always echoes `granularity: '5m'` in the success response. So `result.response.granularity === '1d'` is never true in production — the kWh branch is dead code. The test passes only because the mock bypasses the real rejection. Either the '1d' granularity check in the handler is vestigial dead code that should be removed, or Growatt's `fetchHistoricalData` should be extended to support '1d' natively.

high

Growatt partial-success test describes an impossible '1d' scenario

services/metrics/integrations/growatt/src/__tests__/metrics-handler.test.ts:102

The partial-success test mocks the '1d' channel failing with `{ ok: false, reason: 'upstream-error' }`. In production a '1d' channel would fail with `{ ok: false, reason: 'unsupported-granularity' }` (not an upstream error), which maps to `TranslationFailed`, not `UpstreamUnavailable`. The plumbing (partial success returns one payload) is still exercised, but the semantic scenario described — a real granularity fetch that fails transiently — cannot actually happen for '1d' on Growatt.

medium

eGauge csv() helper passes only by coincidence of fromIso matching window.from

services/metrics/integrations/egauge/src/__tests__/metrics-handler.test.ts:26

`pointsToBatu` clips rows to [fromUtc, toUtc). The csv() helper always starts rows at `fromIso`, which happens to match `window.from = '2026-01-01T00:00:00Z'`. If the helper's fromIso drifted outside the window, `points[0]` would be undefined and the `p5.points[0].value` assertion would throw a non-obvious error. Passing `fromIso = window.from` explicitly or adding a comment explaining the dependency would harden the helper.

medium

Aggregator multi-gran test doesn't validate backward-compat scalar params

services/metrics/engine/src/handlers/__tests__/site-context-aggregator.test.ts:188

The existing single-channel test locks in that `params.externalVariableId` and `params.metricSourcePublicId` mirror `sources[0]` for backward-compat with direct-invoke callers. The new multi-granularity test (two sources, one variable) doesn't assert these scalar fields, leaving it possible for a regression to assign them to the wrong source without detection.

info

No test for eGauge describe mode with multi-granularity sources

services/metrics/integrations/egauge/src/__tests__/metrics-handler.test.ts:116

The describe mode (line ~116 of metrics.lambda.ts) fetches at '5m' hardcoded regardless of channel granularities. Pre-existing gap, not introduced by this diff, but worth noting for a future test pass.

improvement4

low

channelErrors mixes fetch-level and channel-level failures in the done log

services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:160

When a granularity fetch fails, one entry per channel in that group is pushed to channelErrors (alongside projection-level failures). On total failure the log 'fail.translate' details string echoes both — fetch events appear N times (once per channel). Separating fetch-level echoes from projection failures, or only appending to channelErrors on projection failure, would make the log unambiguous.

info

granularitiesFetched logs attempted count, not successful count

services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:195

groups.size counts all granularities including those whose fetch failed. On partial success, `granularitiesFetched: 2` is logged alongside `channelsProjected: 1`, which is misleading. Renaming to `granularitiesAttempted` or logging `{ granularitiesOk: groups.size - fetchErrors.length, granularitiesAttempted: groups.size }` would make metrics self-consistent.

info

ChannelGranularity type alias is an indirection without added expressiveness

services/metrics/engine/src/lib/group-channels-by-granularity.ts:26

The alias adds no constraint beyond MetricChannelSpec['granularity']. If not intended as a stable re-export for consumers, inlining and dropping the export keeps the public surface minimal.

info

Sequential granularity fetches — fine for 1-2 gran case, note if fan-out grows

services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:149

Granularity fetches are awaited serially. eGauge has no session state between fetches (unlike Growatt), so parallelising via Promise.allSettled would be safe. At 1-2 granularities the latency difference is negligible. Worth a short comment to signal the serialisation is intentional, not an oversight.

History · 3 commits

  1. e4f79c8safeincremental0H · 0M · 1L2026-07-09 05:04
  2. 9312968needs attentionincremental2H · 3M · 5L2026-07-09 04:04current
  3. 0544fe2needs attentionfull7H · 6M · 6L2026-07-09 03:40