feat/multi-gran
needs attentionviewing older commit9312968 · incrementalPR #289reviewed 2026-07-09 04:04 UTC2H · 3M · 5L · 4I- Purpose
- Fix a correctness bug where the eGauge worker fetched the vendor CSV once at sources[0].granularity and mislabeled all channels with that cadence, even when the same variable was provisioned at two different native granularities (e.g. 5m recent / 15m aged for the eGauge retention split).
- Goal
- Let a single metrics collection run collect multiple granularities per device — each variable may declare several cadences — so the worker returns one MetricsPayload per (variable × granularity), keeping granularity keyed at the MetricSource row level with no schema or wire-shape change.
- Sub-goals
- SG-1: New pure groupChannelsByGranularity helper + contract doc clarification
- SG-2: eGauge worker — fetch per distinct granularity, project each group from its own CSV
- SG-3: Growatt worker — regression-test multi-granularity (no code change — already correct)
- SG-4: Aggregator — prove per-(variable,granularity) channel emission (no code change)
- SG-5: End-to-end SFN validation — DEFERRED (needs live AWS + seeded device)
- SG-6: Framework docs fold-back (metrics-pipeline.md, integrations CLAUDE.md, engine CLAUDE.md)
- What
- New pure `groupChannelsByGranularity` helper exported from `@batu/metrics-engine`. eGauge worker refactored to group channels by granularity, fetch the vendor CSV once per distinct cadence, project each group's channels from its own response. New Growatt handler test (package had none). New aggregator test proving two-source-row multi-gran grouping. Doc clarifications to invocation.types.ts (no field changes). Documentation fold-back: metrics-pipeline.md new section, integrations/CLAUDE.md new worker rule, engine/CLAUDE.md file map update.
- Why
- eGauge's minute DB uses a different decimation stride per granularity (5m→s=4, 15m→s=14), so a 15m channel MUST come from its own fetch — projecting it from a 5m CSV produced mislabeled cadence data. No schema or wire-shape change was needed: granularity is already part of the MetricSource uniqueness key and params.sources[] / MetricsPayload[] are already element-granular (BAT-189).
- Areas
- services/metrics/engine+175−9services/metrics/integrations/egauge+155−40services/metrics/integrations/growatt+113−0.claude/rules/metrics-pipeline.md+23−4.branch+150−0
- Blast
- 13 files, +625/-53 lines. Touches eGauge worker (production), engine lib + types (shared), new Growatt + aggregator tests, and docs. No schema changes, no API surface changes, no read-path changes. Claim-check / persist / Tinybird untouched.
Findings · 14
security2
params.sources[] cast bypasses runtime validation at the Lambda boundary
services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:53
resolveChannels returns `sources as MetricChannelSpec[]` — a TypeScript-only assertion on a `Record<string,unknown>`. The downstream `fetchHistoricalData` defends against unknown granularity strings via the `SLOT_SECONDS_BY_GRANULARITY` lookup, so no injection into the URL's `?s=` parameter is possible. However, other fields (`metricSourcePublicId`, `externalVariableId`, `egaugeColumn`) flow unvalidated into log lines and error strings. Risk is low because the invocation is AWS-internal (SFN→Lambda), but a Zod parse of `sources[]` would harden the boundary.
Multi-granularity fan-out fetch count is bounded by SLOT_SECONDS_BY_GRANULARITY lookup
services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:149
The loop issues one HTTP call per distinct granularity. Any unrecognized granularity string hits the SLOT_SECONDS_BY_GRANULARITY undefined check in fetchHistoricalData and returns immediately without a network call, bounding real fetch count to ≤2 (5m and 15m). Informational — no action required.
conventions3
Multi-paragraph docstring on a pure helper violates CLAUDE.md
services/metrics/engine/src/lib/group-channels-by-granularity.ts:1
CLAUDE.md prohibits multi-paragraph docstrings and comments that explain WHAT the code does. The 20-line file-level JSDoc has three paragraphs; paragraphs 1–2 describe the channel fan-out and sources[] layout — information the function name, parameter type, and @example already express. Only the FCIS justification ('Pure, no I/O → plain Map') and the ordering guarantee are non-obvious WHY content. Trim to one short comment covering those two points.
Barrel comment restates what the export name already says
services/metrics/engine/src/index.ts:64
The two-line comment ('Buckets a device's flat channel list… Pure; safe for workers.') explains WHAT `groupChannelsByGranularity` does, which its name already communicates. CLAUDE.md: 'don't explain WHAT the code does.' The section header '-- Multi-granularity collection helper --' is sufficient.
ChannelGranularity re-export has zero external consumers
services/metrics/engine/src/lib/group-channels-by-granularity.ts:26
ChannelGranularity is MetricChannelSpec['granularity'] under a new name, exported from the package barrel. Codebase grep finds no callers importing it — not the eGauge worker, not the tests (which use MetricChannelSpec['granularity'] inline). Adding it as a public surface without a consumer adds indirection without benefit. Keep as file-local or drop the export.
tests5
Growatt '1d' unit rule is dead code — test proves an unreachable branch
services/metrics/integrations/growatt/src/__tests__/metrics-handler.test.ts:98
The test asserts `p1d.units['solar_generation'] === 'kWh'` by mocking `fetchHistoricalData` to return `response.granularity = '1d'`. In production the real Growatt `fetchHistoricalData` rejects any `granularity !== '5m'` with `{ ok: false, reason: 'unsupported-granularity' }` before issuing any HTTP call, and always echoes `granularity: '5m'` in the success response. So `result.response.granularity === '1d'` is never true in production — the kWh branch is dead code. The test passes only because the mock bypasses the real rejection. Either the '1d' granularity check in the handler is vestigial dead code that should be removed, or Growatt's `fetchHistoricalData` should be extended to support '1d' natively.
Growatt partial-success test describes an impossible '1d' scenario
services/metrics/integrations/growatt/src/__tests__/metrics-handler.test.ts:102
The partial-success test mocks the '1d' channel failing with `{ ok: false, reason: 'upstream-error' }`. In production a '1d' channel would fail with `{ ok: false, reason: 'unsupported-granularity' }` (not an upstream error), which maps to `TranslationFailed`, not `UpstreamUnavailable`. The plumbing (partial success returns one payload) is still exercised, but the semantic scenario described — a real granularity fetch that fails transiently — cannot actually happen for '1d' on Growatt.
eGauge csv() helper passes only by coincidence of fromIso matching window.from
services/metrics/integrations/egauge/src/__tests__/metrics-handler.test.ts:26
`pointsToBatu` clips rows to [fromUtc, toUtc). The csv() helper always starts rows at `fromIso`, which happens to match `window.from = '2026-01-01T00:00:00Z'`. If the helper's fromIso drifted outside the window, `points[0]` would be undefined and the `p5.points[0].value` assertion would throw a non-obvious error. Passing `fromIso = window.from` explicitly or adding a comment explaining the dependency would harden the helper.
Aggregator multi-gran test doesn't validate backward-compat scalar params
services/metrics/engine/src/handlers/__tests__/site-context-aggregator.test.ts:188
The existing single-channel test locks in that `params.externalVariableId` and `params.metricSourcePublicId` mirror `sources[0]` for backward-compat with direct-invoke callers. The new multi-granularity test (two sources, one variable) doesn't assert these scalar fields, leaving it possible for a regression to assign them to the wrong source without detection.
No test for eGauge describe mode with multi-granularity sources
services/metrics/integrations/egauge/src/__tests__/metrics-handler.test.ts:116
The describe mode (line ~116 of metrics.lambda.ts) fetches at '5m' hardcoded regardless of channel granularities. Pre-existing gap, not introduced by this diff, but worth noting for a future test pass.
improvement4
channelErrors mixes fetch-level and channel-level failures in the done log
services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:160
When a granularity fetch fails, one entry per channel in that group is pushed to channelErrors (alongside projection-level failures). On total failure the log 'fail.translate' details string echoes both — fetch events appear N times (once per channel). Separating fetch-level echoes from projection failures, or only appending to channelErrors on projection failure, would make the log unambiguous.
granularitiesFetched logs attempted count, not successful count
services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:195
groups.size counts all granularities including those whose fetch failed. On partial success, `granularitiesFetched: 2` is logged alongside `channelsProjected: 1`, which is misleading. Renaming to `granularitiesAttempted` or logging `{ granularitiesOk: groups.size - fetchErrors.length, granularitiesAttempted: groups.size }` would make metrics self-consistent.
ChannelGranularity type alias is an indirection without added expressiveness
services/metrics/engine/src/lib/group-channels-by-granularity.ts:26
The alias adds no constraint beyond MetricChannelSpec['granularity']. If not intended as a stable re-export for consumers, inlining and dropping the export keeps the public surface minimal.
Sequential granularity fetches — fine for 1-2 gran case, note if fan-out grows
services/metrics/integrations/egauge/src/handlers/metrics.lambda.ts:149
Granularity fetches are awaited serially. eGauge has no session state between fetches (unlike Growatt), so parallelising via Promise.allSettled would be safe. At 1-2 granularities the latency difference is negligible. Worth a short comment to signal the serialisation is intentional, not an oversight.