← all branches

feat/multi-gran

needs attentionviewing older commit
0544fe2 · fullpre-PRreviewed 2026-07-09 03:40 UTC7H · 6M · 6L · 3I
The branch
Purpose
Fix the eGauge worker so a device configured with the same variable at multiple granularities (e.g. solar_generation at 5m and 15m) collects each granularity independently in a single Site SFN run.
Goal
Let a single metrics collection run collect multiple granularities per device — one MetricsPayload per (variable × granularity), keeping granularity keyed at the MetricSource row level (no schema change).
Sub-goals
  • SG-1: groupChannelsByGranularity helper + contract doc clarification
  • SG-2: eGauge worker — fetch per distinct granularity
  • SG-3: Growatt worker — verify + regression-test multi-granularity
  • SG-4: Aggregator — prove per-(variable,granularity) channel emission
  • SG-5: End-to-end validation via Site SFN (branch preview)
  • SG-6: Framework improvements — fold learnings back
The changes (whole branch)
What
Branch provisioned with detailed scope.md (99 lines) and empty intent.md. No code yet.
Why
eGauge devices serve 5m data on the live tier and 15m natively once the fine tier ages out (~12 months). The current worker fetches once at channels[0].granularity and projects all channels from that single CSV — a 15m MetricSource gets 5m-resolution data, violating metrics-pipeline.md P4 (store native granularity only).
Areas
.branch/scope.md+990.branch/intent.md+00
Blast
2 files, +99 lines. Provision-only. When implemented: egauge worker (primary), engine/lib (new helper), growatt (tests), .claude/rules (docs). No schema/auth/API changes.
ci· provision-only commit, no CIcoderabbit· no .coderabbit.yaml

Findings · 23

correctness5

critical

SC-1 uses '@ 1d' — eGauge rejects 1d with unsupported-granularity

.branch/scope.md

Success Criterion 1 gives 'solar_generation @ 5m and @ 1d' as the target example. fetchHistoricalData only supports 5m and 15m; 1d returns {ok:false, reason:'unsupported-granularity'}. Under the fan-out loop, the 1d group fails and that MetricSource never gets points. SC-1 should use 5m + 15m.

high

Bug described as 'mislabeling' — actual impact is wrong-cadence data (wrong density)

.branch/scope.md

The scope says 'no more channels[0].granularity mislabeling'. The real bug: a 15m channel's payload contains 5m-resolution data from the 5m CSV fetch. Wrong data density, not just a label issue. Tests must verify data cadence, not just payload granularity field.

high

fail-only-if-ALL shifts meaning under per-granularity fan-out — undocumented

.branch/scope.md

Under the new design, 'fail only if ALL fail' means 'fail only if ALL granularity groups produce zero payloads'. A 15m group failure is invisible at the invocation level. SG-2 details must explicitly state the cross-group fail semantics and how per-group failures are reported.

medium

1h/1d provisioning gap becomes more reachable — no guard documented

.branch/scope.md

MetricChannelSpec.granularity accepts '1h' | '1d' but eGauge rejects them at runtime. The multi-gran feature makes this more likely to be hit in practice. Document as a known gap, defer provision-time guard to a future issue.

low

Growatt same-variable-two-granularities routing needs pre-verification

.branch/scope.md

The plan assumes Growatt 'already works' for same-variable at two granularities. Verify growatt/src/engine/historical-data.ts routes on (channel, granularity) before writing SG-3 tests.

security3

medium

No cap on distinct granularities per device — unbounded fan-out

.branch/scope.md

groupChannelsByGranularity issues one fetchHistoricalData call per distinct granularity. No stated upper bound. Recommend capping at ≤4 distinct granularities per device, returning InvocationError if exceeded. Bounds Lambda cost and external API hammering.

low

S3 write amplification scales with granularity count

.branch/scope.md

More granularities = more S3 PUTs and Tinybird ingest ops per collection run. Bounded by provisioning access control, but not truly 'free' at scale. Acknowledge in cost model.

info

No new auth surface — confirmed safe

.branch/scope.md

Credential handling is unchanged. No new I/O, secret access, or trust boundary.

conventions3

medium

No `Granularity` type exists — Map key must use MetricChannelSpec['granularity']

.branch/scope.md

The plan names the helper `Map<Granularity, MetricChannelSpec[]>` but no `Granularity` type is exported from @batu/metrics-engine. Either export `type Granularity = MetricChannelSpec['granularity']` from the engine barrel or use the indexed access type directly.

info

Per-granularity fail is a new failure mode — should be in SG-2 details

.branch/scope.md

The 'fail only if ALL' cross-group semantics need to be documented explicitly in SG-2 to avoid an implementer defaulting to fail-fast on the first granularity group error.

info

SG-6 framework doc update follows established pattern

.branch/scope.md

Modifying .claude/rules/metrics-pipeline.md and services/metrics/integrations/CLAUDE.md in SG-6 is consistent with the project framework.

tests7

critical

Partial fetch failure not tested: 5m fetch succeeds, 15m fetch fails

.branch/scope.md

The most dangerous scenario — one granularity group fails while others succeed — has no planned test. The invocation would return success with only the successful group's payloads while the other MetricSource silently stops collecting. Need an explicit test for this partial-failure path.

high

eGauge test plan must assert fetchHistoricalData call count and granularity args

.branch/scope.md

SG-2 plan says 'two-gran ⇒ two fetches' but does not specify spying on fetchHistoricalData to assert EXACTLY TWO calls with distinct granularity arguments. The test could pass even with a single-fetch implementation if both test fixture channels have the same granularity.

high

describe mode regression not planned — must survive the refactor

.branch/scope.md

The plan preserves describe mode but no test is planned. After the refactor, if groupChannelsByGranularity logic accidentally touches the describe path, the result silently shifts from DescribeColumnsPayload to MetricsPayload[]. Add: mode=describe → single fetch at '5m', result is DescribeColumnsPayload.

high

SG-1 helper test missing critical edge cases

.branch/scope.md

Test cases unspecified. Required: (1) empty array → empty Map; (2) all-same-granularity → 1-entry Map; (3) order preservation within buckets; (4) behavior for '1h'/'1d' entries — included in Map (causing later fetch failure) or pre-filtered? This contract choice has downstream error consequences.

high

SG-4 aggregator test must assert each channel's granularity field

.branch/scope.md

The test description does not mention asserting that each channel in params.sources[] carries the CORRECT granularity. Without this, the test passes even if the aggregator collapses both to the same granularity — the same class of bug as the eGauge worker, shifted upstream.

medium

No test for unsupported-granularity path in the fan-out loop

.branch/scope.md

fetchHistoricalData returns unsupported-granularity for 1h/1d. Under the new fan-out, a 1d channel group hits this path. Need an integration test: 5m group succeeds, 1d group hits unsupported-granularity → TranslationFailed → invocation succeeds with only 5m payloads.

low

SG-5 E2E should verify per-source coverage window advancement

.branch/scope.md

Verify that metric_coverages advanced independently for each MetricSource after the SFN run. If not, subsequent collection triggers will skip one granularity's re-collection.

improvement5

high

'Verify non-5m cadences' is misleading — 15m already confirmed; critical check is 1h/1d rejection

.branch/scope.md

fetchHistoricalData already supports 15m (in SLOT_SECONDS_BY_GRANULARITY). The important verification is confirming 1h/1d returns unsupported-granularity and the fan-out loop handles it gracefully. Rephrase SG-2's verification step.

medium

DevicePayload top-level granularity scalar becomes misleading for multi-granularity devices

.branch/scope.md

The site-context-aggregator sets a device-level granularity from the first channel. For a 5m+15m device this scalar is meaningless. Acknowledge as a known limitation in SG-6 (params.sources[].granularity is the authoritative field).

medium

Sequential granularity fetches may double Lambda wall-clock time

.branch/scope.md

Two granularities = two sequential fetchHistoricalData calls. If each is ~3-5s, the invocation wall-clock doubles. SG-2 should document whether concurrent fetches (Promise.all over Map entries) are better or sequential is acceptable given Lambda timeout headroom.

low

SG-3 Growatt test uses '1d' — verify Growatt manifest declares 1d

.branch/scope.md

If the Growatt manifest only declares 5m, a 1d test fixture creates a misleading regression guard. Verify @batu/integration-manifests/src/manifests/growatt.ts before choosing fixture granularities.

low

Map iteration order should be pinned as an invariant in the helper JSDoc

.branch/scope.md

The plan says 'preserving order, stable'. Pin this explicitly: channels appear in each bucket in their input order; granularity keys appear in order of first occurrence. Tests should assert ordering.

History · 3 commits

  1. e4f79c8safeincremental0H · 0M · 1L2026-07-09 05:04
  2. 9312968needs attentionincremental2H · 3M · 5L2026-07-09 04:04
  3. 0544fe2needs attentionfull7H · 6M · 6L2026-07-09 03:40current