feat/multi-gran
needs attentionviewing older commit0544fe2 · fullpre-PRreviewed 2026-07-09 03:40 UTC7H · 6M · 6L · 3I- Purpose
- Fix the eGauge worker so a device configured with the same variable at multiple granularities (e.g. solar_generation at 5m and 15m) collects each granularity independently in a single Site SFN run.
- Goal
- Let a single metrics collection run collect multiple granularities per device — one MetricsPayload per (variable × granularity), keeping granularity keyed at the MetricSource row level (no schema change).
- Sub-goals
- SG-1: groupChannelsByGranularity helper + contract doc clarification
- SG-2: eGauge worker — fetch per distinct granularity
- SG-3: Growatt worker — verify + regression-test multi-granularity
- SG-4: Aggregator — prove per-(variable,granularity) channel emission
- SG-5: End-to-end validation via Site SFN (branch preview)
- SG-6: Framework improvements — fold learnings back
- What
- Branch provisioned with detailed scope.md (99 lines) and empty intent.md. No code yet.
- Why
- eGauge devices serve 5m data on the live tier and 15m natively once the fine tier ages out (~12 months). The current worker fetches once at channels[0].granularity and projects all channels from that single CSV — a 15m MetricSource gets 5m-resolution data, violating metrics-pipeline.md P4 (store native granularity only).
- Areas
- .branch/scope.md+99−0.branch/intent.md+0−0
- Blast
- 2 files, +99 lines. Provision-only. When implemented: egauge worker (primary), engine/lib (new helper), growatt (tests), .claude/rules (docs). No schema/auth/API changes.
Findings · 23
correctness5
SC-1 uses '@ 1d' — eGauge rejects 1d with unsupported-granularity
.branch/scope.md
Success Criterion 1 gives 'solar_generation @ 5m and @ 1d' as the target example. fetchHistoricalData only supports 5m and 15m; 1d returns {ok:false, reason:'unsupported-granularity'}. Under the fan-out loop, the 1d group fails and that MetricSource never gets points. SC-1 should use 5m + 15m.
Bug described as 'mislabeling' — actual impact is wrong-cadence data (wrong density)
.branch/scope.md
The scope says 'no more channels[0].granularity mislabeling'. The real bug: a 15m channel's payload contains 5m-resolution data from the 5m CSV fetch. Wrong data density, not just a label issue. Tests must verify data cadence, not just payload granularity field.
fail-only-if-ALL shifts meaning under per-granularity fan-out — undocumented
.branch/scope.md
Under the new design, 'fail only if ALL fail' means 'fail only if ALL granularity groups produce zero payloads'. A 15m group failure is invisible at the invocation level. SG-2 details must explicitly state the cross-group fail semantics and how per-group failures are reported.
1h/1d provisioning gap becomes more reachable — no guard documented
.branch/scope.md
MetricChannelSpec.granularity accepts '1h' | '1d' but eGauge rejects them at runtime. The multi-gran feature makes this more likely to be hit in practice. Document as a known gap, defer provision-time guard to a future issue.
Growatt same-variable-two-granularities routing needs pre-verification
.branch/scope.md
The plan assumes Growatt 'already works' for same-variable at two granularities. Verify growatt/src/engine/historical-data.ts routes on (channel, granularity) before writing SG-3 tests.
security3
No cap on distinct granularities per device — unbounded fan-out
.branch/scope.md
groupChannelsByGranularity issues one fetchHistoricalData call per distinct granularity. No stated upper bound. Recommend capping at ≤4 distinct granularities per device, returning InvocationError if exceeded. Bounds Lambda cost and external API hammering.
S3 write amplification scales with granularity count
.branch/scope.md
More granularities = more S3 PUTs and Tinybird ingest ops per collection run. Bounded by provisioning access control, but not truly 'free' at scale. Acknowledge in cost model.
No new auth surface — confirmed safe
.branch/scope.md
Credential handling is unchanged. No new I/O, secret access, or trust boundary.
conventions3
No `Granularity` type exists — Map key must use MetricChannelSpec['granularity']
.branch/scope.md
The plan names the helper `Map<Granularity, MetricChannelSpec[]>` but no `Granularity` type is exported from @batu/metrics-engine. Either export `type Granularity = MetricChannelSpec['granularity']` from the engine barrel or use the indexed access type directly.
Per-granularity fail is a new failure mode — should be in SG-2 details
.branch/scope.md
The 'fail only if ALL' cross-group semantics need to be documented explicitly in SG-2 to avoid an implementer defaulting to fail-fast on the first granularity group error.
SG-6 framework doc update follows established pattern
.branch/scope.md
Modifying .claude/rules/metrics-pipeline.md and services/metrics/integrations/CLAUDE.md in SG-6 is consistent with the project framework.
tests7
Partial fetch failure not tested: 5m fetch succeeds, 15m fetch fails
.branch/scope.md
The most dangerous scenario — one granularity group fails while others succeed — has no planned test. The invocation would return success with only the successful group's payloads while the other MetricSource silently stops collecting. Need an explicit test for this partial-failure path.
eGauge test plan must assert fetchHistoricalData call count and granularity args
.branch/scope.md
SG-2 plan says 'two-gran ⇒ two fetches' but does not specify spying on fetchHistoricalData to assert EXACTLY TWO calls with distinct granularity arguments. The test could pass even with a single-fetch implementation if both test fixture channels have the same granularity.
describe mode regression not planned — must survive the refactor
.branch/scope.md
The plan preserves describe mode but no test is planned. After the refactor, if groupChannelsByGranularity logic accidentally touches the describe path, the result silently shifts from DescribeColumnsPayload to MetricsPayload[]. Add: mode=describe → single fetch at '5m', result is DescribeColumnsPayload.
SG-1 helper test missing critical edge cases
.branch/scope.md
Test cases unspecified. Required: (1) empty array → empty Map; (2) all-same-granularity → 1-entry Map; (3) order preservation within buckets; (4) behavior for '1h'/'1d' entries — included in Map (causing later fetch failure) or pre-filtered? This contract choice has downstream error consequences.
SG-4 aggregator test must assert each channel's granularity field
.branch/scope.md
The test description does not mention asserting that each channel in params.sources[] carries the CORRECT granularity. Without this, the test passes even if the aggregator collapses both to the same granularity — the same class of bug as the eGauge worker, shifted upstream.
No test for unsupported-granularity path in the fan-out loop
.branch/scope.md
fetchHistoricalData returns unsupported-granularity for 1h/1d. Under the new fan-out, a 1d channel group hits this path. Need an integration test: 5m group succeeds, 1d group hits unsupported-granularity → TranslationFailed → invocation succeeds with only 5m payloads.
SG-5 E2E should verify per-source coverage window advancement
.branch/scope.md
Verify that metric_coverages advanced independently for each MetricSource after the SFN run. If not, subsequent collection triggers will skip one granularity's re-collection.
improvement5
'Verify non-5m cadences' is misleading — 15m already confirmed; critical check is 1h/1d rejection
.branch/scope.md
fetchHistoricalData already supports 15m (in SLOT_SECONDS_BY_GRANULARITY). The important verification is confirming 1h/1d returns unsupported-granularity and the fan-out loop handles it gracefully. Rephrase SG-2's verification step.
DevicePayload top-level granularity scalar becomes misleading for multi-granularity devices
.branch/scope.md
The site-context-aggregator sets a device-level granularity from the first channel. For a 5m+15m device this scalar is meaningless. Acknowledge as a known limitation in SG-6 (params.sources[].granularity is the authoritative field).
Sequential granularity fetches may double Lambda wall-clock time
.branch/scope.md
Two granularities = two sequential fetchHistoricalData calls. If each is ~3-5s, the invocation wall-clock doubles. SG-2 should document whether concurrent fetches (Promise.all over Map entries) are better or sequential is acceptable given Lambda timeout headroom.
SG-3 Growatt test uses '1d' — verify Growatt manifest declares 1d
.branch/scope.md
If the Growatt manifest only declares 5m, a 1d test fixture creates a misleading regression guard. Verify @batu/integration-manifests/src/manifests/growatt.ts before choosing fixture granularities.
Map iteration order should be pinned as an invariant in the helper JSDoc
.branch/scope.md
The plan says 'preserving order, stable'. Pin this explicitly: channels appear in each bucket in their input order; granularity keys appear in order of first occurrence. Tests should assert ordering.