feat/shelly
blocked4350d25 · fullPR #273reviewed 2026-07-08 02:36 UTC2H · 11M · 7L- Purpose
- Port the Shelly Cloud integration (216 shellypro3em grid meters) into the v2 metrics engine at native 1h granularity
- Goal
- Metrics worker Lambda + CDK stack delivering 1h native collection with 30/30 local-day kWh parity vs legacy DDB
- Sub-goals
- SG-1: API probe + session strategy decision + worker + 50 unit tests
- SG-2: CDK stack + main.ts + coordinator grants; synth clean
- SG-3: Preview data plane via victron recipe (3 sites seeded)
- SG-4: Live Site-SFN validation + Tinybird + 30/30 legacy parity (max Δ 0.085%)
- SG-N: 9 framework learnings → scope.md + CLAUDE.md
- What
- Adds Shelly Cloud worker (per-invocation login + warm-container memo, 48h chunking, device-tz self-heal, grain verification); CDK Lambda + IAM stack; shelly-cloud manifest + 11 other brand manifests (foundation seam); seed catalog expanded with 12 Makes + Integration rows + 3 MetricType rows; seed-integration-arns REGISTRY + --only scoping; D2/D3 migration scripts
- Why
- Legacy prod fleet (115 Shelly devices) still collecting via batu-monorepo. Port to v2 engine enables unified monitoring, Tinybird storage, and future decommission of the legacy stack
- Areas
- services/metrics+1854−6scripts/metrics+1719−0packages/integration-manifests+1068−2infra/cdk+360−1packages/database+257−14.branch/scope.md+67−167domains/core+16−1domains/metrics+5−4
- Blast
- 8 areas touched, +4562/−198 net. Core risk surface: new Lambda worker (services/metrics) + CDK stack (infra/cdk). Seed expansions affect catalog + ARN routing but are idempotent upserts. Migration scripts are dry-run by default.
Findings · 22
correctness1
shelly-cloud Integration seed has wrong credentialFields (api_key/access_token → must be user/password)
packages/database/src/seed-metrics-catalog.ts:165
Seed declares [api_key, access_token] but the manifest, worker loadCredentialFromArn, domain types, and probe evidence all require [user, password]. The seed comment explicitly says 'MIRRORS the manifest — keep the two in lockstep'. Impact: UI wizard prompts operators for wrong field names; secrets stored with wrong keys; loadCredentialFromArn returns null → MissingSecret on every collection for newly provisioned Shelly orgs. Fix: replace the two credentialFields entries with [{ name: 'user', label: 'Shelly Cloud account email', secret: false }, { name: 'password', label: 'Shelly Cloud password', secret: true }].
security4
JWT user_api_url drives all stats fetches with only https:// prefix check — no hostname pinning
services/metrics/integrations/shelly/src/engine/shelly-client.ts:141
The shard URL extracted from the JWT user_api_url claim is only validated for startsWith('https://') before being used to construct all statistics fetch URLs. A compromised Shelly Cloud login endpoint could return a JWT with an arbitrary https URL and redirect the Lambda's fetches to any HTTPS endpoint. Mitigation: add new URL(apiBaseUrl).hostname.endsWith('.shelly.cloud') validation alongside the https:// check.
Vendor-controlled timezone string passed unvalidated to Intl.DateTimeFormat
services/metrics/integrations/shelly/src/translation/points-to-batu.ts:92
A malformed timezone from the Shelly stats response throws in tzOffsetMs; top-level catch converts to UpstreamUnavailable 502. No credential leak; effective collection DoS for that device. Mitigate with a try/catch probe on Intl.DateTimeFormat instantiation.
SHA1 password hashing — weak crypto, but Shelly Cloud API protocol requirement
services/metrics/integrations/shelly/src/engine/shelly-client.ts:97
SHA1 is cryptographically broken but this is what the Shelly Cloud API mandates (probe-verified). Accept as-is; document constraint explicitly in the comment.
Module-scope session memo accumulates tokens without size eviction bound
services/metrics/integrations/shelly/src/engine/shelly-client.ts:153
One entry per org-ARN accumulates until container recycles. At 115 Shelly orgs this is not a practical concern. Inherent to in-process session caching.
conventions3
device-not-found mapped to TranslationFailed (statusCode 500) → JSend error instead of fail
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:355
A decommissioned device is a known provisioning fact (4xx-class), not a server error. TranslationFailed with statusCode 500 classifies it as error in the JSend envelope, which may trigger unnecessary SFN retries and mislead monitoring. The InvocationError contract ties TranslationFailed to statusCode:500, so a fix requires either a new _tag (AssetDecommissioned) or a contract change.
Missing mandatory CDK tags batu:env, batu:owner, batu:costCenter on Shelly stack
infra/cdk/src/stacks/services/metrics/integrations/shelly/lambda.stack.ts:49
infrastructure.md lists batu:env, batu:owner, batu:costCenter as mandatory. The Shelly stack applies only domain/subdomain/provider/service/dataClass. Same gap exists in growatt/egauge peers — systemic issue, but each new stack is an opportunity to fix it.
mapFetchError switch lacks exhaustiveness never-guard
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:345
Adding a new ShellyFetchFailure.reason will cause a TypeScript error at the unhandled case, but no explicit default: never assertion makes this a guaranteed compile-time check.
tests9
No direct tests for shelly-client.ts login, JWT parsing, and session memo
services/metrics/integrations/shelly/src/engine/shelly-client.ts
login(), decodeJwtClaims(), and getSession() memo (expiry-margin hit/miss, forceRefresh, memo eviction) have zero independent test coverage. The handler tests mock getSession/fetchStatistics entirely. The wrong-shard → silent-200-all-missing failure mode (documented as a critical operational hazard) is not pinned by any test.
Initial getSession failure (login fails before any fetch) not covered by handler tests
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:272
Existing auth tests only exercise invalid-token mid-collection (fetchStatistics fails). The case where the initial getSession call returns ok:false (e.g. auth-failed at login, rate-limited at login) is not tested. A broken credential that prevents initial login would produce an error of unpredictable tag.
tz-correction across multiple chunks not verified
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:280
The single-chunk tz test confirms the corrective refetch fires. It does not verify chunks 2+ use the device tz in their wall-clock strings. Real fleet has Europe/London devices in Mexican orgs. A bug where requestTz were re-initialized per chunk would corrupt UTC anchoring on chunks 2+.
Mixed-granularity invocation (1h + 1d channels) not tested
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:164
Handler explicitly supports multiple granularities in one call via the granularities Set. No test covers this path. A bug in the fetches Map iteration would silently return data from the wrong granularity's entries.
Mid-window timezone change defensive guard (hard failure) not tested
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:322
The guard at line 322 that hard-fails when a device reports a different timezone on chunk N vs chunk 1 is untested. A refactor removing it would be invisible.
All-channels-unsupported handler path (payloads.length === 0 via pointsToBatu failure) not tested
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:222
The handler path where ALL channels fail with unsupported-channel and payloads.length === 0 → failure is not exercised. Distinct from the all-missing-device path tested at line 262.
mapFetchError coverage missing for http-error, vendor-rejected, unparseable → UpstreamUnavailable
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:273
The tag test covers rate-limited, timeout, unreachable, auth-failed. Three remaining cases (http-error, vendor-rejected, unparseable) map to UpstreamUnavailable and are not pinned.
Auth retry on non-first chunk (token expires mid-collection) not tested
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:302
Existing test exercises invalid-token on the first and only chunk. A realistic scenario where chunks 1–2 succeed but chunk 3 returns invalid-token (JWT expired mid-collection) is not covered.
Session memo warm-container hit (no re-login on second invocation) not tested
services/metrics/integrations/shelly/src/engine/shelly-client.ts:165
clearSessionMemo() export exists as a test hook but is unused in tests. The memo saving ~250ms per warm invocation is not pinned.
improvement5
alignWindowStart called twice with identical args per channel — second call is redundant
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:211
alignedFrom is already computed at line 177 for the current granularity and passed into fetchGranularity. The identical call at line 211 (pointsToBatu fromUtc: arg) should reference alignedFrom directly.
authRetried/tzCorrected flags are function-scope (per-granularity) but comments say 'per-chunk'
services/metrics/integrations/shelly/src/handlers/metrics.lambda.ts:280
Comment at line 301: 'One forced re-login on an expired/revoked token, then retry the chunk' implies per-chunk scope. Code is per-granularity (across all chunks). If a token expires between chunks, the second invalid-token returns UpstreamAuthFailed instead of refreshing again. Add a clarifying comment or restructure.
Content-Type: application/json header on GET statistics request
services/metrics/integrations/shelly/src/engine/shelly-client.ts:223
GET requests have no body; Content-Type is semantically meaningless and some strict proxies reject it. Only Authorization header is needed.
Tz helpers copied verbatim from victron — should be extracted to _shared/tz-helpers.ts
services/metrics/integrations/shelly/src/translation/points-to-batu.ts:86
12 more ports are planned. Each that uses wall-clock vendor APIs will copy these helpers and diverge. Extract to services/metrics/integrations/_shared/tz-helpers.ts (or @batu/tz-helpers).
INTERVALS array manually mirrors ShellyInterval union without type-level enforcement
services/metrics/integrations/shelly/src/engine/shelly-client.ts:186
Use `(['minute', 'hour', 'day'] as const satisfies ReadonlyArray<ShellyInterval>)` to make TypeScript enforce sync between the union and the runtime array.