docs/metrics-ret
needs attentionf060efc · fullPR #318reviewed 2026-07-16 18:43 UTC4H · 6M · 5L · 2I- Purpose
- Record empirically verified per-vendor retention boundaries and investigation runbooks from the Niko (oid=33) device migration sweep, so the next on-call engineer does not have to rediscover the same failure modes.
- Goal
- Add a per-integration granularity-retention matrix (fact-base) + four read-only investigation runbooks (hoymiles / huawei / shelly / solis) covering the full dead-plant diagnostic path from CloudWatch logs → portal probe → classification → deliverable.
- Sub-goals
- SG-1: metrics-granularity-retention.md — retention matrix with empirically verified boundaries (bisected 2026-07-16 on prod)
- SG-2: site-metrics-seam.md update — cross-reference to matrix, backfill choreography subsection, inversion-hazard fix (#311) noted as order-independent
- SG-3: Four investigate-* skills — hoymiles, huawei, shelly, solis — read-only runbooks with decision trees
- What
- 1 new .claude/rules file (118 lines), 1 modified .claude/rules file (+23/-21 lines), 4 new .claude/skills runbooks (238+236+213+213 lines). No application code, no infra, no migrations.
- Why
- The fine retention tier varies 15 days to plant lifetime by vendor — the team was about to settle for daily history on vendors that serve 5-minute data to install. The investigation runbooks encode 10+ trap patterns that produced false boundaries during the 2026-07-16 probes.
- Areas
- .claude/rules+141−21.claude/skills+900−0
- Blast
- 6 files, +1041/-21. Docs only — no runtime impact.
Findings · 18
correctness7
_try used before definition — NameError if copy-pasted
.claude/skills/investigate-hoymiles/SKILL.md:77
Snippet A (the heredoc block) calls _try(m[i:]) inside the for-loop but defines def _try(s) after it. Python raises NameError at runtime. Snippet B (lines 86–106) is the correct working replacement. Remove Snippet A entirely or add a clear 'do not run this' callout.
Month probe window exceeds hoymiles 2-year per-request span cap
.claude/skills/investigate-hoymiles/SKILL.md:163
The mon probe uses start_date='2024-01-01' and end_date='2026-07-15' — a ~926-day span. The retention matrix explicitly documents a 2-year request-span cap for the 1d endpoint ('Requested date range exceeds two years'). If this applies to monthly calls too, the probe returns an error instead of data, quietly sabotaging the GONE-vs-STALE classification. Cap the window to ≤24 months (e.g. start_date='2024-07-01').
Stray closing triple-backtick at end of site-metrics-seam.md
.claude/rules/site-metrics-seam.md:161
Line 161 is a standalone ``` with no matching opening fence in the 'Rates are reference data' section. In GitHub's renderer this opens an unterminated code block that swallows whatever follows, making the Rates section unreadable. Remove the stray fence.
Python snippets use literal string 'SCRATCH/...' — FileNotFoundError at runtime
.claude/skills/investigate-shelly/SKILL.md:74
Lines 74, 123, and 147 pass 'SCRATCH/shelly-logs-raw.json' and 'SCRATCH/.shelly_session.json' as literal Python strings. The shell variable $SCRATCH is never exported into the Python interpreter, so open() looks for a directory literally named 'SCRATCH' in cwd and raises FileNotFoundError. Fix: use f"{os.environ['SCRATCH']}/..." (import os), or pass SCRATCH via sys.argv as the hoymiles skill does.
Shelly 1m retention stated as '~120–150 d' — conflicts with matrix's verified 129 d
.claude/skills/investigate-shelly/SKILL.md:181
The error taxonomy gives a loose '~120–150 d' range for Shelly 1m retention; the retention matrix records a bisected empirical boundary of 'today − 129d ≈ 130d ✅'. The matrix's verified number supersedes the estimate. Update to '~129–130 d (bisected; see metrics-granularity-retention.md)' for consistency.
Huawei fine-grain bracket notation '(12, 25]' implies >12 months; empirical data shows present at 365 d (exactly 12 mo)
.claude/rules/metrics-granularity-retention.md:26
Mathematical open-lower-bound notation (12, 25] means strictly greater than 12 months. The empirical check found data present at 365 days (~12 months) — so the lower bound should be closed: [12, 25] or accompanied by a prose note 'present at 12mo (365d), absent at 25mo (761d)'.
'VRM 31-day' span cap cited but Victron has no API token — source is theoretical
.claude/rules/site-metrics-seam.md:75
Line 75 lists 'VRM 31-day' as an active operational span cap alongside ABB/hoymiles. The retention matrix marks Victron as 'credential is portal-only (no VRM API token) — unverifiable'. The 31-day cap is from VRM API docs, not empirical. Qualify: 'VRM 31-day (theoretical — no API token yet)' or remove until a token is available.
security3
Hoymiles api_key written to disk with no cleanup step
.claude/skills/investigate-hoymiles/SKILL.md:141
The runbook writes the raw Secrets Manager payload (containing the live api_key) to $SCRATCH/.sec.json but has no rm step afterward. On a shared or persistent directory (/tmp, a dev home, CI workspace) the secret persists indefinitely. Add an explicit rm -f "$SCRATCH/.sec.json" at step 5 (Deliverable), matching the Shelly skill's cleanup pattern.
Cleanup rm uses bare 'SCRATCH' literal instead of shell variable $SCRATCH
.claude/skills/investigate-shelly/SKILL.md:212
rm -f SCRATCH/.shelly_session.json treats SCRATCH as a directory name, not the shell variable. The bearer-token session file is therefore never deleted. Fix: rm -f "$SCRATCH/.shelly_session.json" with dollar sign and quotes.
Shelly session file written with default permissions (potentially 0644)
.claude/skills/investigate-shelly/SKILL.md:123
open('SCRATCH/.shelly_session.json','w') uses the process umask. On systems with umask 022, the file is mode 0644 — readable by all local users. After the write, add os.chmod(path, 0o600) or use os.open with O_CREAT|O_WRONLY|O_TRUNC and mode 0o600.
conventions3
investigate-shelly missing ## 5. Deliverable and ## Gotchas recap sections
.claude/skills/investigate-shelly/SKILL.md:185
All three sibling skills (hoymiles, huawei, solis) have a numbered '## 5. Deliverable' describing the expected output (category breakdown with counts, per-device table, evidence, bottom line, recommendations) and a '## Gotchas recap' for quick reference. investigate-shelly skips both, going straight from Error taxonomy to Cleanup. Add a Deliverable section (write investigation-shelly.md to scratch with the same structure) and a Gotchas recap (wrong shard = silent all-missing; narrow-grain sweep ≠ dead; GHOST = present but zero history).
Four new investigate-* skills not registered in CLAUDE.md skills table
CLAUDE.md:60
The skills index table lists all operator-facing skills and their triggers. investigate-hoymiles, investigate-huawei, investigate-shelly, and investigate-solis are absent. Without a table entry, an operator asking 'why is Hoymiles returning no data?' has no hint the skill exists. Add one line per skill with trigger description.
metrics-granularity-retention.md path scope may miss billing/metrics domain code
.claude/rules/metrics-granularity-retention.md:1
The frontmatter scopes auto-loading to services/metrics/integrations/**, scripts/metrics/**, packages/integration-manifests/**. But site-metrics-seam.md (always-loaded) cross-links to this file as relevant to billing/savings consumers in domains/utility/** and domains/metrics/**. Consider adding those paths to the frontmatter so the retention rules are visible when editing billing code.
improvement5
SWEEP date hardcoded to 2026-07-14 — investigator must update manually, not told to
.claude/skills/investigate-hoymiles/SKILL.md:158
SWEEP="2026-07-14" is a specific authoring-session date. An investigator copy-pasting the snippet later will probe the wrong day. This also contradicts retention-matrix trap 11: 'never hardcode a date'. Replace with SWEEP=$(date -d 'yesterday' +%F) and add a comment: '# change to the day you are investigating'.
SAMPLE variable used in probe loop but never defined
.claude/skills/investigate-hoymiles/SKILL.md:159
for plant in SAMPLE: references a variable that appears nowhere else in the skill. An investigator must guess that SAMPLE comes from the dead list assembled in step 2. Add two lines before the loop: SAMPLE = [pid for pid, _ in dead] # or a subset; and note that ~15-20 plants at 4 calls × 300ms ≈ 20-25 min.
No investigation runbook for growatt despite documented 95-day gap
.claude/rules/metrics-granularity-retention.md:47
The matrix documents growatt's unrecoverable >95d history with two endpoint gotchas (year key vs date; autoEnergy summation). No investigate-growatt skill exists and the matrix doesn't note this absence. Add a note: 'No investigation runbook — see gotchas above.' so an operator hitting dead growatt plants has clear instructions.
investigate-shelly worker code paths omit /src/ segment
.claude/skills/investigate-shelly/SKILL.md:44
The inputs table lists engine/shelly-client.ts, handlers/metrics.lambda.ts, translation/points-to-batu.ts — but the actual layout is services/metrics/integrations/shelly/src/engine/..., services/metrics/integrations/shelly/src/handlers/..., etc. All other integrations use the same src/ layout. A developer following the path literally won't find the files.
CloudWatch Insights sleep 6 is fragile — no retry shown
.claude/skills/investigate-huawei/SKILL.md:77
A fixed 6-second wait before get-query-results can return status=Running for large log groups (large fleets may take 30+ s). Add a polling note: 'If status is Running, re-run get-query-results after 10 s; allow up to 60 s total.'