← all branches

fix/tariff-cov

needs attentionviewing older commit
e44f6bb · fullpre-PRreviewed 2026-08-05 19:29 UTC3H · 6M · 9L
The branch
Purpose
Fix silent data-loss bugs in the CFE tariff rate scraper — three bugs each passed green while writing wrong prices or skipping divisions
Goal
Ensure the CFE tariff scraper writes correct prices for all rate types and pricing zones, with loud failures when it cannot
Sub-goals
  • One CFE page per rate type (never fan GDMTH cells across all tariffs)
  • Divisions fan out to zones (emit individual zone rows, not merged labels)
  • Target month verified not assumed (fail loudly on unsupported month/year)
  • Stale job reaping (dead job blocking the monthly cron is reaped before creating new job)
  • Zone skips surface as failures (ZONE_SKIPPED counted in partial_success, not silently dropped)
  • Add tariff-coverage-watchdog CI to catch missing months/zones post-deploy
  • Document scraper invariants in ontology.md
The changes (whole branch)
What
CFE scraper Lambdas (collector, persist, transition) refactored to fix 5 data-loss bugs. New tariff-job domain entity (decisions/queries/shells/errors). SFN and CDK infra updated. New collector and config-integrity tests. New tariff-coverage-watchdog CI workflow.
Why
Three historical bugs all reported success (job.status = completed, divisionsFailed: 0) while writing wrong data or skipping zones entirely — wrong tariff prices are a billing/money bug, not a display bug
Areas
services/utility/tariffs/cfe+424128services/utility/tariffs/cfe/__tests__+3700domains/utility/src/tariff-job+10814infra/cdk/src/stacks/services/utility/tariffs/cfe+246101.github/workflows+1445.claude/rules+950apps/platform/src/api+130scripts/db+1040
Blast
19 files, +1613/-258 across CFE scraper Lambdas, tariff-job domain, SFN/CDK infra, CI watchdog, new collector+config tests, and ontology docs
billing-impact infra-change money-bug-scar no-ci-runs-found
github-ci· No CI runs found for branchcoderabbit· No .coderabbit.yaml present

Findings · 18

correctness4

medium

divisionsFailed counter inflated: ZONE_SKIPPED failures duplicate across rate types

domains/utility/src/tariff-job/tariff-job.decisions.ts:197

decideFinalizeJob sets divisionsFailed = input.failures.length. Each Map branch runs its own persist Lambda for one rate type. aggregatePersistResults flatMaps all branches' failures. A zone ZONE_SKIPPED under all 9 rate types appears 9x in the final array, so divisionsFailed is stored as 9 instead of 1. Status determination unaffected, but the job result misleads operators.

medium

failTariffJobShell alreadyFailed path returns event.id = '' violating ShellResult contract

domains/utility/src/tariff-job/tariff-job.shells.ts:244

Empty string is not a valid outbox event id. No current caller breaks, but a future caller inspecting event.id could treat '' as real. A null event or discriminated return type would be structurally unambiguous.

low

resolveCronYearMonth uses UTC month — Mexico calendar alignment is by accident

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:40

Not a bug with 14:00 UTC schedule, but if rescheduled to very late UTC, getUTCMonth() could return the new month while Mexico is still in the old month.

low

SFN Choice condition order silently enforces 'explicit yearMonth wins' — undocumented contract

infra/cdk/src/stacks/services/utility/tariffs/cfe/stepfunctions.stack.ts:186

Reordering ASL conditions would silently change behavior. Document at the state definition.

security3

medium

TLS certificate verification disabled for CFE scrape HTTP client

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:599

rejectUnauthorized: false disables TLS peer verification. Pre-existing pattern from bill collector, necessitated by CFE cert chain incompatibility. Risk: network-positioned attacker could substitute rate values. Document as accepted risk; consider fingerprint pinning.

low

rateTypes override accepted without allow-list validation

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:200

Arbitrary strings can be written to tariff_jobs.config. Attack surface limited to platform operators.

low

CI watchdog annotation expands step output without scope-limiting

.github/workflows/tariff-coverage-watchdog.yml:278

Low risk with current SQL. If query modified to include sensitive columns, annotation would expose them.

conventions2

medium

finalizeTariffJobShell always emits JOB_COMPLETED for failed/partial_success outcomes

domains/utility/src/tariff-job/tariff-job.shells.ts:218

decideFinalizeJob returns 'failed' or 'partial_success' but shell unconditionally emits JOB_COMPLETED. Consumers routing on event type alone receive JOB_COMPLETED even for a failed job. Consider separate event types per outcome matching the cfe-job sibling pattern.

low

reapStaleActiveJob throws Error instead of returning Result at shell boundary

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:115

Acceptable at Lambda boundary, but thrown errors propagate to the SFN as untyped exceptions.

tests7

high

Bug 5 fix (zone skips surface as failures) has no test

services/utility/tariffs/cfe/__tests__/collector.test.ts:1

The zone-skip-as-failure fix lives in tariff-persist.lambda.ts. No test stubs an unresolvable zone through the persist layer and verifies the skipped entry appears in failures. The fix that un-silenced VDM data loss is untested.

high

resolveTargetYearMonth (determines which month gets scraped) is entirely untested

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:59

Pure function. Wrong month = wrong prices under the right key. Both branches (explicit yearMonth wins over monthsBack; malformed yearMonth throws) are untested. Export and test: malformed yearMonth, monthsBack=0/1, explicit overriding monthsBack.

high

Stale job reaping (bug 4 fix) has no test

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:109

reapStaleActiveJob fixes a dead job blocking the monthly cron. Neither path (young job -> throw; old job -> failTariffJobShell) is tested. A regression here would silently lose an entire month's data.

medium

aggregatePersistResults has no unit test

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:76

Pure function that feeds decideFinalizeJob. A miscalculation would cause wrong job status. Trivially testable by exporting and passing mock inputs.

medium

decideFinalizeJob not tested for divisionsPersisted=0 with empty failures

domains/utility/src/tariff-job/__tests__/tariff-job.decisions.test.ts:136

All none-persisted tests pass non-empty failures. The divisionsPersisted === 0 -> failed branch fires regardless of failures.length — untested edge case.

low

failTariffJobShell shell-level idempotency not integration-tested

domains/utility/src/tariff-job/__tests__/tariff-job.decisions.test.ts:175

Decision-level idempotency tested, but not the shell-level guarantee (one DB update, one outbox event on double-call).

low

Year-unavailable branch in month verification not tested

services/utility/tariffs/cfe/__tests__/collector.test.ts:238

Guard rejecting requests for unavailable years is untested. StubClient makes it trivially testable.

improvement2

low

Stale-job age check roundtrips ms -> minutes -> ms losing precision

services/utility/tariffs/cfe/src/handlers/tariff-transition.lambda.ts:114

Use const ageMs = Date.now() - active.queuedAt.getTime(); derive ageMinutes for error message only.

low

Duplicated all-zones fan-out loop for UNSUPPORTED_RATE_TYPE and RATE_PAGE_FAILED

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:520

Extract failAllZones(targets, rateType, errorCode, message) to remove identical nested for-loops at both error paths.

History · 14 commits

  1. 1ed035eneeds attentionincremental0H · 2M · 5L2026-08-11 02:03
  2. fcbe80dblockedfull6H · 12M · 14L2026-08-10 22:02
  3. 92353bdblockedincremental1H · 9M · 7L2026-08-10 19:32
  4. 5f2213eneeds attentionincremental1H · 6M · 10L2026-08-09 05:35
  5. c30da44needs attentionincremental1H · 4M · 2L2026-08-09 04:54
  6. 5940f56needs attentionincremental0H · 3M · 7L2026-08-07 19:15
  7. f222512needs attentionfull2H · 9M · 15L2026-08-07 18:58
  8. eec3b04needs attentionincremental2H · 1M · 4L2026-08-07 18:27
  9. 0b43396needs attentionincremental0H · 2M · 6L2026-08-07 01:39
  10. 3eb9789needs attentionincremental0H · 3M · 5L2026-08-06 18:30
  11. 87fc06aneeds attentionincremental0H · 4M · 9L2026-08-06 01:21
  12. 76bcc76needs attentionincremental0H · 1M · 5L2026-08-06 00:51
  13. 8948608needs attentionincremental0H · 3M · 4L2026-08-05 23:49
  14. e44f6bbneeds attentionfull3H · 6M · 9L2026-08-05 19:29current