fix/tariff-cov
needs attentionviewing older commit87fc06a · incrementalpre-PRreviewed 2026-08-06 01:21 UTC0H · 4M · 9L · 9I- Purpose
- Fix silent data-loss bugs in the CFE tariff rate collector Lambda that caused whole tariff×zone rows to go missing for a month
- Goal
- Make the tariff collector resilient to transient CFE HTTP failures and stop a single bad division response from cascading to kill every subsequent division
- Sub-goals
- SG-1: Capture baseResp snapshot after month selection so each division re-drives the cascade from a known-good page
- SG-2: Add per-division retry loop (3 attempts, linear backoff) to absorb CFE's intermittent no-table responses
- SG-3: Add FlakyStubClient test double and regression pins that prove containment vs cascade and retry vs permanent failure
- SG-4: Fix TOU schedule seeding, Valle de México zone pairing, month navigation, and other collector correctness bugs (earlier commits)
- What
- Two files changed in this commit: tariff-collector.lambda.ts gains the baseResp snapshot pattern and a 3-attempt retry loop with 250ms linear backoff; collector.test.ts gains FlakyStubClient (an ASP.NET-aware stub that enforces the __VIEWSTATE cascade rule) and two new behavioral tests.
- Why
- CFE intermittently serves rate pages without their rate table under load — 2 of 6 validation runs lost divisions this way. The old code threaded one mutable `resp` through the division loop, so a bad response from division N became the base for division N+1's postback, which lacked a viewstate, causing another failure — cascading to the end. The fix contains failures per-division (baseResp isolation) and recovers transient ones (retry).
- Areas
- services/utility/tariffs/cfe/src/handlers+427−127services/utility/tariffs/cfe/__tests__+507−0infra/cdk/src/stacks/services/utility/tariffs/cfe+246−99domains/utility/src/tariff-job+108−23packages/database/src/seed-tou-schedules.ts+75−35.github/workflows+144−5scripts/db+104−0apps/platform/src/api+13−0
- Blast
- 20 files, +1936/−312 across tariff collector service, CDK infra, domain tariff-job, seed-tou-schedules, and CI workflows. No customer-facing UI changes; blast radius is internal pipeline reliability.
Findings · 19
correctness3
formState delete list must stay in sync with cascade steps or retries leak state
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:477
scrapeDivision deletes NAMES.state/municipality/division from the shared formState before each attempt, then re-sets them via post(). This is correct today — the deletes at the top of every call clear any stale values from a prior attempt. But the invariant is implicit: if a future maintainer adds a cascade step (e.g. a 'region' dropdown) and adds a post() call without also adding the corresponding delete at the top, stale values from a failed attempt pollute the next retry. The pattern is correct but fragile — the delete list and the post list are a paired contract with no enforcement.
lastError is undefined if DIVISION_SCRAPE_ATTEMPTS is ever reduced to 0
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:501
lastError is declared without initializer. If DIVISION_SCRAPE_ATTEMPTS were 0 the loop body never runs, scraped stays null, and the rethrow becomes `new Error('undefined')`. Currently impossible since it's a constant set to 3, but the post-loop check relies on an implicit loop-entry invariant the type system cannot enforce. Initialising as `let lastError: unknown = new Error('no attempts made')` makes the contract self-documenting.
formState accumulates year and month keys permanently — intentional
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
Year and month keys are set once before the division loop and never deleted. Every division postback includes them. This is correct — ASP.NET requires all form fields and the month must stay selected across the cascade. Confirmed intentional.
security2
Total sleep budget is implicit — Lambda timeout could be silently consumed by retry backoff
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
sleep(RETRY_BACKOFF_MS * attempt) with RETRY_BACKOFF_MS=250 and DIVISION_SCRAPE_ATTEMPTS=3 adds at most 250+500=750ms per division. With 15+ divisions this is ~11s of additional sleep in the worst case, on top of real request latency. At current constants this is well within any reasonable Lambda timeout. Worth noting if either constant grows.
Error wrapping in lastError rethrow discards non-Error throwables
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
String(lastError) on a non-Error thrown value is safe here — the only source of thrown values is the HTML parser or HTTP client, neither of which carries credentials or sensitive data.
conventions2
Multi-paragraph block comments are high-volume for a single Lambda handler
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
CLAUDE.md prohibits multi-paragraph docstrings. The file now has ~8 multi-paragraph /** */ blocks. Each documents a non-obvious constraint or prior failure mode — which is exactly what the rule's WHY exception is for. The content is load-bearing (bug archaeology). No change strictly required, but the density is notable.
sleep in Lambda handler follows established codebase precedent — no violation
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:179
10+ identical patterns in other Lambda handlers and API clients (fronius, solark, ABB). The CLAUDE.md intent is to prevent sleep in domain decision functions. This is an imperative I/O shell — the usage is correct.
tests8
Retry boundary (DIVISION_SCRAPE_ATTEMPTS=3) is untested — only 1-fail-then-success path covered
services/utility/tariffs/cfe/__tests__/collector.test.ts:295
failTimes=1 proves the retry path works, but two boundaries are uncovered: (a) exactly 2 failures then success — the last-attempt edge — and (b) all 3 attempts exhausted — which should record the division as a failure. Without (b), reducing DIVISION_SCRAPE_ATTEMPTS from 3 to 1 would leave the transient test green while breaking the retry budget guarantee.
No test for first or last division failure — only a middle-of-list division (BAJIO, index 2)
services/utility/tariffs/cfe/__tests__/collector.test.ts:304
The containment test uses BAJIO (3rd of 15 configs). The last-division case (Valle de México, a multi-zone config) is structurally different: permanent failure there should produce 3 zone entries in failures and divisionsCollected=14 — never tested. A first-division failure would also make the cascade-infection scenario impossible to demonstrate (nothing precedes it), but the counter handling is worth pinning.
Real sleep (up to 750ms) in tests — no fake timers, RETRY_BACKOFF_MS not injectable
services/utility/tariffs/cfe/__tests__/collector.test.ts:295
RETRY_BACKOFF_MS is a module-private constant. The flaky tests incur real wall-clock sleeps. Tests stay within the timeout but are noticeably slower. Exporting RETRY_BACKOFF_MS or accepting it as injected config would let tests pass 0 and run instantly — also makes the retry constant auditable from the test.
FlakyStubClient attempt counter is instance-global — coupling is undocumented
services/utility/tariffs/cfe/__tests__/collector.test.ts:154
private attempts counts only the target division's postbacks, so non-target divisions don't consume the budget. This is correct but implicit. A per-divisionId Map would make the counter scope self-documenting and robust if a test ever drives multiple rate types through the same client instance.
Test 1 does not assert attempt count — retry success is inferred, not pinned
services/utility/tariffs/cfe/__tests__/collector.test.ts:293
The test proves BAJIO is collected and failures is empty — sufficient to establish that a retry occurred. It does not assert how many attempts were made (e.g. via client.urls length). Low severity because the containment test provides complementary structural proof.
FlakyStubClient __VIEWSTATE check correctly models the infection mechanism, not just luck
services/utility/tariffs/cfe/__tests__/collector.test.ts:176
A broken page has no __VIEWSTATE. Any postback built from it is rejected immediately (another broken page). The fix (baseResp baseline) ensures all division attempts start from a page that has a viewstate. The stub enforces this exactly — the test proves structural containment, not coincidental success.
divisionsDiscovered is not asserted in the two new tests
services/utility/tariffs/cfe/__tests__/collector.test.ts:301
Both tests assert divisionsCollected. Asserting divisionsDiscovered alongside (expected 17 in both) would pin the accounting separately and catch a regression where failures are silently suppressed into divisionsCollected. The existing 'counts zones, not division configs' test at line 261 checks both — establishing the pattern.
data.rates[0]! non-null assertion is safe for test inputs but would throw silently if rates were empty
services/utility/tariffs/cfe/__tests__/collector.test.ts:300
Both new tests have at least one rate type configured and at least one successful division, so rates[0] always exists. Adding expect(data.rates).toHaveLength(1) before the chain would produce a clearer failure message if the invariant ever broke.
improvement4
scrapeDivision closes over shared mutable formState without documenting the contract
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:474
The function signature shows only `cfg` in, but it also reads/writes the outer formState and captures baseResp, post, and NAMES. This hidden coupling is correct today (serial loop, deletes at top of each call) but is invisible at the call site. A one-line contract comment at the function boundary — naming that it resets formState to year+month baseline then replays state→municipality→division off baseResp — would make the invariant explicit without restructuring.
Asymmetric delete counts before state vs. municipality POST are undocumented
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:477
Before the state POST, three keys are deleted (state, municipality, division). Before the municipality POST, only division is deleted. This correctly mirrors ASP.NET cascade semantics (selecting a new state invalidates both downstream keys; selecting a new municipality only invalidates division). The asymmetry is silent — a brief inline comment explaining the cascade rule would prevent a symmetry 'fix' from introducing a bug.
Inline retry loop could be a small named helper
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:500
The 12-line attempt/lastError/sleep pattern is used once. A module-local `withRetry<T>(fn, attempts, backoffMs)` would collapse it to one call, eliminate the nested try/catch, and make the retry policy readable at the call site. Not urgent at this size; useful if a second caller is added (e.g. the year/month postbacks also fail under load).
scrapeDivision altitude: inner closure is correct — parameters vs. closure is a stylistic tradeoff
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:474
Making scrapeDivision a module-level function would require passing baseResp, formState, post, and NAMES as parameters — noisier than the closure at this size. The existing JSDoc at lines 470-473 already explains the isolation rationale. No change needed.
History · 14 commits
- 1ed035eneeds attentionincremental0H · 2M · 5L2026-08-11 02:03
- fcbe80dblockedfull6H · 12M · 14L2026-08-10 22:02
- 92353bdblockedincremental1H · 9M · 7L2026-08-10 19:32
- 5f2213eneeds attentionincremental1H · 6M · 10L2026-08-09 05:35
- c30da44needs attentionincremental1H · 4M · 2L2026-08-09 04:54
- 5940f56needs attentionincremental0H · 3M · 7L2026-08-07 19:15
- f222512needs attentionfull2H · 9M · 15L2026-08-07 18:58
- eec3b04needs attentionincremental2H · 1M · 4L2026-08-07 18:27
- 0b43396needs attentionincremental0H · 2M · 6L2026-08-07 01:39
- 3eb9789needs attentionincremental0H · 3M · 5L2026-08-06 18:30
- 87fc06aneeds attentionincremental0H · 4M · 9L2026-08-06 01:21current
- 76bcc76needs attentionincremental0H · 1M · 5L2026-08-06 00:51
- 8948608needs attentionincremental0H · 3M · 4L2026-08-05 23:49
- e44f6bbneeds attentionfull3H · 6M · 9L2026-08-05 19:29