fix/tariff-cov
needs attentionviewing older commit8948608 · incrementalpre-PRreviewed 2026-08-05 23:49 UTC0H · 3M · 4L · 4I- Purpose
- Fix silent data-loss in the CFE tariff rate scraper: Valle de México (one dropdown selection covering 3 zones) was producing a single flat cell list, so cfeRateRowToComponents kept the last table's values and wrote Sur's prices to Norte and Centro. This branch adds per-table scraping, a table-count guard, and comprehensive test coverage.
- Goal
- Eliminate the Valle de México per-zone price data-loss bug and harden the collector against future shape mismatches with loud failures instead of silent wrong data.
- Sub-goals
- SG-1: Rename extractRateCells→extractRateCellTables returning ScrapedRateCell[][] (one array per table)
- SG-2: Add table-count === zones-count guard with a loud error
- SG-3: Wire componentsByZone[i] to zones[i] (correct pairing)
- SG-4: Add tariff-coverage-watchdog CI workflow to detect future gaps
- SG-5: Add rate-type isolation tests and config-integrity tests
- SG-6: Fix month-reach bug in the rate engine (reach any month, fail loudly)
- SG-7: Plug silent data-loss in the pipeline (stop overwriting with nulls)
- What
- tariff-collector.lambda.ts: extractRateCells refactored to extractRateCellTables (returns per-table arrays); table-count mismatch guard added; zones.forEach paired with componentsByZone[i]. collector.test.ts: StubOptions.tables added; rateRows made a function of t for distinct values per table; two Valle tests upgraded from touPages to vallePages (3-table stub); two new tests added pinning per-zone prices and mismatch failures.
- Why
- The last commit (89486083) closes the pairing gap: prior commits fixed the month-reach and pipeline data-loss bugs; this one ensures each of Valle de México's 3 zones receives the rates from its own table, not a shared last-table override.
- Areas
- services/utility/tariffs/cfe+784−127infra/cdk/src/stacks/services/utility/tariffs/cfe+246−101domains/utility/src/tariff-job+108−14.github/workflows+144−5scripts/db+104−0apps/platform/src/api+13−0.claude/rules+95−0
- Blast
- 19 files, +1711/-277 lines across 7 areas; all changes are within the CFE tariff scraping service, utility domain job logic, CDK Lambda/SFN stacks, and CI. No shared domain types or API contracts changed.
Findings · 11
security4
Unbounded table count from CFE HTML response
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
extractRateCellTables iterates all matching tables with no upper bound before the zones.length guard fires. A malformed page could accumulate many tables in memory before the error is thrown. A simple early-exit (break-equivalent once tables.length > max_expected) would bound memory usage.
Unbounded cell count per scraped table row
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
No per-table row count cap on scraped cells. Unexpected page growth could produce unexpectedly long strings passed to cfeRateRowToComponents. Low risk since CFE controls the source, but a max-row guard is cheap defense-in-depth.
Scraped values stored in JSONB without explicit length cap
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
parseRateValue strips non-numeric characters but does not assert a length bound before persisting. In practice a valid CFE rate value is ≤10 chars after cleaning. Not exploitable, noted for defense-in-depth.
Error message zone names sourced from static config — confirmed safe
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
cfg.divisionGroup and zones.join() in the error template string originate from bundled divisions.json (not user input). No injection risk in current form; note this if zones ever sourced from external input.
tests5
Missing NORTE vs CENTRO price-distinctness assertion
services/utility/tariffs/cfe/__tests__/collector.test.ts:217
The 'gives each contained zone ITS OWN table's prices' test checks NORTE≠SUR and CENTRO≠SUR but never checks NORTE≠CENTRO. A bug that assigns table[0]'s prices to both NORTE and CENTRO (while giving SUR table[2]) would produce 3 distinct Set values and pass all current assertions. Add: expect(byZone.get('VALLE DE MEXICO NORTE')).not.toEqual(byZone.get('VALLE DE MEXICO CENTRO')).
Non-null assertion on data.rates[0]! may hide regression
services/utility/tariffs/cfe/__tests__/collector.test.ts:221
In 'gives each zone its own prices', data.rates[0]! throws a TypeError (not a test failure) if runCollection emits no rate entries. A guard expect(data.rates).toHaveLength(1) before the .map() would produce a clear, actionable failure instead.
Mismatch test division assertion is vacuously correct on undefined
services/utility/tariffs/cfe/__tests__/collector.test.ts:242
expect(data.rates[0]?.divisions ?? []).toHaveLength(0) passes both when rates is empty AND when rates[0] exists with an empty divisions array. Verify the production path — if a page-level error fires, rates may be completely empty, and adding expect(data.rates).toHaveLength(1) would pin the expected shape.
Table-to-zone index mapping correct but not explicitly pinned
services/utility/tariffs/cfe/__tests__/collector.test.ts
The test proves three zones have distinct prices but not which zone received which table's prices. A pairing inversion (NORTE ↔ CENTRO swap) passes all current assertions. Low risk given the index comes directly from divisions.json order, but a direct component equality check would fully close the regression surface.
Mismatch failure test does not assert errorCode
services/utility/tariffs/cfe/__tests__/collector.test.ts
The test validates stage and message but not errorCode (expected DIVISION_SCRAPE_FAILED). Adding this assertion would catch a future refactor that changes the code while leaving the message intact.
improvement2
zones.forEach with index is less readable than a for-loop zip
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
zones.forEach((zone, i) => { rates.push({ ..., components: componentsByZone[i]! }) }) pairs two arrays by index with a non-null assertion. A classic for-loop or a pre-zip makes the pairing explicit and drops the !, since the length guard above already guarantees safety.
isTou derivation from .some() assumes all tables share TOU character
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts
isTou = componentsByZone.some(c => ...) marks the whole page as TOU if any zone has TOU fields. This is correct for current CFE structure, but a short comment stating the assumption (all tables on one page are uniformly TOU or non-TOU) would save future readers from reconstructing the intent.
History · 14 commits
- 1ed035eneeds attentionincremental0H · 2M · 5L2026-08-11 02:03
- fcbe80dblockedfull6H · 12M · 14L2026-08-10 22:02
- 92353bdblockedincremental1H · 9M · 7L2026-08-10 19:32
- 5f2213eneeds attentionincremental1H · 6M · 10L2026-08-09 05:35
- c30da44needs attentionincremental1H · 4M · 2L2026-08-09 04:54
- 5940f56needs attentionincremental0H · 3M · 7L2026-08-07 19:15
- f222512needs attentionfull2H · 9M · 15L2026-08-07 18:58
- eec3b04needs attentionincremental2H · 1M · 4L2026-08-07 18:27
- 0b43396needs attentionincremental0H · 2M · 6L2026-08-07 01:39
- 3eb9789needs attentionincremental0H · 3M · 5L2026-08-06 18:30
- 87fc06aneeds attentionincremental0H · 4M · 9L2026-08-06 01:21
- 76bcc76needs attentionincremental0H · 1M · 5L2026-08-06 00:51
- 8948608needs attentionincremental0H · 3M · 4L2026-08-05 23:49current
- e44f6bbneeds attentionfull3H · 6M · 9L2026-08-05 19:29