fix/tariff-scrape
needs attentionee4bea1 · fullpre-PRreviewed 2026-08-11 01:52 UTC4H · 9M · 8L · 5I- Purpose
- Fix silent data-loss bugs in the CFE tariff rate scraper — wrong prices stored under correct keys with zero failures reported, damage invisible until billing discrepancies surfaced.
- Goal
- Correct CFE tariff rate collection so every (tariff x zone x month) pair stores the right rate components, job counters accurately reflect scope, and failure modes fail loudly instead of silently.
- Sub-goals
- SG-1: One CFE page per rate type — never fan the same scraped cells across multiple rate type IDs
- SG-2: Valle de Mexico divisions fan out to three zones (DL/DM/DN) with per-table extraction paired by name not position
- SG-3: Target month verified not assumed — re-resolve month dropdown after year postback; fail loud if month not offered
- SG-4: Per-rate-type SFN fan-out — prevents one tariff timeout killing all 9; keeps payload under 256KB state limit
- SG-5: Shrink guard + empty-scrape guard in decideScrapedRateWrite
- SG-6: Finalize idempotency — redelivered finalize returns stored outcome not InvalidTransition
- SG-7: Counter denominator from REQUEST config not survivors
- SG-8: Division failure isolation — each division uses baseResp snapshot
- SG-9: 12 scraper invariants documented in .claude/rules/ontology.md with supporting tests
- What
- Complete rewrite of CFE tariff collector (843 lines), persist handler, and transition handler. New per-rate-type SFN fan-out. divisions.ts new module. Domain: tariff-rate.decisions.ts shrink guards, tariff-rate.shells.ts baseline-diff logic, tariff-rate.scrape.ts new file. 14 new or substantially extended test files pinning all 9 sub-goals.
- Why
- Scraper produced completed jobs with zero failures while storing wrong prices — GDMTH prices under GDMTO IDs, Sur prices under Norte/Centro zones, current-month prices under past-month keys. Each invariant maps to a specific historical silent-failure mode.
- Areas
- services/utility/tariffs/cfe+2974−235domains/utility/tariff-rate+848−12domains/utility/tariff-job+383−16infra/cdk/tariffs+320−113apps/platform/api+150−2packages/api+107−2.claude/rules+165−0
- Blast
- 32 files, +4947/-380. CFE tariff pipeline only. Requires CDK deploy. No DB migration.
Findings · 24
correctness4
`startTariffJobShell` emits duplicate JOB_STARTED event on SFN retry
domains/utility/src/tariff-job/tariff-job.shells.ts:165
When MarkStarted is retried by the SFN (Lambda succeeds but response is lost), the job is already `running`. `decideStartJob` correctly returns `ok` for the idempotent re-entry — but the shell unconditionally calls `emit(JOB_STARTED, ...)`, inserting a second `utility.tariffs.cfe.job.started` event into the outbox. Compare: `finalizeTariffJobShell` (line 220: `if (decision.value.alreadyFinalized) return ok({ job, event: null })`) and `failTariffJobShell` (line 258: `if (decision.value.alreadyFailed) { return ok({ job, event: null }) }`) both suppress the event on redelivery. Fix: add `alreadyStarted` flag to the decision and guard the emit the same way.
SFN execution timeout budget tight under sustained retry pressure
infra/cdk/src/stacks/services/utility/tariffs/cfe/stepfunctions.stack.ts:173
9 rate types at MaxConcurrency 3 = 3 waves. Each wave's CollectOne ceiling is 900s; with max retries the wave could reach ~56 min. Three waves = 168 min, exceeding the 7200s (120 min) budget. States.Timeout cannot be caught by Catch:States.ALL, so FailJob never runs and the job row is stranded `running` until the reaper. Consider raising to 9000s or reducing MaxAttempts for the slow-retry class.
`formState` shared across division retries — safe by sequential execution
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:504
The `delete` + `post()` pattern correctly resets geographic cascade state per division. Sequential `await` in the `for..of` loop means no race today. A future `Promise.all` refactor would break this silently — a `formState` clone per invocation would make the safety property structural.
All 12 ontology invariants correctly implemented — no regression found
Verified against every invariant in `.claude/rules/ontology.md`. Each of the 12 rules maps to a confirmed implementation in the code. The mutation-verified comment pattern is consistently applied.
security5
TLS certificate verification disabled globally for all CFE HTTP scraping
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:820
`setGlobalDispatcher(new Agent({ connect: { rejectUnauthorized: false } }))` disables TLS verification for the entire Lambda process. Any MITM-capable network path between the Lambda's egress and cfe.mx (e.g. a misconfigured VPC NAT, a compromised transit gateway) can silently feed the scraper arbitrary HTML. The scraper trusts the table it finds and persists those prices as authoritative. This is a financial data pipeline scraping rate data — the cost of a wrong rate is a billing error, not a display artifact. Use certificate pinning or, at minimum, restrict the `rejectUnauthorized: false` to the specific Imperva clearance call rather than setting it globally.
`tariff_jobs` table world-readable via `USING(true)` RLS policy
packages/database/drizzle/0054_illegal_human_cannonball.sql:178
The RLS policy `tariff_jobs_select_authenticated` uses `USING(true)` with no org scoping, making the entire table readable by any authenticated user of any tenant via PostgREST. The table stores SFN execution ARNs and sanitized failure messages. Tracked in BAT-320; no action needed in this PR, but noting for the security record. Failure message sanitization (`sanitizePipelineError`) reduces the impact.
`rateTypes` accepts arbitrary strings — no allowlist validation at API boundary
packages/api/src/schemas/tariff-job.schemas.ts:39
`CreateTariffJobApiInputSchema` validates `rateTypes` as `z.array(z.string().min(1)).min(1)` — any non-empty string is accepted. An unrecognized rate type gets an `UNSUPPORTED_RATE_TYPE` error from the collector (correct loud failure per invariant 1), but it still creates a job row and triggers an SFN execution before failing. Adding `z.enum(['GDMTH','GDMTO','DIT',...])` at the schema level closes this loop and prevents jobs that are guaranteed to fail from accumulating.
Chromium launched with `--ignore-certificate-errors` for Imperva clearance
services/utility/tariffs/cfe/src/lib/browser-clearance.ts:52
The browser clearance call launches Chromium with `--ignore-certificate-errors`. This is isolated to the cookie-harvest step and the browser is discarded after — the actual scraping uses the `undici` HTTP client seeded with the harvested cookies. The risk is that a MITM during the harvest step injects wrong clearance cookies. Lower risk than the `setGlobalDispatcher` finding but worth scoping the flag more tightly if possible.
Auth gate correctly ordered — isPlatformAdmin before input validation
apps/platform/src/api/handlers/tariff-jobs.handler.ts:45
The handler correctly checks `isPlatformAdmin` before parsing or using any user-supplied input. Correct order — reduces attack surface.
conventions5
Mapper returns HTTP 500 for PersistenceError but contract declares no 500 response
apps/platform/src/api/contracts/tariff-jobs.contract.ts:54
The mapper returns `status: 500` for PersistenceError and in the exhaustive-default branch, but the contract's `responses` map only declares 201, 400, 401, 403, 409, 422. ts-rest validates responses at runtime against the declared map — a 500 response will cause a TypeScript error. Add a `500: z.object({ status: z.literal('error'), message: z.string() })` entry to the contract, or re-route PersistenceError to a declared status.
API schemas use `z.infer` instead of `satisfies z.ZodType<ApiType>`
packages/api/src/schemas/tariff-job.schemas.ts:55
Types are derived with `export type TariffJobCreatedResponse = z.infer<typeof TariffJobCreatedResponseSchema>`. The canonical form prescribes `satisfies z.ZodType<ApiType>` to ensure Zod alignment with a hand-authored type rather than deriving the type FROM the schema. With `z.infer`, a change to the schema silently changes the type; with `satisfies`, the divergence is a build error.
Event type strings hardcoded as inline literals in shells, not from events file
domains/utility/src/tariff-rate/tariff-rate.shells.ts:96
The shells use `eventType: 'utility.tariff_rate.created'` as inline string literals. The co-located events file exports Zod schemas with the event shape; the canonical pattern is to use a const from the events file so a rename in one place propagates. A misspelled inline literal would be a silent event-routing failure.
`decideUpdateProgress` does not return `Result<T,E>`
domains/utility/src/tariff-job/tariff-job.decisions.ts:172
All fallible domain decisions should return `Result<T,E>` from `@batu/result`. `decideUpdateProgress` returns void. If this function gains any failure path, the caller has no error channel.
Tariff-job event types defined inline in shells module, not in a co-located events file
domains/utility/src/tariff-job/tariff-job.shells.ts:41
The canonical form says events are co-located in `events/{entity}.events.ts`. The tariff-job event type strings are defined directly in the shells file. Add `domains/utility/src/events/tariff-job.events.ts`.
tests7
Valle de México table-to-zone pairing test pins config ordering, not correctness
services/utility/tariffs/cfe/__tests__/config-integrity.test.ts:65
The test comment explicitly states: 'This entry CONTRADICTS ITSELF: containedDivisions is [NORTE, CENTRO, SUR] while division — which reads like the verbatim CFE dropdown label — says CENTRO, NORTE, SUR.' The test passes because it only validates the config's internal consistency (that containedDivisions entries match known zones), not that the order matches what CFE actually renders. The real-page fixture (valle-real-page.test.ts) is the load-bearing correctness check here — if that fixture is ever replaced with a synthetic one, the pairing breaks silently.
`persistTouSchedulesShell` has no test coverage
domains/utility/src/tariff-rate/:1
`persist-scraped-rates.test.ts` covers `persistScrapedRatesShell` comprehensively. But `persistTouSchedulesShell` — the sibling shell that persists horario schedules from the collector — has no test file and no test coverage. A bug in this shell would produce silent wrong TOU schedules with no detection.
Semipunta band not exercised through the collector invariant-4 gate
services/utility/tariffs/cfe/__tests__/collector.test.ts:577
Invariant 4 (horario gate uses `hasTouEnergyComponent` including `genSpCost`) was the actual bug. The unit test in `tariff-rate.scrape.test.ts` correctly pins that `genSpCost` is required, but the collector integration test doesn't exercise the end-to-end path with a Semipunta tariff — a real-page fixture for a Semipunta tariff would close the highest-risk invariant-4 gap.
`assertApplied` null-selected (missing `selected` attr) path not tested
services/utility/tariffs/cfe/__tests__/collector.test.ts:408
Only the mismatch path is tested. A test with no `selected` attribute confirming it passes through without throwing documents the intentional leniency.
`pairTablesToZones` positional-fallback path not tested
services/utility/tariffs/cfe/__tests__/collector.test.ts:1
The pure positional fallback (no headings on any table, count matches zones) is not tested. A single test case closes this gap.
Mutation-verified comment pattern consistently applied
The 'mutation verified' comments throughout the test files (noting which specific mutation was tried and caught) are exemplary documentation and significantly raise confidence in the test suite quality.
Real CFE page fixture (`valle-real-page.test.ts`) is the strongest coverage in the suite
services/utility/tariffs/cfe/__tests__/valle-real-page.test.ts:1
Parsing a real saved CFE page HTML is what actually proves invariant 8 (per-table extraction) works. This is the right approach — synthetic fixtures can't catch CFE layout surprises.
improvement3
Three independent NFD accent-stripping implementations
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:278
Three separate implementations: (1) `normalizeZoneName` in the collector (NFD + combining strip + uppercase), (2) `norm` in the horario classifier (NFD + combining strip + lowercase), (3) in `divisions.ts`. Extract a shared `normalizeSpanish(s, case='upper'): string` to `lib/normalize.ts`. A future fix in one (e.g. adding non-breaking space handling) won't silently miss the others.
`TIME_RANGE_RE` is a module-level regex with /g flag — safe with `matchAll` but surprising
services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:418
`matchAll` requires the /g flag and internally creates a fresh iterator, so the module-level shared regex is safe. However, `RegExp.lastIndex` state from any other use of `TIME_RANGE_RE.exec()` or `TIME_RANGE_RE.test()` would cause bugs. Use `String.raw` or declare inline within the function to make the safety property structural rather than conventional.
Unsafe double-cast `as unknown as Database` repeated in persist and transition handlers
services/utility/tariffs/cfe/src/handlers/tariff-persist.lambda.ts:65
Both Lambda handlers cast the DB client via `as unknown as Database`. Extract a typed `getLambdaDb(): Database` helper to centralize the cast and make the suppression explicit.