← all branches

fix/tariff-scrape

needs attention
ee4bea1 · fullpre-PRreviewed 2026-08-11 01:52 UTC4H · 9M · 8L · 5I
The branch
Purpose
Fix silent data-loss bugs in the CFE tariff rate scraper — wrong prices stored under correct keys with zero failures reported, damage invisible until billing discrepancies surfaced.
Goal
Correct CFE tariff rate collection so every (tariff x zone x month) pair stores the right rate components, job counters accurately reflect scope, and failure modes fail loudly instead of silently.
Sub-goals
  • SG-1: One CFE page per rate type — never fan the same scraped cells across multiple rate type IDs
  • SG-2: Valle de Mexico divisions fan out to three zones (DL/DM/DN) with per-table extraction paired by name not position
  • SG-3: Target month verified not assumed — re-resolve month dropdown after year postback; fail loud if month not offered
  • SG-4: Per-rate-type SFN fan-out — prevents one tariff timeout killing all 9; keeps payload under 256KB state limit
  • SG-5: Shrink guard + empty-scrape guard in decideScrapedRateWrite
  • SG-6: Finalize idempotency — redelivered finalize returns stored outcome not InvalidTransition
  • SG-7: Counter denominator from REQUEST config not survivors
  • SG-8: Division failure isolation — each division uses baseResp snapshot
  • SG-9: 12 scraper invariants documented in .claude/rules/ontology.md with supporting tests
The changes (whole branch)
What
Complete rewrite of CFE tariff collector (843 lines), persist handler, and transition handler. New per-rate-type SFN fan-out. divisions.ts new module. Domain: tariff-rate.decisions.ts shrink guards, tariff-rate.shells.ts baseline-diff logic, tariff-rate.scrape.ts new file. 14 new or substantially extended test files pinning all 9 sub-goals.
Why
Scraper produced completed jobs with zero failures while storing wrong prices — GDMTH prices under GDMTO IDs, Sur prices under Norte/Centro zones, current-month prices under past-month keys. Each invariant maps to a specific historical silent-failure mode.
Areas
services/utility/tariffs/cfe+2974235domains/utility/tariff-rate+84812domains/utility/tariff-job+38316infra/cdk/tariffs+320113apps/platform/api+1502packages/api+1072.claude/rules+1650
Blast
32 files, +4947/-380. CFE tariff pipeline only. Requires CDK deploy. No DB migration.
requires-cdk-deploy no-db-migration admin-only-api
typecheck· node_modules not installed on runnertests· node_modules not installed on runnerci· no GitHub auth on runnercoderabbit· no .coderabbit.yaml in repo

Findings · 24

correctness4

medium

`startTariffJobShell` emits duplicate JOB_STARTED event on SFN retry

domains/utility/src/tariff-job/tariff-job.shells.ts:165

When MarkStarted is retried by the SFN (Lambda succeeds but response is lost), the job is already `running`. `decideStartJob` correctly returns `ok` for the idempotent re-entry — but the shell unconditionally calls `emit(JOB_STARTED, ...)`, inserting a second `utility.tariffs.cfe.job.started` event into the outbox. Compare: `finalizeTariffJobShell` (line 220: `if (decision.value.alreadyFinalized) return ok({ job, event: null })`) and `failTariffJobShell` (line 258: `if (decision.value.alreadyFailed) { return ok({ job, event: null }) }`) both suppress the event on redelivery. Fix: add `alreadyStarted` flag to the decision and guard the emit the same way.

low

SFN execution timeout budget tight under sustained retry pressure

infra/cdk/src/stacks/services/utility/tariffs/cfe/stepfunctions.stack.ts:173

9 rate types at MaxConcurrency 3 = 3 waves. Each wave's CollectOne ceiling is 900s; with max retries the wave could reach ~56 min. Three waves = 168 min, exceeding the 7200s (120 min) budget. States.Timeout cannot be caught by Catch:States.ALL, so FailJob never runs and the job row is stranded `running` until the reaper. Consider raising to 9000s or reducing MaxAttempts for the slow-retry class.

info

`formState` shared across division retries — safe by sequential execution

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:504

The `delete` + `post()` pattern correctly resets geographic cascade state per division. Sequential `await` in the `for..of` loop means no race today. A future `Promise.all` refactor would break this silently — a `formState` clone per invocation would make the safety property structural.

info

All 12 ontology invariants correctly implemented — no regression found

Verified against every invariant in `.claude/rules/ontology.md`. Each of the 12 rules maps to a confirmed implementation in the code. The mutation-verified comment pattern is consistently applied.

security5

medium

TLS certificate verification disabled globally for all CFE HTTP scraping

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:820

`setGlobalDispatcher(new Agent({ connect: { rejectUnauthorized: false } }))` disables TLS verification for the entire Lambda process. Any MITM-capable network path between the Lambda's egress and cfe.mx (e.g. a misconfigured VPC NAT, a compromised transit gateway) can silently feed the scraper arbitrary HTML. The scraper trusts the table it finds and persists those prices as authoritative. This is a financial data pipeline scraping rate data — the cost of a wrong rate is a billing error, not a display artifact. Use certificate pinning or, at minimum, restrict the `rejectUnauthorized: false` to the specific Imperva clearance call rather than setting it globally.

medium

`tariff_jobs` table world-readable via `USING(true)` RLS policy

packages/database/drizzle/0054_illegal_human_cannonball.sql:178

The RLS policy `tariff_jobs_select_authenticated` uses `USING(true)` with no org scoping, making the entire table readable by any authenticated user of any tenant via PostgREST. The table stores SFN execution ARNs and sanitized failure messages. Tracked in BAT-320; no action needed in this PR, but noting for the security record. Failure message sanitization (`sanitizePipelineError`) reduces the impact.

medium

`rateTypes` accepts arbitrary strings — no allowlist validation at API boundary

packages/api/src/schemas/tariff-job.schemas.ts:39

`CreateTariffJobApiInputSchema` validates `rateTypes` as `z.array(z.string().min(1)).min(1)` — any non-empty string is accepted. An unrecognized rate type gets an `UNSUPPORTED_RATE_TYPE` error from the collector (correct loud failure per invariant 1), but it still creates a job row and triggers an SFN execution before failing. Adding `z.enum(['GDMTH','GDMTO','DIT',...])` at the schema level closes this loop and prevents jobs that are guaranteed to fail from accumulating.

low

Chromium launched with `--ignore-certificate-errors` for Imperva clearance

services/utility/tariffs/cfe/src/lib/browser-clearance.ts:52

The browser clearance call launches Chromium with `--ignore-certificate-errors`. This is isolated to the cookie-harvest step and the browser is discarded after — the actual scraping uses the `undici` HTTP client seeded with the harvested cookies. The risk is that a MITM during the harvest step injects wrong clearance cookies. Lower risk than the `setGlobalDispatcher` finding but worth scoping the flag more tightly if possible.

info

Auth gate correctly ordered — isPlatformAdmin before input validation

apps/platform/src/api/handlers/tariff-jobs.handler.ts:45

The handler correctly checks `isPlatformAdmin` before parsing or using any user-supplied input. Correct order — reduces attack surface.

conventions5

high

Mapper returns HTTP 500 for PersistenceError but contract declares no 500 response

apps/platform/src/api/contracts/tariff-jobs.contract.ts:54

The mapper returns `status: 500` for PersistenceError and in the exhaustive-default branch, but the contract's `responses` map only declares 201, 400, 401, 403, 409, 422. ts-rest validates responses at runtime against the declared map — a 500 response will cause a TypeScript error. Add a `500: z.object({ status: z.literal('error'), message: z.string() })` entry to the contract, or re-route PersistenceError to a declared status.

medium

API schemas use `z.infer` instead of `satisfies z.ZodType<ApiType>`

packages/api/src/schemas/tariff-job.schemas.ts:55

Types are derived with `export type TariffJobCreatedResponse = z.infer<typeof TariffJobCreatedResponseSchema>`. The canonical form prescribes `satisfies z.ZodType<ApiType>` to ensure Zod alignment with a hand-authored type rather than deriving the type FROM the schema. With `z.infer`, a change to the schema silently changes the type; with `satisfies`, the divergence is a build error.

medium

Event type strings hardcoded as inline literals in shells, not from events file

domains/utility/src/tariff-rate/tariff-rate.shells.ts:96

The shells use `eventType: 'utility.tariff_rate.created'` as inline string literals. The co-located events file exports Zod schemas with the event shape; the canonical pattern is to use a const from the events file so a rename in one place propagates. A misspelled inline literal would be a silent event-routing failure.

low

`decideUpdateProgress` does not return `Result<T,E>`

domains/utility/src/tariff-job/tariff-job.decisions.ts:172

All fallible domain decisions should return `Result<T,E>` from `@batu/result`. `decideUpdateProgress` returns void. If this function gains any failure path, the caller has no error channel.

low

Tariff-job event types defined inline in shells module, not in a co-located events file

domains/utility/src/tariff-job/tariff-job.shells.ts:41

The canonical form says events are co-located in `events/{entity}.events.ts`. The tariff-job event type strings are defined directly in the shells file. Add `domains/utility/src/events/tariff-job.events.ts`.

tests7

high

Valle de México table-to-zone pairing test pins config ordering, not correctness

services/utility/tariffs/cfe/__tests__/config-integrity.test.ts:65

The test comment explicitly states: 'This entry CONTRADICTS ITSELF: containedDivisions is [NORTE, CENTRO, SUR] while division — which reads like the verbatim CFE dropdown label — says CENTRO, NORTE, SUR.' The test passes because it only validates the config's internal consistency (that containedDivisions entries match known zones), not that the order matches what CFE actually renders. The real-page fixture (valle-real-page.test.ts) is the load-bearing correctness check here — if that fixture is ever replaced with a synthetic one, the pairing breaks silently.

medium

`persistTouSchedulesShell` has no test coverage

domains/utility/src/tariff-rate/:1

`persist-scraped-rates.test.ts` covers `persistScrapedRatesShell` comprehensively. But `persistTouSchedulesShell` — the sibling shell that persists horario schedules from the collector — has no test file and no test coverage. A bug in this shell would produce silent wrong TOU schedules with no detection.

medium

Semipunta band not exercised through the collector invariant-4 gate

services/utility/tariffs/cfe/__tests__/collector.test.ts:577

Invariant 4 (horario gate uses `hasTouEnergyComponent` including `genSpCost`) was the actual bug. The unit test in `tariff-rate.scrape.test.ts` correctly pins that `genSpCost` is required, but the collector integration test doesn't exercise the end-to-end path with a Semipunta tariff — a real-page fixture for a Semipunta tariff would close the highest-risk invariant-4 gap.

low

`assertApplied` null-selected (missing `selected` attr) path not tested

services/utility/tariffs/cfe/__tests__/collector.test.ts:408

Only the mismatch path is tested. A test with no `selected` attribute confirming it passes through without throwing documents the intentional leniency.

low

`pairTablesToZones` positional-fallback path not tested

services/utility/tariffs/cfe/__tests__/collector.test.ts:1

The pure positional fallback (no headings on any table, count matches zones) is not tested. A single test case closes this gap.

info

Mutation-verified comment pattern consistently applied

The 'mutation verified' comments throughout the test files (noting which specific mutation was tried and caught) are exemplary documentation and significantly raise confidence in the test suite quality.

info

Real CFE page fixture (`valle-real-page.test.ts`) is the strongest coverage in the suite

services/utility/tariffs/cfe/__tests__/valle-real-page.test.ts:1

Parsing a real saved CFE page HTML is what actually proves invariant 8 (per-table extraction) works. This is the right approach — synthetic fixtures can't catch CFE layout surprises.

improvement3

medium

Three independent NFD accent-stripping implementations

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:278

Three separate implementations: (1) `normalizeZoneName` in the collector (NFD + combining strip + uppercase), (2) `norm` in the horario classifier (NFD + combining strip + lowercase), (3) in `divisions.ts`. Extract a shared `normalizeSpanish(s, case='upper'): string` to `lib/normalize.ts`. A future fix in one (e.g. adding non-breaking space handling) won't silently miss the others.

medium

`TIME_RANGE_RE` is a module-level regex with /g flag — safe with `matchAll` but surprising

services/utility/tariffs/cfe/src/handlers/tariff-collector.lambda.ts:418

`matchAll` requires the /g flag and internally creates a fresh iterator, so the module-level shared regex is safe. However, `RegExp.lastIndex` state from any other use of `TIME_RANGE_RE.exec()` or `TIME_RANGE_RE.test()` would cause bugs. Use `String.raw` or declare inline within the function to make the safety property structural rather than conventional.

low

Unsafe double-cast `as unknown as Database` repeated in persist and transition handlers

services/utility/tariffs/cfe/src/handlers/tariff-persist.lambda.ts:65

Both Lambda handlers cast the DB client via `as unknown as Database`. Extract a typed `getLambdaDb(): Database` helper to centralize the cast and make the suppression explicit.