← all branches

feat/atlas-scan

needs attentionviewing older commit
28fde5b · incrementalpre-PRreviewed 2026-08-12 01:53 UTC1H · 2M · 2L · 3I
The branch
Purpose
Build an energy grid intelligence atlas for Mexico: scrape, process, and serve CFE distribution grid data (circuits, substations, hosting capacity, congestion, PLAND sites) for the Batu platform.
Goal
Add a private atlas viewer served through the platform API with auth-gated proxy, plus research scripts for CFE distribution grid capacity pipeline and site-screening (PLAND).
Sub-goals
  • SG-1: Atlas data moved from apps/web/public to apps/platform/atlas-private (private, auth-gated)
  • SG-2: Auth-gated proxy route in apps/platform API (atlas/[...path]/route.ts)
  • SG-3: CFE distribution grid scraping pipeline (dist_enumerate, dist_capacity, dist_common)
  • SG-4: Congestion layer pipeline (GeoServer WFS → node/zone analysis)
  • SG-5: PLAND site screening pipeline (geo scoring, exclusions, portafolio merge)
  • SG-6: Rotating proxy support + pilot watcher (autonomous polling)
  • SG-7: Bug fix — WFS CERTIFICATE_VERIFY_FAILED confused with IP block (this commit)
The changes (whole branch)
What
Single-file fix: dist_common.py. Adds SSL context with CERT_NONE to bypass CFE's non-validating cert chain in Python's trust store. Captures exceptions as last_err to distinguish network/SSL failures from rate-limit shell responses. Raises RuntimeError (not ShellBlocked) when retries exhaust on a hard error.
Why
The WFS retry loop was treating CERTIFICATE_VERIFY_FAILED as a rate-limit block, causing the pipeline to wait ~11 min per failure before giving up, masking the real error for hours.
Areas
apps/platform+109058apps/web+1921scripts/atlas/distribucion+5940scripts/atlas/congestion+7160scripts/atlas/pland+80000data files (json/csv/geojson)+00
Blast
~100+ files, +10k/−79 lines across platform API, research scripts, and data files. Platform changes are additive (new proxy route). Scripts are standalone research tools.
SSL verification disabled for app.cfe.mx — correct fix is truststore or system CA bundle Data files are large static blobs — check git-lfs policy if sizes grow
ci· No PR — no CI rollup availablecoderabbit· No .coderabbit.yaml

Findings · 8

correctness4

medium

Exponential backoff still runs on hard SSL/network errors

scripts/atlas/distribucion/dist_common.py:101

When an exception fires, `last_err = e` is set and `txt = ''`, but execution falls through to `time.sleep(delay)`. On a persistent hard error (the very case this fix addresses), the caller still waits ~11 minutes (3+6+12+…+120s) before RuntimeError is raised. A `continue` after `last_err = e` would skip the sleep on connection-level failures. With CERT_NONE now active, SSL errors are resolved upstream, so this is dormant in practice — but the logic remains wrong conceptually.

low

Mixed-failure scenario: last exception may shadow the real root cause

scripts/atlas/distribucion/dist_common.py:96

`last_err` is overwritten on every retry. If early retries fail with SSL and later retries get shell HTML (no exception), `last_err` is still set but `ShellBlocked` is raised — the SSL cause is lost. Storing only the first error (`if last_err is None: last_err = e`) would preserve the original root cause for debugging.

info

`last_err` overwritten each iteration — only last exception survives

scripts/atlas/distribucion/dist_common.py:100

If multiple different errors occur across retries, only the last one is reported in RuntimeError. Use `if last_err is None: last_err = e` to preserve the first (and typically most diagnostic) error.

info

Fix correctly distinguishes SSL errors from rate-limit blocks

scripts/atlas/distribucion/dist_common.py

The core logic is sound: after exhausting retries, `last_err is not None` → RuntimeError (network/SSL); `last_err is None` → ShellBlocked (rate-limit). The distinction works correctly for the common case.

security2

high

CERT_NONE is the wrong fix — truststore or system CA bundle preserves verification

scripts/atlas/distribucion/dist_common.py:29

The comment itself states CFE's cert validates in the system/curl trust store — just not in Python's bundled certifi. The correct fix is `pip install truststore` + `truststore.inject_into_ssl()` at script startup, or passing `cafile='/etc/ssl/certs/ca-certificates.crt'` to `ssl.create_default_context()`. CERT_NONE removes ALL TLS integrity permanently, even if CFE fixes their chain or this module is reused elsewhere. For a one-off research script on a trusted network the operational risk is low (no credentials transmitted, public data only), but the pattern is wrong and the correct fix is straightforward.

medium

Module-level _SSL singleton with CERT_NONE affects all callers

scripts/atlas/distribucion/dist_common.py:27

`_SSL` is a mutable module-level singleton. Any code that imports `dist_common` and calls `_opener()` will silently get a no-verify TLS context. Acceptable for a single-file script, but should be annotated as `# INTERNAL — do not reuse outside this module` to prevent accidental reuse.

conventions2

low

capacidad() has the same exception-swallowing pattern — not fixed by this diff

scripts/atlas/distribucion/dist_common.py:120

`capacidad()` still uses `except Exception: time.sleep(...)` returning None. The same rationale (SSL errors ≠ rate-limit) applies. Out of scope for this commit but worth a follow-up.

info

`attempt` variable unused in loop — use `_`

scripts/atlas/distribucion/dist_common.py:97

`for attempt in range(retries)` — `attempt` is never read; backoff uses `delay` directly. Replace with `for _ in range(retries)` to signal the variable is intentionally unused.

History · 16 commits

  1. 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
  2. cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
  3. 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01
  4. 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
  5. f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
  6. 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53current
  7. 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
  8. 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
  9. 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
  10. f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
  11. 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
  12. ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
  13. 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
  14. 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
  15. 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
  16. e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53