feat/atlas-scan
needs attentionviewing older commit28fde5b · incrementalpre-PRreviewed 2026-08-12 01:53 UTC1H · 2M · 2L · 3I- Purpose
- Build an energy grid intelligence atlas for Mexico: scrape, process, and serve CFE distribution grid data (circuits, substations, hosting capacity, congestion, PLAND sites) for the Batu platform.
- Goal
- Add a private atlas viewer served through the platform API with auth-gated proxy, plus research scripts for CFE distribution grid capacity pipeline and site-screening (PLAND).
- Sub-goals
- SG-1: Atlas data moved from apps/web/public to apps/platform/atlas-private (private, auth-gated)
- SG-2: Auth-gated proxy route in apps/platform API (atlas/[...path]/route.ts)
- SG-3: CFE distribution grid scraping pipeline (dist_enumerate, dist_capacity, dist_common)
- SG-4: Congestion layer pipeline (GeoServer WFS → node/zone analysis)
- SG-5: PLAND site screening pipeline (geo scoring, exclusions, portafolio merge)
- SG-6: Rotating proxy support + pilot watcher (autonomous polling)
- SG-7: Bug fix — WFS CERTIFICATE_VERIFY_FAILED confused with IP block (this commit)
- What
- Single-file fix: dist_common.py. Adds SSL context with CERT_NONE to bypass CFE's non-validating cert chain in Python's trust store. Captures exceptions as last_err to distinguish network/SSL failures from rate-limit shell responses. Raises RuntimeError (not ShellBlocked) when retries exhaust on a hard error.
- Why
- The WFS retry loop was treating CERTIFICATE_VERIFY_FAILED as a rate-limit block, causing the pipeline to wait ~11 min per failure before giving up, masking the real error for hours.
- Areas
- apps/platform+1090−58apps/web+19−21scripts/atlas/distribucion+594−0scripts/atlas/congestion+716−0scripts/atlas/pland+8000−0data files (json/csv/geojson)+0−0
- Blast
- ~100+ files, +10k/−79 lines across platform API, research scripts, and data files. Platform changes are additive (new proxy route). Scripts are standalone research tools.
Findings · 8
correctness4
Exponential backoff still runs on hard SSL/network errors
scripts/atlas/distribucion/dist_common.py:101
When an exception fires, `last_err = e` is set and `txt = ''`, but execution falls through to `time.sleep(delay)`. On a persistent hard error (the very case this fix addresses), the caller still waits ~11 minutes (3+6+12+…+120s) before RuntimeError is raised. A `continue` after `last_err = e` would skip the sleep on connection-level failures. With CERT_NONE now active, SSL errors are resolved upstream, so this is dormant in practice — but the logic remains wrong conceptually.
Mixed-failure scenario: last exception may shadow the real root cause
scripts/atlas/distribucion/dist_common.py:96
`last_err` is overwritten on every retry. If early retries fail with SSL and later retries get shell HTML (no exception), `last_err` is still set but `ShellBlocked` is raised — the SSL cause is lost. Storing only the first error (`if last_err is None: last_err = e`) would preserve the original root cause for debugging.
`last_err` overwritten each iteration — only last exception survives
scripts/atlas/distribucion/dist_common.py:100
If multiple different errors occur across retries, only the last one is reported in RuntimeError. Use `if last_err is None: last_err = e` to preserve the first (and typically most diagnostic) error.
Fix correctly distinguishes SSL errors from rate-limit blocks
scripts/atlas/distribucion/dist_common.py
The core logic is sound: after exhausting retries, `last_err is not None` → RuntimeError (network/SSL); `last_err is None` → ShellBlocked (rate-limit). The distinction works correctly for the common case.
security2
CERT_NONE is the wrong fix — truststore or system CA bundle preserves verification
scripts/atlas/distribucion/dist_common.py:29
The comment itself states CFE's cert validates in the system/curl trust store — just not in Python's bundled certifi. The correct fix is `pip install truststore` + `truststore.inject_into_ssl()` at script startup, or passing `cafile='/etc/ssl/certs/ca-certificates.crt'` to `ssl.create_default_context()`. CERT_NONE removes ALL TLS integrity permanently, even if CFE fixes their chain or this module is reused elsewhere. For a one-off research script on a trusted network the operational risk is low (no credentials transmitted, public data only), but the pattern is wrong and the correct fix is straightforward.
Module-level _SSL singleton with CERT_NONE affects all callers
scripts/atlas/distribucion/dist_common.py:27
`_SSL` is a mutable module-level singleton. Any code that imports `dist_common` and calls `_opener()` will silently get a no-verify TLS context. Acceptable for a single-file script, but should be annotated as `# INTERNAL — do not reuse outside this module` to prevent accidental reuse.
conventions2
capacidad() has the same exception-swallowing pattern — not fixed by this diff
scripts/atlas/distribucion/dist_common.py:120
`capacidad()` still uses `except Exception: time.sleep(...)` returning None. The same rationale (SSL errors ≠ rate-limit) applies. Out of scope for this commit but worth a follow-up.
`attempt` variable unused in loop — use `_`
scripts/atlas/distribucion/dist_common.py:97
`for attempt in range(retries)` — `attempt` is never read; backoff uses `delay` directly. Replace with `for _ in range(retries)` to signal the variable is intentionally unused.
History · 16 commits
- 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
- cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
- 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01
- 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
- f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
- 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53current
- 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
- 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
- 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
- f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
- 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
- ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
- 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
- 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
- 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
- e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53