← all branches

feat/atlas-scan

needs attentionviewing older commit
f5630b2 · incrementalpre-PRreviewed 2026-08-12 02:17 UTC0H · 4M · 5L · 8I
The branch
Purpose
Build a CFE energy atlas — capturing public grid data (distribution circuits, substations, hosting capacity) and private/processed layers as a gated map product.
Goal
Scrape CFE's public WFS to enumerate distribution circuits with hosting capacity per circuit across Mexico, enabling solar site qualification against grid proximity and available capacity.
Sub-goals
  • SG-1: Serve the atlas privately from /api/atlas (authenticated, gated from public /atlas)
  • SG-2: CFE distribution WFS pipeline — enumerate substations + circuits + hosting capacity
  • SG-3: Pilot watcher — polls WFS every 15 min until unblocked, then runs full pipeline
  • SG-4: Rotating-proxy support for WFS at national scale
  • SG-5: Error-handling hardening — distinguish transient vs permanent vs rate-limit failures
The changes (whole branch)
What
Incremental: refactors wfs() exception classification (4xx abort, 5xx retry, SSLError abort) and adds propertyName filter to substations call to avoid 165KB polygon causing HTTP 500s.
Why
Prior SSL confusion bug (CERTIFICATE_VERIFY_FAILED masked as blocked) wasted hours of wait time; full subestacion layer hit HTTP 500 at scale.
Areas
apps/platform/atlas-private+14000−0apps/platform/src/app/api/atlas + lib/atlas+107−0apps/platform/src/lib/supabase+140−1apps/web/src/app/(marketing)/atlas+19−21scripts/atlas/distribucion+345−30scripts/atlas/pland + congestion+900−0
Blast
123 files, +14,533/−59. Heavy on data files (atlas-private/ JSON/GeoJSON). Code changes are platform API routes + atlas serving + Python research scripts. No domain TypeScript code touched.
research-script data-pipeline no-pr-yet
➖CI· No open PR — no CI run available➖coderabbit· No .coderabbit.yaml in repo

Findings · 11

correctness3

medium

5xx on probe (retries=1) raises RuntimeError instead of returning False — watcher crashes

scripts/atlas/distribucion/pilot_run.py:31

disponible() calls wfs(..., retries=1). A transient 5xx exhausts the single attempt and raises RuntimeError (not ShellBlocked). disponible() does not catch RuntimeError, so it propagates and crashes the watcher. Fix: catch RuntimeError in disponible(), or use retries=2.

low

'coordenada' as property_name — verify it is the geometry field, not a string attribute

scripts/atlas/distribucion/dist_enumerate.py:27

If 'coordenada' is a WFS attribute column (string), not the GML geometry field, then f['geometry'] may be null downstream. Code reads f['geometry']['coordinates'] — KeyError/TypeError if geometry is excluded.

info

Exception ordering is correct — HTTPError before generic Exception

scripts/atlas/distribucion/dist_common.py:101

urllib.error.HTTPError is a subclass of Exception; placing it first is required and correct.

security1

info

type_name in RuntimeError messages — no risk (hardcoded callers only)

scripts/atlas/distribucion/dist_common.py:106

type_name is always a hardcoded string literal; no injection or data-exposure risk.

conventions1

info

property_name field list covers all downstream .get() calls

scripts/atlas/distribucion/dist_enumerate.py:27

All fields accessed via p.get() are present. Comment is accurate and idiomatic.

tests2

medium

No tests for refactored exception-handling logic — prior SSL bug was exactly this type of misclassification

scripts/atlas/distribucion/dist_common.py

A small pytest file mocking urllib responses to cover 4xx-abort/5xx-retry/SSLError-abort branches would prevent regression. Not a block for research scripts.

info

No tests is the established pattern for this tooling — not a block

scripts/atlas/distribucion/

Research/scraping script with no prior test suite.

improvement4

medium

Retry sleep is implicit (fall-through on parse failure) rather than explicit after error

scripts/atlas/distribucion/dist_common.py

After 5xx, txt='' → parse fails → falls through to time.sleep(). Correct but obscured. A continue after setting last_err/txt='' would make intent explicit.

medium

`attempt` loop variable is unused — communicates wrong intent

scripts/atlas/distribucion/dist_common.py:99

Counter is never read in the loop body. Using _ instead signals an iteration-count loop.

low

Probe and enumeration property_name lists intentionally differ — add comment or constant

scripts/atlas/distribucion/pilot_run.py:31

Probe uses 'sub,coordenada'; enumeration uses a longer list. Intentional subset, but prone to drift if enumeration list changes.

info

_pi proxy-rotation counter is fine for sequential use

scripts/atlas/distribucion/dist_common.py

No thread-safety concern for single-threaded usage.

History · 16 commits

  1. 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
  2. cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
  3. 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01
  4. 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
  5. f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17current
  6. 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53
  7. 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
  8. 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
  9. 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
  10. f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
  11. 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
  12. ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
  13. 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
  14. 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
  15. 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
  16. e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53