feat/atlas-scan
needs attentionviewing older commit50e8a8c · incrementalpre-PRreviewed 2026-08-11 23:58 UTC3H · 5M · 7L · 3I- Purpose
- Build an authenticated, private energy infrastructure atlas for Mexico — combining transmission (CENACE nodal prices), distribution (CFE GD capacity), and site intelligence (Power Finder) into a single internal decision tool for prospective data-center customers.
- Goal
- Add CFE distribution-grid capture pipeline (WFS + per-circuit hosting capacity) and integrate SCR (short-circuit ratio) into Power Finder ranking so MW load size actually changes which sites rank highest.
- Sub-goals
- SG-1: Authenticated /atlas route behind Supabase platform login
- SG-2: Power Finder SCR/cociente-de-cortocircuito tier sort (this commit)
- SG-3: CFE distribution-grid capture pipeline (enum → capacity → compare vs PF)
- SG-4: Pilot analysis workflow (concordancia / cobertura / accionables lenses)
- What
- This commit adds SCR-based tier sorting to the Power Finder (network strength now primary sort key, savings secondary) and ships the full distribution-grid capture pipeline: 5 Python scripts that enumerate circuits by tiling the WFS, fill per-circuit hosting capacity, and cross-join the result against Power Finder substations by haversine distance. Includes a multi-agent Claude Code workflow for pilot analysis.
- Why
- Price-based sorting alone gives identical rankings for 10 MW and 300 MW loads; introducing SCR tier makes the finder sensitive to load size, surfacing Guaymas and Chihuahua over Los Mochis at 300 MW. The distribution pipeline adds an independent corroboration signal (two-source: CENACE transmission + CFE distribution) for site recommendations.
- Areas
- apps/platform/atlas-private+9462−32apps/platform/src+195−11apps/web/src/app/(marketing)/atlas+75−0scripts/atlas/congestion+1523−0scripts/atlas/distribucion+511−0scripts/atlas/pland+2624−14
- Blast
- 122 files, +14444 / -59 lines across atlas private app, platform API routes, marketing page, and research scripts. All changes are either private/internal (atlas-private HTML, scripts/) or additive (new routes, new lenses). No breaking changes to existing platform APIs. Scripts are not deployed — local-only ETL.
Findings · 22
correctness5
SQLite connection shared across ThreadPoolExecutor workers
scripts/atlas/distribucion/dist_capacity.py:60
A single `con` is passed to all threads via concurrent `ex.map`. The write happens sequentially in the consumer loop (so no crash today), but this is an invisible invariant: any future `con.execute` inside `fetch_one` will hit `ProgrammingError` or silently corrupt the DB. Add `check_same_thread=False` + a threading.Lock, or open a connection per worker and collect results.
_pi global proxy counter has a race condition under ThreadPoolExecutor
scripts/atlas/distribucion/dist_common.py:49
`_opener()` reads and increments module-level `_pi` without a lock. With 4 workers calling `_opener()` concurrently, two threads can read the same value and pick the same proxy, defeating round-robin rotation. Fix: protect the read-modify-write with a `threading.Lock`.
substations_of() fetches ALL divisions, filters in Python — silent data gap at scale
scripts/atlas/distribucion/dist_enumerate.py:23
`C.wfs('sigdis:subestacion')` passes no `cql_filter`; if the WFS paginates, target-division substations may be absent and the bbox will be silently wrong. Pass `cql=f"division='{div}'"` so the server-side filter is applied before any page cap.
SCR tier conflates 'no data' (null) with measured-medium (SCR 5–20) — both tier 1
apps/platform/atlas-private/index.html:1694
A node with zero SCC data (unknown risk) sorts equal to a node confirmed at SCR 10 (moderate). For 300 MW loads, an unknown SCR is materially riskier than a measured one. Consider a separate `tierUnknown` (-1) that sorts after tier 0 above a MW threshold.
disp fallback (tot - ins) overstates available hosting capacity
scripts/atlas/distribucion/dist_capacity.py:18
CFE's `capacidadDisponible` deducts committed loads and reserves; `tot - ins` is just the uninstalled portion — an upper bound. At minimum tag this field as 'estimated' or add a comment so downstream consumers don't treat it as the regulated figure.
security4
Hardcoded developer machine absolute path committed to repo
scripts/atlas/distribucion/analyze_pilot.workflow.js:10
`DIR = '/Users/alvaromigoya/Projects/batu-codebase/.claude/worktrees/atlas-scan/...'` — breaks on any other machine, leaks username and directory layout, and makes the workflow non-reproducible. Derive from `__dirname` / an `args` parameter, or accept `DIR` and `DATA` as workflow `args`.
File handle leak: json.dump(out, open(dst, 'w')) never closes the handle
scripts/atlas/distribucion/dist_compare.py:103
Use `with open(dst, 'w') as fh: json.dump(out, fh, ...)`. Same issue for `json.load(open(PF))` on line 61.
TLS certificate verification is default-on — positive finding
scripts/atlas/distribucion/dist_common.py
urllib.request is used without any ssl context override, so TLS validation is active. No action needed.
Proxy URL env var should be documented as trusted-operator-only input
scripts/atlas/distribucion/dist_common.py
DIST_PROXIES values are used verbatim in ProxyHandler. A comment in dist_common.py already explains the business isolation rationale; consider adding that the env var is trusted-operator-only.
conventions2
Workflow CONTEXT strings embed the hardcoded DIR path — agents fail on other machines
scripts/atlas/distribucion/analyze_pilot.workflow.js:19
Every agent subprompt includes `${DATA}` which expands to the hardcoded absolute path. On another machine the agents are instructed to Read from a non-existent path, causing silent analysis failures. Fix flows automatically from fixing `DIR`.
Silent exception swallow in wfs() hides root cause on network failures
scripts/atlas/distribucion/dist_common.py:91
`except Exception: txt = ''` discards DNS failures, SSL errors, and timeouts — the script will spin through all retries with no diagnostic output. At minimum log `str(e)` before suppressing.
tests4
SCR tier + sort logic is user-facing with no tests; scc=0 silently treated as no-data
apps/platform/atlas-private/index.html:1696
`scr=s.scc?s.scc/MW:null` — scc=0 is falsy in JS, so a node with zero short-circuit capacity gets tier=1 (middle) instead of tier=0 (weak). This can rank a genuinely weak node above a confirmed-weak one. MW=0 also produces NaN/null. A small Vitest or Node unit test on the two pure functions would catch these edge cases.
pf_key() classification has no unit tests; verde boundary is non-obvious
scripts/atlas/distribucion/dist_compare.py:39
atrap>=40 AND sat<=10 = verde — counterintuitive (high atrap + low sat). A node with atrap=39, sat=10 is amarillo, not verde. A 10-line doctest would make the boundary auditable.
disp fallback has no test for negative result (ins > tot)
scripts/atlas/distribucion/dist_capacity.py:27
If CFE data quality issues produce ins > tot, disp is negative — propagates into the DB and inflates 'holgura' counts in concordance tables.
Python ETL scripts without test framework — consistent with repo convention for atlas research tools
scripts/atlas/distribucion/dist_common.py
The congestion/ and pland/ scripts in the same directory follow the same no-test pattern. These are internal analyst tools, not production platform code. Acceptable for this tier.
improvement7
substations_of() fetches entire national dataset on every division call
scripts/atlas/distribucion/dist_enumerate.py:23
At national scale (16 divisions) this sends 16 full-dataset WFS requests — the same load pattern that triggers the HTML-shell rate block documented in the code. Pass `cql_filter` and/or a bbox per division to reduce response size and block probability.
Tier=1 default for missing SCC may mislead at large MW loads
apps/platform/atlas-private/index.html:1697
Showing 'red ajustada' (amber) for a node with no grid data is a reasonable UX hedge at 10 MW, but at 100–300 MW an unknown SCR deserves a distinct label ('sin datos de red') and lower priority. Consider a 4-tier scheme above a MW threshold.
capacidad() returns None for both 404 and transient failures — causes infinite retry
scripts/atlas/distribucion/dist_common.py:102
Circuit IDs that return HTTP 404 will always return None; they stay `cap_fetched_at IS NULL` and are retried on every run. Distinguish HTTP 404 (mark as 'NOT_FOUND') from transient errors to avoid wasted calls at scale.
No ETA or throughput reporting during tile enumeration
scripts/atlas/distribucion/dist_enumerate.py:98
With 6 s/tile sleep a national run could run for hours. Add `elapsed / i * (len(tl) - i)` to estimate remaining time, and log skipped tiles so operators can judge data completeness mid-run.
sys.argv parsed manually — no --help, crashes on missing value
scripts/atlas/distribucion/dist_capacity.py:80
All three pipeline scripts use manual `sys.argv` parsing: silently ignores unknown flags, crashes with IndexError if a flag lacks a value, no --help. A 10-line `argparse` replacement adds standard help docs and type safety.
Circuit centroid computed as mean of segment endpoints, not geographic centroid
scripts/atlas/distribucion/dist_enumerate.py:93
The mean of first-segment endpoints biases toward areas with denser segmentation near the substation. For the haversine join in dist_compare.py this may create false associations. Acceptable for pilot; document if used operationally.
Tile step 0.12° has no documented rationale or tuning guidance
scripts/atlas/distribucion/dist_enumerate.py:125
0.12° is ~13 km — adequate for northern Mexico but may trigger shell-blocks in dense metros (CDMX, MTY, GDL). Add a comment explaining the derivation and what symptom to look for when it needs tuning.
History · 16 commits
- 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
- cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
- 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01
- 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
- f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
- 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53
- 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
- 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58current
- 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
- f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
- 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
- ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
- 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
- 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
- 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
- e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53