feat/atlas-scan
needs attentionviewing older commit0babe51 · incrementalpre-PRreviewed 2026-08-12 01:23 UTC1H · 2M · 5L · 3I- Purpose
- Build CFE distribution grid intelligence: scrape publicly accessible hosting-capacity data (WFS + circuit capacity API) to identify constrained vs available circuits across Mexico, cross-referenced with Power Finder substations.
- Goal
- Private atlas behind platform auth + CFE distribution grid capture pipeline (enumerate → capacity → compare) with an autonomous watcher that unblocks and runs the pilot hands-free.
- Sub-goals
- SG-1: Private /atlas route behind Batu platform login (authenticated edge rewrite)
- SG-2: CFE distribution grid WFS client + SQLite pipeline (dist_common, dist_enumerate, dist_capacity, dist_compare)
- SG-3: Optional rotating-proxy support for scaling beyond single-IP throttle
- SG-4: Autonomous pilot watcher — polls WFS every 15min, runs full pipeline on unblock
- What
- Adds pilot_run.py — a single-file orchestrator that waits for the CFE WFS to become unblocked (polling every 15 min), then chains the three pipeline stages (enumerate → capacity → compare) for the Los Mochis corridor (División Noroeste, DB).
- Why
- The CFE WFS blocks by burst rate, not daily cap; an operator running the pipeline manually has to wait and re-try. This watcher automates the wait loop so the full pilot runs hands-free overnight.
- Areas
- scripts/atlas+13303−1apps/platform+345−1apps/web+852−57.gitignore+2−0
- Blast
- 123 files, +14502/-59. Mostly scripts/atlas (83 files, research scripts + data). apps/platform adds the private /atlas route (15 files). No production SaaS logic changed.
Findings · 11
correctness5
dist_enumerate.main() calls sys.exit() — silently aborts the full pipeline
scripts/atlas/distribucion/pilot_run.py:49
If no substations are found for the division, dist_enumerate.main() calls sys.exit() and the process dies. Stages 2/3 and 3/3 never run and 'PILOTO COMPLETO' is never printed. Consider wrapping in try/except SystemExit to emit a clear diagnostic and exit cleanly.
Proxy presence skips WFS availability probe unconditionally
scripts/atlas/distribucion/pilot_run.py:41
When C._PROXIES is non-empty the poll loop is skipped entirely and the pipeline runs immediately — even if the WFS is still returning the shell HTML. dist_enumerate.main() would then operate on empty/malformed data. Consider running at least one availability probe even with proxies configured, or document the assumption explicitly.
No per-stage error handling — a failure in stage N silently skips remaining stages
scripts/atlas/distribucion/pilot_run.py:49
dist_enumerate.main(), dist_capacity.main(), and dist_compare.main() are called sequentially with no try/except. If any stage raises, subsequent stages are skipped with no message. Wrap each call to at minimum print the stage name before re-raising.
Actual poll interval is longer than POLL_MIN due to probe execution time
scripts/atlas/distribucion/pilot_run.py:45
time.sleep(POLL_MIN * 60) is called after the probe completes. On slow connections the probe itself adds seconds. Negligible at 15-minute intervals, but the interval is drift-by-design rather than fixed.
C._PROXIES accesses a private/internal attribute
scripts/atlas/distribucion/pilot_run.py:41
If dist_common renames or restructures _PROXIES, the check silently evaluates to falsy and the poll loop always runs — even with proxies configured. Prefer a documented public constant or predicate in dist_common.
security2
Duplicate 'import dist_common as C' statement
scripts/atlas/distribucion/pilot_run.py:14
dist_common is imported twice (lines 13-14). Harmless in Python but is a copy-paste artifact. Remove the redundant import.
Broad except swallows unexpected errors in disponible()
scripts/atlas/distribucion/pilot_run.py:35
except Exception as e catches all non-ShellBlocked errors (including bugs like AttributeError) and treats them as 'blocked'. Intentional per the comment, but could hide bugs during development. Acceptable for a research script.
tests1
No tests — consistent with directory-wide pattern
scripts/atlas/distribucion/pilot_run.py
No tests exist for any sibling scripts in this directory. This is a research/operational script; the pattern is consistent. No action required.
improvement3
15-minute sleep is not interruptible — Ctrl-C unresponsive mid-wait
scripts/atlas/distribucion/pilot_run.py:41
time.sleep(POLL_MIN * 60) holds for up to 15 minutes. Splitting into 60 × 1-second sleeps or using threading.Event.wait() would let KeyboardInterrupt respond immediately without changing the effective poll frequency.
POLL_MIN unit conversion repeated in log message and sleep call
scripts/atlas/distribucion/pilot_run.py:45
POLL_MIN * 60 appears implicitly in both the sleep and the printed message. Naming poll_seconds = POLL_MIN * 60 at the top and using it in both places eliminates the silent unit conversion.
Pipeline stages called with positional args — fragile if signatures change
scripts/atlas/distribucion/pilot_run.py:49
dist_enumerate.main(DIV, STEP, BBOX) relies on positional order where the third param is bbox_override. Keyword args (bbox_override=BBOX) would survive reordering and clarify intent.
History · 16 commits
- 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
- cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
- 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01
- 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
- f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
- 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53
- 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23current
- 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
- 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
- f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
- 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
- ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
- 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
- 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
- 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
- e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53