feat/atlas-scan
needs attentionviewing older commit4cbbe8a · incrementalpre-PRreviewed 2026-08-12 14:01 UTC0H · 3M · 8L · 5I- Purpose
- Build a Mexican energy distribution-grid intelligence layer by scraping CFE's public WFS GeoServer: enumerate circuits, georeference them via transformer coordinates, and measure per-circuit solar-hosting capacity.
- Goal
- Produce a SQLite database of CFE distribution circuits (coordinates, kVA, hosting capacity) for use in the Atlas Powered Land site-screening tool, correlating distribution headroom with transmission-grid signals from Power Finder.
- Sub-goals
- SG-1: Enumerate circuits division-by-division via WFS `linea_mta_hc` tile scan
- SG-2: Georeference each circuit by averaging transformer (`banco_trans_dis`) coordinates
- SG-3: Fetch per-circuit solar-hosting capacity from CRE's published data
- SG-4: Compare distribution headroom vs Power Finder transmission signals (cross-validation)
- SG-5: Scale beyond the pilot corridor to a full CFE division despite 403 rate-limiting
- What
- Two commits: (1) `division_run.py` — new orchestrator that runs the 3-phase pipeline (enumerate → trafos → capacity) as a resumable loop, sleeping 45 min on CFE 403 blocks; `dist_enumerate.py` refactored to incremental per-tile persistence and 403 propagation. (2) `dist_compare.py` — corrects 3 pilot-audit bugs: circuit assigned to nearest sub only (was counted at every sub within radius, inflating ~56%), utilization % replaces raw MW as primary metric, and degenerate-test detection added.
- Why
- The pilot proved the data pipeline works but surfaced methodological errors in the comparison analysis and scalability limits: CFE blocks IPs at ~10k requests/hr, so a full division requires a resumable, block-tolerant runner. The comparison arithmetic was also shown to be degenerate (6/6 green agreement with no negative cases = no discriminating power).
- Areas
- scripts/atlas/distribucion+200−80apps/platform/atlas-private+13000−0apps/platform/src+120−10apps/web/src+60−0scripts/atlas/pland+3000−0
- Blast
- 125 files, +14,765/−59 lines across the branch. Bulk is data files and atlas scripts; platform code changes are limited to 2 route handlers + a proxy for the private atlas. No TypeScript domain or schema changes.
Findings · 16
correctness4
seg_count double-counts on restart — upsert accumulates across runs
scripts/atlas/distribucion/dist_enumerate.py:102
The ON CONFLICT upsert uses `seg_count=COALESCE(seg_count,0)+excluded.seg_count`. There is no per-tile checkpoint, so a restart re-processes every tile and doubles segment counts for all circuits already written. `seg_count` is analysis-only (not billing), but it silently corrupts a key sizing proxy.
In-memory `enumerado` flag causes re-enumeration on restart, compounding seg_count inflation
scripts/atlas/distribucion/division_run.py:51
`enumerado = False` is reset on every process start. If killed after enumeration completes, the next run re-enumerates the full division, doubling seg_count for all circuits via the additive upsert. The other two phases (trafos, capacity) are genuinely resumable via DB-state checks; enumeration has no such guard.
`asign` keyed by substation name — name collisions merge distinct subs into one bucket
scripts/atlas/distribucion/dist_compare.py:94
Two distinct substations with the same name (plausible in CFE's dataset, e.g. two 'HERMOSILLO' nodes in different zones) would share one `asign` key, routing circuits of one to the other. Fix: key by coordinate tuple `(round(lat,4), round(lng,4))` instead of `s['n']`.
`porpunto` rounding may collapse physically distinct substations 11 m apart
scripts/atlas/distribucion/dist_compare.py:75
4 decimal places ≈ 11 m precision. Two distinct substations within 11 m (e.g. in a dense urban compound with slightly different stored coordinates) are merged, losing one's PF signals. Tighter rounding (5 decimal places ≈ 1.1 m) reduces false merges.
security2
TLS certificate verification disabled (`CERT_NONE`) in dist_common.py
scripts/atlas/distribucion/dist_compare.py
Intentional — CFE's cert doesn't chain to Python's trust store. Acceptable for a research script reading public data. Risk escalates if this code graduates to a production ingestion path. Consider cert pinning as a middle ground.
Unclosed file handle on read path (json.load(open(PF)))
scripts/atlas/distribucion/dist_compare.py:70
CPython's GC closes it at scope end; no real risk. Wrap in `with` for hygiene.
conventions5
File handles opened without context managers — write path risks truncated output on exception
scripts/atlas/distribucion/dist_compare.py:145
`json.dump(out, open(dst, 'w'), ...)` — if json.dump raises mid-write, the file is left truncated with no cleanup. Wrap in `with open(dst, 'w') as fh: json.dump(out, fh, ...)`.
`defaultdict` imported but used inconsistently — rest of file uses `setdefault`
scripts/atlas/distribucion/dist_compare.py:106
`porpunto` and `asign` use `dict.setdefault()`; only the `kv` accumulator inside the per-circuit loop uses `defaultdict(int)`. Remove the import and use `kv.get(k, 0) + 1` for consistency.
`dist_capacity.main(div, 3)` hardcodes worker count, not surfaced to operator
scripts/atlas/distribucion/division_run.py:73
The `3` is the thread worker count for the capacity phase — a meaningful performance knob. It is invisible in the CLI help and the docstring. Either document it or expose `--workers`.
`# (1b)` comment marker appears in code but not in the module docstring
scripts/atlas/distribucion/dist_compare.py:85
The docstring enumerates fixes as (1), (2), (3). The `# (1b)` marker in the code has no matching entry. Either add '1b' to the docstring or merge the comment into '# (1) continued'.
`str(e)[:60]` in exception handler drops the error class name
scripts/atlas/distribucion/division_run.py:79
Logs truncate to 60 chars of the message, losing the exception type. `f'{type(e).__name__}: {str(e)[:50]}'` gives faster diagnosis at zero cost.
tests1
No tests for the one-to-one circuit assignment invariant in dist_compare.py
scripts/atlas/distribucion/dist_compare.py
Fix 1 corrects a real double-counting bug (111 assignments over 71 circuits). A 3-line pytest asserting `len(assigned) == len(set(c['circuito'] for c in assigned))` would guard against regression. Low priority for a research script; high value if this becomes a recurring pipeline.
improvement4
403 abort loses current tile's fetched data; restart replays entire tile loop from tile 1
scripts/atlas/distribucion/dist_enumerate.py:89
When a 403 is encountered, `raise` fires before the current tile's `acc` dict is committed. The tile's data is dropped. On restart, the tile loop begins at index 0, re-fetching all previously committed tiles. Quick fix: `con.commit()` before `raise`. For full resilience, store `last_tile_committed` in a `meta` table and slice `tl[last_tile:]` on re-entry.
`libre()` probes `subestacion` layer, not the layers actually being scraped
scripts/atlas/distribucion/division_run.py:31
The availability probe queries `sigdis:subestacion` but the active phases scrape `sigdis:linea_mta_hc` and `sigdis:banco_trans_dis`. Layer-specific rate-limiting would cause a false 'unblocked' signal, burning a cycle before hitting 403. Probe the layer matching the current phase.
`huerfanos` list uses dict-identity O(n×m) check — fragile if circuits are reloaded from DB
scripts/atlas/distribucion/dist_compare.py:129
`not any(c in v for v in asign.values())` relies on the same dict objects being in both `circuits` and `asign.values()`. Works now, but breaks silently if circuits are ever reloaded from SQLite. Fix: `asignados = {c['circuito'] for v in asign.values() for c in v}` then test by ID.
O(n×m) nearest-sub loop will not scale to full-division or national runs
scripts/atlas/distribucion/dist_compare.py:87
Pilot: 74 circuits × handful of subs = fine. Full NW division: potentially thousands × hundreds = ~10⁶ Haversine calls in pure Python. A spatial index (scipy cKDTree or bounding-box pre-filter) would give O(n log m) lookup. Not urgent now; note for national run.
History · 16 commits
- 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
- cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
- 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01current
- 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
- f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
- 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53
- 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
- 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
- 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
- f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
- 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
- ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
- 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
- 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
- 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
- e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53