← all branches

feat/atlas-scan

needs attentionviewing older commit
4cbbe8a · incrementalpre-PRreviewed 2026-08-12 14:01 UTC0H · 3M · 8L · 5I
The branch
Purpose
Build a Mexican energy distribution-grid intelligence layer by scraping CFE's public WFS GeoServer: enumerate circuits, georeference them via transformer coordinates, and measure per-circuit solar-hosting capacity.
Goal
Produce a SQLite database of CFE distribution circuits (coordinates, kVA, hosting capacity) for use in the Atlas Powered Land site-screening tool, correlating distribution headroom with transmission-grid signals from Power Finder.
Sub-goals
  • SG-1: Enumerate circuits division-by-division via WFS `linea_mta_hc` tile scan
  • SG-2: Georeference each circuit by averaging transformer (`banco_trans_dis`) coordinates
  • SG-3: Fetch per-circuit solar-hosting capacity from CRE's published data
  • SG-4: Compare distribution headroom vs Power Finder transmission signals (cross-validation)
  • SG-5: Scale beyond the pilot corridor to a full CFE division despite 403 rate-limiting
The changes (whole branch)
What
Two commits: (1) `division_run.py` — new orchestrator that runs the 3-phase pipeline (enumerate → trafos → capacity) as a resumable loop, sleeping 45 min on CFE 403 blocks; `dist_enumerate.py` refactored to incremental per-tile persistence and 403 propagation. (2) `dist_compare.py` — corrects 3 pilot-audit bugs: circuit assigned to nearest sub only (was counted at every sub within radius, inflating ~56%), utilization % replaces raw MW as primary metric, and degenerate-test detection added.
Why
The pilot proved the data pipeline works but surfaced methodological errors in the comparison analysis and scalability limits: CFE blocks IPs at ~10k requests/hr, so a full division requires a resumable, block-tolerant runner. The comparison arithmetic was also shown to be degenerate (6/6 green agreement with no negative cases = no discriminating power).
Areas
scripts/atlas/distribucion+20080apps/platform/atlas-private+130000apps/platform/src+12010apps/web/src+600scripts/atlas/pland+30000
Blast
125 files, +14,765/−59 lines across the branch. Bulk is data files and atlas scripts; platform code changes are limited to 2 route handlers + a proxy for the private atlas. No TypeScript domain or schema changes.
research-scripts-only no-production-domain-changes data-pipeline
typecheck· Python scripts — no TypeScript typechecktests· No test suite for atlas Python scriptscoderabbit· No .coderabbit.yaml in repogithub-ci· No GH auth available on runner

Findings · 16

correctness4

medium

seg_count double-counts on restart — upsert accumulates across runs

scripts/atlas/distribucion/dist_enumerate.py:102

The ON CONFLICT upsert uses `seg_count=COALESCE(seg_count,0)+excluded.seg_count`. There is no per-tile checkpoint, so a restart re-processes every tile and doubles segment counts for all circuits already written. `seg_count` is analysis-only (not billing), but it silently corrupts a key sizing proxy.

medium

In-memory `enumerado` flag causes re-enumeration on restart, compounding seg_count inflation

scripts/atlas/distribucion/division_run.py:51

`enumerado = False` is reset on every process start. If killed after enumeration completes, the next run re-enumerates the full division, doubling seg_count for all circuits via the additive upsert. The other two phases (trafos, capacity) are genuinely resumable via DB-state checks; enumeration has no such guard.

low

`asign` keyed by substation name — name collisions merge distinct subs into one bucket

scripts/atlas/distribucion/dist_compare.py:94

Two distinct substations with the same name (plausible in CFE's dataset, e.g. two 'HERMOSILLO' nodes in different zones) would share one `asign` key, routing circuits of one to the other. Fix: key by coordinate tuple `(round(lat,4), round(lng,4))` instead of `s['n']`.

low

`porpunto` rounding may collapse physically distinct substations 11 m apart

scripts/atlas/distribucion/dist_compare.py:75

4 decimal places ≈ 11 m precision. Two distinct substations within 11 m (e.g. in a dense urban compound with slightly different stored coordinates) are merged, losing one's PF signals. Tighter rounding (5 decimal places ≈ 1.1 m) reduces false merges.

security2

low

TLS certificate verification disabled (`CERT_NONE`) in dist_common.py

scripts/atlas/distribucion/dist_compare.py

Intentional — CFE's cert doesn't chain to Python's trust store. Acceptable for a research script reading public data. Risk escalates if this code graduates to a production ingestion path. Consider cert pinning as a middle ground.

info

Unclosed file handle on read path (json.load(open(PF)))

scripts/atlas/distribucion/dist_compare.py:70

CPython's GC closes it at scope end; no real risk. Wrap in `with` for hygiene.

conventions5

low

File handles opened without context managers — write path risks truncated output on exception

scripts/atlas/distribucion/dist_compare.py:145

`json.dump(out, open(dst, 'w'), ...)` — if json.dump raises mid-write, the file is left truncated with no cleanup. Wrap in `with open(dst, 'w') as fh: json.dump(out, fh, ...)`.

low

`defaultdict` imported but used inconsistently — rest of file uses `setdefault`

scripts/atlas/distribucion/dist_compare.py:106

`porpunto` and `asign` use `dict.setdefault()`; only the `kv` accumulator inside the per-circuit loop uses `defaultdict(int)`. Remove the import and use `kv.get(k, 0) + 1` for consistency.

low

`dist_capacity.main(div, 3)` hardcodes worker count, not surfaced to operator

scripts/atlas/distribucion/division_run.py:73

The `3` is the thread worker count for the capacity phase — a meaningful performance knob. It is invisible in the CLI help and the docstring. Either document it or expose `--workers`.

low

`# (1b)` comment marker appears in code but not in the module docstring

scripts/atlas/distribucion/dist_compare.py:85

The docstring enumerates fixes as (1), (2), (3). The `# (1b)` marker in the code has no matching entry. Either add '1b' to the docstring or merge the comment into '# (1) continued'.

info

`str(e)[:60]` in exception handler drops the error class name

scripts/atlas/distribucion/division_run.py:79

Logs truncate to 60 chars of the message, losing the exception type. `f'{type(e).__name__}: {str(e)[:50]}'` gives faster diagnosis at zero cost.

tests1

info

No tests for the one-to-one circuit assignment invariant in dist_compare.py

scripts/atlas/distribucion/dist_compare.py

Fix 1 corrects a real double-counting bug (111 assignments over 71 circuits). A 3-line pytest asserting `len(assigned) == len(set(c['circuito'] for c in assigned))` would guard against regression. Low priority for a research script; high value if this becomes a recurring pipeline.

improvement4

medium

403 abort loses current tile's fetched data; restart replays entire tile loop from tile 1

scripts/atlas/distribucion/dist_enumerate.py:89

When a 403 is encountered, `raise` fires before the current tile's `acc` dict is committed. The tile's data is dropped. On restart, the tile loop begins at index 0, re-fetching all previously committed tiles. Quick fix: `con.commit()` before `raise`. For full resilience, store `last_tile_committed` in a `meta` table and slice `tl[last_tile:]` on re-entry.

low

`libre()` probes `subestacion` layer, not the layers actually being scraped

scripts/atlas/distribucion/division_run.py:31

The availability probe queries `sigdis:subestacion` but the active phases scrape `sigdis:linea_mta_hc` and `sigdis:banco_trans_dis`. Layer-specific rate-limiting would cause a false 'unblocked' signal, burning a cycle before hitting 403. Probe the layer matching the current phase.

low

`huerfanos` list uses dict-identity O(n×m) check — fragile if circuits are reloaded from DB

scripts/atlas/distribucion/dist_compare.py:129

`not any(c in v for v in asign.values())` relies on the same dict objects being in both `circuits` and `asign.values()`. Works now, but breaks silently if circuits are ever reloaded from SQLite. Fix: `asignados = {c['circuito'] for v in asign.values() for c in v}` then test by ID.

info

O(n×m) nearest-sub loop will not scale to full-division or national runs

scripts/atlas/distribucion/dist_compare.py:87

Pilot: 74 circuits × handful of subs = fine. Full NW division: potentially thousands × hundreds = ~10⁶ Haversine calls in pure Python. A spatial index (scipy cKDTree or bounding-box pre-filter) would give O(n log m) lookup. Not urgent now; note for national run.

History · 16 commits

  1. 91aaedfsafeincremental0H · 1M · 1L2026-08-12 17:35
  2. cb8d915needs attentionincremental2H · 5M · 3L2026-08-12 17:29
  3. 4cbbe8aneeds attentionincremental0H · 3M · 8L2026-08-12 14:01current
  4. 4aa3c02needs attentionincremental2H · 3M · 3L2026-08-12 02:35
  5. f5630b2needs attentionincremental0H · 4M · 5L2026-08-12 02:17
  6. 28fde5bneeds attentionincremental1H · 2M · 2L2026-08-12 01:53
  7. 0babe51needs attentionincremental1H · 2M · 5L2026-08-12 01:23
  8. 50e8a8cneeds attentionincremental3H · 5M · 7L2026-08-11 23:58
  9. 3af4686needs attentionincremental3H · 5M · 5L2026-08-03 20:07
  10. f5d3266needs attentionincremental0H · 3M · 8L2026-08-03 19:40
  11. 9f8b61aneeds attentionincremental0H · 2M · 3L2026-08-03 19:15
  12. ea51fa0needs attentionincremental1H · 4M · 4L2026-08-03 19:02
  13. 2b33f2fneeds attentionincremental0H · 1M · 1L2026-07-18 05:09
  14. 2f8cf79needs attentionincremental1H · 3M · 4L2026-07-18 00:59
  15. 1616332safeincremental0H · 0M · 1L2026-07-18 00:22
  16. e997fd8needs attentionfull1H · 4M · 6L2026-07-17 23:53