OLDGABE — research log

Does the CAUTION-recovery re-ranker actually beat the production tier scorer? — head-to-head + de-confounding

The framing. OLDGABE's fusion stage sorts every candidate fire into tiers; the great majority land in a low-confidence CAUTION tier and are never surfaced. The question that had never been measured: among those CAUTION events, can a learned recovery model rank the genuine hidden fires better than the production tier/confidence scorer does?

BLUF. On the operationally meaningful metrics the recovery model decisively beats production. At an 80%-precision operating point it recovers 60.5% of confirmed real fires vs production's 9.5% (Truth1), and 27.2% vs 0% under a second, HMS-independent truth (Truth2). Production holds a small global-AUC edge on Truth1 (0.951 vs 0.936) — but that edge is a confound: production uses the NOAA HMS smoke channel, and Truth1 is 91.5% HMS-confirmed, so it is partly scoring HMS against HMS. When the truth is switched to an independent polar-satellite corroboration, production's AUC edge reverses (recovery 0.845 vs production 0.786).

Populations & truths (reproducible). Candidate set = every canonical event with event_corroboration_tier.tier='CAUTION' that has an event_ground_truth row (n = 197,325). Truth1 (multi-source) = event_ground_truth.confirmed (6,867 positive, 3.48%; for CAUTION this channel is 91.5% NOAA-HMS-confirmed). Truth2 (neutral polar) = a VIIRS/MODIS active-fire detection from a different canonical event within 5 km and −1..+14 days of the event's first detection (7,791 positive, 3.95%). The two truths overlap only partially (both-positive 1,753; Truth1-only 5,114; Truth2-only 6,038), so Truth2 is a genuine cross-check, not a relabelling.

Features (non-leaky, detection-time only). Evidence is joined excluding the confirmation channels IRWIN, HMS, SAR and S2. Per event: n_det, n_fam, max/mean_frp, max/mean_conf, span_h, n_days, max_ti4, ti4−ti5 contrast, frac_night, mean_pixel_area, max_ti5, abi_temp_k, abi_area, modis_brightness, mean_hour, fp_dist_km, frp_per_pixel. Model: gradient-boosted trees (HistGradientBoostingClassifier, class-weight balanced), scored out-of-fold under 5-fold GroupKFold grouped on 0.2° spatial cells (a held-out cell is never in training). The production scorer is the fixed tier_and_conf() confidence; its value is pinned at 0.02 for ~94% of CAUTION events (the GOES/ABI-only majority), which is why it cannot rank within the tier.

Result.

De-confounding (does the win survive removing the reporting channel?). Feature-group AUCs, per truth — month-only / geography-only (lat,lon) / detection-only / all:

Reads: (1) Season is not a confound — the candidate window is summer-only, so the summer-share of positives and of negatives is identical (gap = 0) and month-only ranking is near-chance (~0.57). (2) Geography is a large prior under both truths (0.903 vs 0.877), which already argues it is real fire geography rather than pure reporting bias — a reporting-only signal would predict the HMS truth far better than the polar truth. (3) Production's Truth1 AUC lead does not survive the neutral truth. The honest operating-point story is unchanged: production cannot deliver a high-precision set within CAUTION; the recovery model can.

Adding geography to the recovery re-ranker — a defensible lift that grows under the neutral truth

The framing. Entry 0001 showed geography alone ranks CAUTION fires surprisingly well. Wildfires cluster by fuel, terrain and climate, so a location prior should carry real signal — but a location prior can also just memorise where fires get reported. This entry adds geography to the recovery model and tests, honestly, whether the lift is real fire behaviour or a reporting/coverage artifact.

BLUF. Adding raw lat/lon to the detection feature set (recovery+geo) is a defensible improvement over both the detection-only recovery model and production, on both truths and every metric. The decisive de-confounding result: the geo lift is larger under the HMS-independent polar truth (+0.081 AUC) than under the reporting-heavy truth (+0.036 AUC). If geography were merely encoding reporting bias, its benefit would shrink when the truth stops depending on reporting — instead it grows. So the geographic signal is real fire geography.

What "geo" is (hard data bound). OLDGABE currently has no populated land-cover / fuel / terrain / climatology layer — the fire-weather cell table is empty, weather stations are a current snapshot (temporally invalid for historical events), and the raw detection JSON carries no geographic fields. So the only non-leaky geographic priors available are raw lat/lon + distance-to-known-false-source. A Truth1-only supplementary that adds an out-of-neighbourhood polar-thermal climatology density (a fuel/fire-proneness proxy; circular against Truth2, hence Truth1-only) lifts a little further (AUC 0.978, recall@80% 0.771), indicating headroom if a real independent fuel/climatology layer were ingested.

Method delta from entry 0001. Same populations (n = 202,005; the DB ingested more June/July events between runs), same non-leaky features, same 5-fold GroupKFold on 0.2° cells, same two truths. New models: recovery+geo = detection features + lat + lon; geo-only = lat,lon; all = detection + lat + lon + month. A June↔July temporal-transport split is added as the only distribution shift the data permits.

Result — head-to-head.

The geo lift over detection-only is +0.036 AUC on Truth1 and +0.081 AUC on Truth2 — it grows under the neutral truth. recovery+geo also closes and reverses the one metric where production had led (Truth1 AUC, now 0.973 vs 0.950). Detection and geography are complementary: recovery+geo (0.941) beats both detection-only (0.860) and geo-only (0.886), so neither is redundant. Month adds nothing (all ≈ recovery+geo).

Result — temporal transport (train one month, test the other).

recovery+geo beats detection-only in all four cells and production in three of four (it loses only Jun→Jul on the HMS-confounded Truth1, by 0.007). Under the neutral truth it wins both directions by wide margins, and it stabilises the hard direction (Jun→Jul detection-only falls to 0.888; recovery+geo holds 0.934). Within the available window, the geographic prior transports across time and does not hurt.

The cross-sensor fusion safety-net: two independent satellites agreeing is 4.5× better than one

The framing. Each satellite makes its own noisy fire hunches. OLDGABE's fusion daemon is the “two-witnesses” rule: it links detections of the same fire across sensors and only trusts a candidate strongly when independent instruments agree. This entry measures how much that agreement is worth.

BLUF. OLDGABE ingests ~1.80 million raw per-sensor fire detections. The fusion daemon cross-links them into 11,441 multi-sensor clusters and grades them. Requiring ≥2 independent CORE thermal sensors to agree (the “confirmed” tier, n=620) raises the IRWIN/perimeter match rate to 37.7%, versus 8.4% for a random raw single detection — a 4.5× precision lift — while GOES-only single-frame candidates (the “early” tier) match just 5.5%. Separately, the false-positive catalog + status logic demote 620,160 raw candidates (34.4%) to masked-fp so they never reach the map.

Result.

Dozier subpixel strict-gate: a clean precision-for-recall trade, and why we keep it out of fusion

The framing. The Dozier subpixel method estimates what fraction of a coarse GOES pixel is actually burning (p_subpixel). Tightening a threshold on that fraction is a dial between “catch everything, mostly noise” and “only the unambiguous, but almost nothing.”

BLUF. Gating on p_subpixel trades recall for precision cleanly: the IRWIN match rate rises from 2.8% (ungated) to ~18–27% at p_subpixel ≥ 0.005–0.01, but keeps only ~10–19% of Dozier's own positives. Because the high-precision operating point yields so few detections — and essentially none the polar sensors don't already provide — OLDGABE's fusion daemon deliberately excludes Dozier. A measured “keep it, but don't fuse it” decision, not an oversight.

Result (60,000 sampled Dozier detections; 1,681 IRWIN-positive = 2.8% base):

The deployed strict gate independently scores 26.3% precision (G19) / 48.3% (G18) at ~1.3% whole-fire recall in the daily-scorer (entry 0007) — consistent with this sweep.

Negative result: GOES temporal persistence does not add defensible recall beyond polar

The framing. A geostationary satellite stares at the same spot every few minutes, so a tempting idea is: “if GOES keeps seeing a hotspot in one place for hours, that persistence is extra evidence of a real fire the polar satellites missed between overpasses.” We tested it. It does not hold up.

BLUF. GOES near-match “recall” is saturated by coverage: 92.4% of IRWIN fire cells have some GOES detection within ~5 km, simply because GOES-ABI emits candidate hotspots almost everywhere over CONUS (its measured precision is only ~15% — 85% are not fire). Requiring temporal persistence (a cell seen in ≥3 distinct hours) only drops this coverage-inflated number (92.4% → 66.1%); it never adds genuine recall. The apparent 43.8% “marginal over polar” is the same coverage/false-positive floor, not independent detections. Conclusion: GOES temporal persistence is not a validated recall lever — OLDGABE's defensible cross-sensor recall comes from polar corroboration and fusion (entry 0003).

Result (45-day window; 2,592 IRWIN truth cells at 0.05°):

Wildfire-camera smoke detection: highest precision of any source, but coverage/visibility-bound

The framing. OLDGABE runs a smoke/flame classifier on public wildfire-camera frames and, when two cameras see the same plume, triangulates the bearings into a map location. Cameras see smoke a satellite can't — but only where a camera is pointed, in daylight, through clear air, within range.

BLUF. When a camera fires on a fire IRWIN also records, it is usually right — daily-scorer precision 66.7%, the highest of any single OLDGABE source. But it matches only 1.2% of IRWIN fires: coverage and visibility (line-of-sight, daylight, haze, range) are the binding constraint, not the classifier. Honest status: a high-precision corroborator where cameras can see, with no proven standalone field value yet at network scale.

Result (daily-scorer, vs IRWIN 10 nm / 24 h):

Per-sensor scorecard vs NIFC IRWIN — what each US datalink actually contributes

The framing. Rather than argue which feed matters, OLDGABE runs a daily replay scorer that matches every sensor's detections to NIFC IRWIN ground truth and records precision, recall and count per source. This entry is that scorecard, unedited.

BLUF. Polar VIIRS is the best single recall+precision workhorse; the fused OLDGABE output leads recall (72.4%); cameras lead precision (66.7%); GOES-ABI is high-volume/low-precision; SAR and Sentinel-2 SWIR contributed nothing in this window. This is the empirical basis for weighting the fusion daemon (entry 0003) toward polar + agreement.

Result (daily-scorer run 18, 24 h window ending 2026-07-08, match 10 nm / 24 h vs IRWIN):

Reframing the camera 1.2%: coverage-conditional recall — cameras detect up to ~25% of fires they can actually see

The framing. Entry 0006's 1.2% camera recall was measured against every US fire — but a camera can only see fires inside its viewshed. The fair question is: where a camera can physically see, how often does it catch the fire? And is the gap infrastructure (coverage) or algorithm (detection)?

BLUF. Conditioning on coverage reframes the number by 4–20×: within 60 km of an online camera, cameras detect 24.7% of IRWIN fires; within 20–40 km, 8.8–19.0%. But two gaps remain and split cleanly: coverage — only 28–59% of fires are within R km of an online camera — and detection-in-coverage — even where a camera can see, only ~5–25% of fires are caught (night, haze, range, azimuth). Precision rises with R to 72% (matching entry 0006/0007). Cameras remain a high-precision corroborator; the primary lever is detection-in-coverage, but it is capped by coverage.

Result (3,903 deduped IRWIN incidents; 10,227 online cameras; 1,016 smoke≥0.5 hits):

Repositioning Dozier: a median 6-hour early cue, and a multi-feature gate that dominates the single threshold

The framing. Entry 0004 showed the single-p_subpixel Dozier gate is high-precision but low-yield, and excluded from fusion. Two follow-ups ask whether Dozier is worth more than that: is it early, and can a smarter gate move the curve out rather than just slide along it?

BLUF. Both hold. (1) Lead-time: for fires seen by both Dozier and a polar sensor (n=472), Dozier is earlier 53% of the time, median ~6 hours earlier — an early-cue / lead-time tier, not a recall source. (2) Multi-feature gate: replacing the single threshold with a 13-feature classifier lifts PR-AUC from 0.044 to 0.179 (4×) and precision-at-recall from 8.0% to 39.4% (at 10% recall). The multi-feature curve dominates the single threshold at every operating point.

Result A — lead-time (472 fires seen by both Dozier and polar):

Result B — multi-feature gate vs single p_subpixel (out-of-fold):

Is re-enabling Sentinel-2 SWIR worth it? Excellent coverage, ~4% active-fire marginal recall — a measured no

The framing. Sentinel-2 SWIR sits at zero in the scorecard (entry 0007). SWIR can see active flame at 20 m — so would re-enabling full CONUS S2 processing recover fires the current sensors miss, and is it worth the cost?

BLUF. Of 1,764 IRWIN fires in the S2 window, only 191 (10.8%) were missed by every current sensor. Over those missed fires, S2 scene availability is 100% — but an actual SWIR active-fire signal appears at only 4% of them (2/50, CI 1–14%), and already-ingested S2 added 0 unique fires. Extrapolated unique marginal recall ≈ 4% of the missed set — small — at a compute cost of ~1–2 TB/week and up to 5-day latency. Verdict: not worth full re-enablement for timely alerting.

Result.

Making standalone Dozier usable: a multi-feature gate (49% vs 12% precision) and an early-cue tier (median 4.3 h before polar) — shadow-only

The framing. The production pipeline hides standalone GOES ABI Dozier subpixel detections entirely — on their own they are ~0.05% wildfire precision (entry 0004/0007), so Dozier only counts as corroboration for a fire another sensor already carries. Two levers could change that without touching the live gate: (1) replace the single p_subpixel threshold with a multi-feature gate, and (2) surface a Dozier early-cue tier for detections that fire before polar confirmation. This entry measures both, shadow-only.

BLUF. On 100,000 random Dozier candidates (IRWIN/perimeter truth, 2.04% base, 5-fold spatial CV), a 13-feature classifier reaches 49.3% precision at the ~10% operating recall where the single p_subpixel threshold sits at only 11.6% — a +37.7 pp gain (PR-AUC 0.234 vs 0.051). It survives dropping the detection-density feature (39.6% precision, PR-AUC unchanged), so it is not a spatial-clustering artifact. Separately, Dozier detections that precede a polar detection are 38.4% IRWIN-confirmed — ~19× the 2.0% Dozier baseline — at a median 4.3 h lead. Both are GO candidates. Nothing is surfaced.

Piece 1 — multi-feature gate vs the single threshold.

Flip operating point (go/no-go). Cut over to shadow-score τ = 0.965 → 49.3% precision at 10% recall (the recall the current gate already runs at, so no recall is lost while precision ×4.3). For a stricter 40% surfacing bar the gate still returns 32.8% recall. At matched precision the single threshold collapses to 0.2% recall — the gate holds ~50× more. OOF ROC-AUC 0.869.

Piece 2 — Dozier early-cue tier.

Pre-flip check kills the flip: the multi-feature Dozier gate collapses out-of-time — we are NOT promoting it

The framing. Entry 0011 reported the 13-feature Dozier gate reaching ~49% precision vs 12% for the single p_subpixel threshold under 5-fold spatial cross-validation, and flagged it a GO candidate pending a fresh out-of-window confirmation. This entry is that confirmation. Before wiring anything into the live gate we ran the strongest temporal test the data allows — train on an earlier window, test on a strictly-later held-out window — with a hard decision rule: promote only if the multi-feature gate still clearly dominates out-of-time; otherwise stop.

BLUF. It does not dominate — it collapses. Trained on the earlier window and tested on strictly-later data, the multi-feature gate falls to near-chance ranking (holdout ROC-AUC 0.60–0.64) and is beaten by the trivial single p_subpixel threshold at every operating point, in both a count-based split and a positives-balanced split. The exact flip point τ=0.965 transfers to zero holdout recall. Decision: HOLD — do not flip. The 49% spatial-CV figure was time-correlated leakage. The production Dozier gate stays exactly as-is (standalone Dozier still hidden), and the shadow canary stays gated.

Result — holdout precision at the current gate's operating recall (~10%).

Precision across the recall curve (single vs multi-feature, holdout). Single wins at every point:

Does the Dozier early-cue tier survive out-of-time? Not validated — its 38% lives entirely in a not-yet-truth-matured window

The framing. The gate lever failed out-of-time (entry 0012). The other Dozier lever from entry 0011 is the early-cue tier: a Dozier detection that precedes a polar (VIIRS/MODIS) detection of the same fire within 5 km/+48 h, reported there at 38.4% IRWIN-confirmed precision (~19× the ~2% Dozier baseline) with a median 4.3 h lead. It is a deterministic rule, not a fitted model, so it cannot “overfit” in the ML sense — the honest test is temporal stability: measure precision + lead on an earlier vs a strictly-later, truth-matured window.

BLUF. A clean matured earlier-vs-later split turned out to be impossible, and that fact is the finding. Dozier detected_at is bimodal: ~82% is a 2-day historical backfill (2026-05-30/31), June 01–25 has zero Dozier rows, and live ingestion only resumed ~07-01. So the only fully ±14 d truth-settled data is the backfill; the recent live window is days old and its IRWIN truth is right-censored. On the settled-truth backfill the tier scores just 2.5% precision; the headline 38% survives only in the recent window — 77% of all early-cue detections come from that not-yet-matured window — where truth cannot yet be confirmed. Verdict: NOT validated out-of-time. The tier stays gated; re-test after ~2026-07-23, when the live window's ±14 d truth settles.

Result — the same tier, measured on each cohort.

*Cohort B truth is right-censored (detections 0–7 days old vs the ±14 d IRWIN matching window), so 39.5% is a lower bound that is not yet confirmable, not a validated number.

What this shows. Entry 0011's 38.4%/19×/4.3 h is faithfully reproduced as the blended figure (31.1%/18.3×/4.8 h) — but it is supplied almost entirely by cohort B: 2,037 of 2,637 early-cue detections (77%) are in the recent, unsettled-truth window. On the only settled-truth data (cohort A) the tier is near baseline (2.5% vs 1.0%), and its 24.6 h median “lead” is far longer than a polar revisit — consistent with coincidental later polar hits, not genuine early warning. Cohort B's tight 3.7 h leads look like a real early-warning signal, but its truth is not settled. Polar availability is comparable across the two windows (11,268 vs 10,594 dedup polar cells), so cohort A's low precision is not a missing-polar artifact.

Why cameras miss fires they can see — and a lever that lifts detection-in-coverage 37→49% at flat precision

The framing. Entry 0008 split the camera gap in two: coverage (infrastructure — is a fire near an online camera at all) and detection-in-coverage (algorithm — where a camera can see, how often it catches the fire), the latter only ~5–25%. Coverage is a build-out problem; detection-in-coverage is the algorithm lever. This entry asks why the in-coverage misses happen and tests the best fix.

BLUF. The dominant miss driver is range, not darkness: within 15 km cameras catch 47% of in-coverage fires, falling to 20% at 15–30 km and 4% at 30–40 km (undetectable at sensor resolution). The assumed night floor does not exist — night recall (45%) actually beats day (31%), almost certainly flame glow. The best lever is the simplest: lower the single-frame smoke trigger 0.50→0.30, which lifts detection-in-coverage 36.9%→49.4% (+12.5 pts, +34% relative) at essentially flat precision (58.1%→57.2%), monotonic across the sweep and stable in 5-fold spatial CV (0.50±0.11 vs 0.37±0.11); at a tighter 20 km viewshed it nearly doubles recall (17.1%→34.2%). Adding temporal persistence was a mistake — it hurt recall (to 24.7%). We flag the threshold lever as a shadow-first build candidate, gated until it passes the temporal-transport test that killed the Dozier gate (entry 0012).

Why the misses happen (in-coverage, R=40 km, 1,083 fires, current 0.5 trigger).

Range dominates (a clean 12× swing, 47%→4%); day/night runs the “wrong” way (glow); size is noisy at tiny n and not a usable driver. Where cameras do catch a fire, they flag smoke a median 14.7 h before the IRWIN discovery timestamp (within the ±24 h match window) — cameras are an early corroborator when they fire.

The lever — single-frame threshold sweep.

The physical floor (achievable headroom). Two hard limits, neither fixable by thresholding: (1) range — 30–40 km fires are ~undetectable (3.9%); (2) among near (<15 km) misses, 39% have a best smoke-probability of ~0 (the model never fired at all — out-of-field-of-view / occlusion / too faint; camera azimuth+FOV are recorded but unused, so a fire behind a fixed camera is invisible regardless of threshold). Only ~26% of near misses sit in the 0.30–0.50 band the lever actually recovers. So the lever captures the genuinely-recoverable slice; it does not (and cannot) fix the geometry floor.

The camera-threshold lever survives out-of-time (where the Dozier gate died) — validated, and built shadow-first

The framing. Entry 0014 found that lowering the single-frame smoke trigger 0.50→0.30 lifts detection-in-coverage 36.9%→49.4% at flat precision. Before building, we applied the exact discipline that killed the Dozier gate (entry 0012): does it hold out-of-time, and does it generalize across camera networks?

BLUF. It holds. Trained on 2026-07-01–04 and tested on the strictly-later 07-06–08, the 0.30 trigger still gains +8.3 pts of detection-in-coverage (34.8%→43.1%) at ~flat precision (56.1%→54.0%). It generalizes across every network — recall rises on all four vendors with precision essentially unchanged; on the purpose-built fire networks it is dramatic (HPWREN 45%→91% recall at 79% precision). This is the opposite of the Dozier gate, which inverted out-of-time. We therefore built it — shadow-first and gated: a hidden canary logs the would-surface detections; production is untouched until an explicit flip.

Temporal transport (train 07-01–04 → test 07-06–08, gap 07-05).

Out-of-time gain +8.3 pts recall, precision −2.1 pts (within the flat band). Verdict: HOLDS.

Per-network generalization (full window).

The 0.30 trigger lifts recall on every vendor with precision flat. Precision is network-driven (purpose-built fire cams 78–81% vs webcams 40–56%), not threshold-driven — so 0.30 generalizes as a global trigger, though surfacing tiers should still be per-network.

The operating point we would flip to, and its gate. A global single-frame trigger of 0.30 (replacing the per-network 0.55–0.85). Not flipped: it is built as a hidden shadow canary (cam_smoke_lowthresh_shadow) that logs every ≥0.30 detection, flags which are newly recovered vs the current per-network production thresholds, and records IRWIN proximity for a precision audit. On this run: 2,926 detections ≥0.30, of which 2,506 are newly recovered vs production, and 1,394 of those sit near a real IRWIN fire. The table is surface_allowed=0 with a DO_NOT_SURFACE marker and no serving-app read path; the production cam_smoke_daemon.py is byte-for-byte unchanged. Flip requires Mark's go plus a live precision audit of the newly-recovered set.