Does the CAUTION-recovery re-ranker actually beat the production tier scorer? — head-to-head + de-confounding
Date: 2026-07-08 Status: defensible (measured; 5-fold spatial CV; two independent truths)
The framing. OLDGABE's fusion stage sorts every candidate fire into tiers; the great majority
land in a low-confidence CAUTION tier and are never surfaced. The question that had never been measured:
among those CAUTION events, can a learned recovery model rank the genuine hidden fires better than the
production tier/confidence scorer does?
BLUF. On the operationally meaningful metrics the recovery model decisively beats
production. At an 80%-precision operating point it recovers 60.5% of confirmed real fires vs production's
9.5% (Truth1), and 27.2% vs 0% under a second, HMS-independent truth (Truth2). Production
holds a small global-AUC edge on Truth1 (0.951 vs 0.936) — but that edge is a confound: production
uses the NOAA HMS smoke channel, and Truth1 is 91.5% HMS-confirmed, so it is partly scoring HMS against HMS. When
the truth is switched to an independent polar-satellite corroboration, production's AUC edge reverses
(recovery 0.845 vs production 0.786).
Populations & truths (reproducible). Candidate set = every canonical event with
event_corroboration_tier.tier='CAUTION' that has an event_ground_truth row
(n = 197,325). Truth1 (multi-source) = event_ground_truth.confirmed
(6,867 positive, 3.48%; for CAUTION this channel is 91.5% NOAA-HMS-confirmed). Truth2 (neutral polar) = a
VIIRS/MODIS active-fire detection from a different canonical event within 5 km and −1..+14 days
of the event's first detection (7,791 positive, 3.95%). The two truths overlap only partially (both-positive 1,753;
Truth1-only 5,114; Truth2-only 6,038), so Truth2 is a genuine cross-check, not a relabelling.
Features (non-leaky, detection-time only). Evidence is joined excluding the confirmation
channels IRWIN, HMS, SAR and S2. Per event: n_det, n_fam, max/mean_frp, max/mean_conf, span_h, n_days,
max_ti4, ti4−ti5 contrast, frac_night, mean_pixel_area, max_ti5, abi_temp_k, abi_area, modis_brightness,
mean_hour, fp_dist_km, frp_per_pixel. Model: gradient-boosted trees
(HistGradientBoostingClassifier, class-weight balanced), scored out-of-fold under
5-fold GroupKFold grouped on 0.2° spatial cells (a held-out cell is never in training). The
production scorer is the fixed tier_and_conf() confidence; its value is pinned at 0.02 for ~94% of
CAUTION events (the GOES/ABI-only majority), which is why it cannot rank within the tier.
Result.
| Truth1 (multi-source, HMS-heavy) | AUC | PR-AUC | recall@90%p | recall@80%p |
|---|
| recovery model (detection features) | 0.936 | 0.735 | 0.379 | 0.605 |
production tier_and_conf | 0.951 | 0.544 | 0.065 | 0.095 |
| production indep-families | 0.500 | — | 0.000 | 0.000 |
| production max-FRP | 0.787 | 0.161 | 0.006 | 0.019 |
| Truth2 (neutral polar, HMS-independent) | AUC | PR-AUC | recall@90%p | recall@80%p |
|---|
| recovery model (detection features) | 0.845 | 0.511 | 0.064 | 0.272 |
production tier_and_conf | 0.786 | 0.273 | 0.000 | 0.000 |
De-confounding (does the win survive removing the reporting channel?). Feature-group AUCs, per truth
— month-only / geography-only (lat,lon) / detection-only / all:
| truth | month-only | geo-only | detection-only | all | summer-share gap |
|---|
| Truth1 | 0.577 | 0.903 | 0.936 | 0.973 | 0.00 |
| Truth2 | 0.567 | 0.877 | 0.845 | 0.932 | 0.00 |
Reads: (1) Season is not a confound — the candidate window is summer-only, so the summer-share of
positives and of negatives is identical (gap = 0) and month-only ranking is near-chance (~0.57). (2) Geography is a
large prior under both truths (0.903 vs 0.877), which already argues it is real fire geography rather than
pure reporting bias — a reporting-only signal would predict the HMS truth far better than the polar truth.
(3) Production's Truth1 AUC lead does not survive the neutral truth. The honest operating-point story is unchanged:
production cannot deliver a high-precision set within CAUTION; the recovery model can.
Claim status: DEFENSIBLE (measured, out-of-fold, two independent truths).
Literature alignment. Verdict: AGREEMENT. Two well-established remote-sensing results are
reproduced here on OLDGABE's own data: (a) a single geostationary quality/confidence flag is a weak within-tier
ranker [1][2], and (b) independent multi-sensor corroboration is the
strongest precision lever for active-fire detection [3]. A learned re-ranker that fuses
detection-time features outperforms a hand-tuned tier threshold, consistent with the broader move from fixed
thresholds to learned confidence in EO detection.
Where we are careful / diverge: AUC and PR-AUC disagree here because production's confidence is
near-constant within CAUTION; we report the operating-point metric (recall at fixed precision) as primary because the
use-case is surfacing a high-precision hidden set, not global ranking. We treat the HMS-heavy Truth1 as
reporting-biased and lead with the polar Truth2.
What this does not claim: neither truth is a fire perimeter; Truth1 (HMS smoke) is spatially broad,
Truth2 (polar overpass) is recall-limited. These are corroboration proxies.
Next research test: add geographic priors and re-validate on the neutral truth (see entry 0002);
extend beyond the summer window before any surfacing.
References: [1] · [2] · [3]
Public data sources: NASA FIRMS (VIIRS/MODIS
active fire) · NOAA GOES-R ABI ·
NIFC / IRWIN & perimeters (US ground truth). Every figure is
reproducible from these public feeds plus the recipe above; no OLDGABE-internal state is required to recompute the
model, only OLDGABE's aggregation of these feeds into per-event evidence.
Statistical reporting: ranking metrics are out-of-fold (5-fold GroupKFold, 0.2° cells) to
remove in-sample optimism; base rates are stated with counts; both a reporting-biased and a reporting-independent
truth are reported side by side rather than a single headline.
entry 0001
Adding geography to the recovery re-ranker — a defensible lift that grows under the neutral truth
Date: 2026-07-08 Status: defensible (measured; gated hidden shadow — NOT surfaced)
The framing. Entry 0001 showed geography alone ranks CAUTION fires surprisingly well. Wildfires
cluster by fuel, terrain and climate, so a location prior should carry real signal — but a location prior can also
just memorise where fires get reported. This entry adds geography to the recovery model and tests, honestly,
whether the lift is real fire behaviour or a reporting/coverage artifact.
BLUF. Adding raw lat/lon to the detection feature set (recovery+geo) is a
defensible improvement over both the detection-only recovery model and production, on
both truths and every metric. The decisive de-confounding result: the geo lift is larger under
the HMS-independent polar truth (+0.081 AUC) than under the reporting-heavy truth (+0.036 AUC). If geography
were merely encoding reporting bias, its benefit would shrink when the truth stops depending on reporting — instead it
grows. So the geographic signal is real fire geography.
What "geo" is (hard data bound). OLDGABE currently has no populated land-cover / fuel /
terrain / climatology layer — the fire-weather cell table is empty, weather stations are a current snapshot
(temporally invalid for historical events), and the raw detection JSON carries no geographic fields. So the only
non-leaky geographic priors available are raw lat/lon + distance-to-known-false-source. A Truth1-only
supplementary that adds an out-of-neighbourhood polar-thermal climatology density (a fuel/fire-proneness
proxy; circular against Truth2, hence Truth1-only) lifts a little further (AUC 0.978, recall@80% 0.771), indicating
headroom if a real independent fuel/climatology layer were ingested.
Method delta from entry 0001. Same populations (n = 202,005; the DB ingested more June/July events
between runs), same non-leaky features, same 5-fold GroupKFold on 0.2° cells, same two truths. New models:
recovery+geo = detection features + lat + lon; geo-only = lat,lon; all = detection + lat +
lon + month. A June↔July temporal-transport split is added as the only distribution shift the data permits.
Result — head-to-head.
| Truth1 (multi-source) | AUC | PR-AUC | recall@80%p |
|---|
| detection-only recovery (deployed) | 0.938 | 0.736 | 0.583 |
| recovery + geo | 0.973 | 0.797 | 0.719 |
| all (+month) | 0.974 | 0.803 | 0.722 |
| geo-only | 0.901 | — | — |
production tier_and_conf | 0.950 | 0.539 | 0.092 |
| Truth2 (neutral polar) | AUC | PR-AUC | recall@80%p |
|---|
| detection-only recovery (deployed) | 0.860 | 0.573 | 0.368 |
| recovery + geo | 0.941 | 0.688 | 0.531 |
| all (+month) | 0.940 | 0.686 | 0.526 |
| geo-only | 0.886 | — | — |
production tier_and_conf | 0.807 | 0.325 | 0.000 |
The geo lift over detection-only is +0.036 AUC on Truth1 and +0.081 AUC on Truth2 — it grows under
the neutral truth. recovery+geo also closes and reverses the one metric where production had led (Truth1 AUC, now
0.973 vs 0.950). Detection and geography are complementary: recovery+geo (0.941) beats both detection-only (0.860)
and geo-only (0.886), so neither is redundant. Month adds nothing (all ≈ recovery+geo).
Result — temporal transport (train one month, test the other).
| train → test | truth | detection-only | recovery+geo | production |
|---|
| Jul → Jun | Truth1 | 0.961 | 0.985 | 0.963 |
| Jul → Jun | Truth2 | 0.876 | 0.955 | 0.834 |
| Jun → Jul | Truth1 | 0.888 | 0.934 | 0.941 |
| Jun → Jul | Truth2 | 0.840 | 0.913 | 0.794 |
recovery+geo beats detection-only in all four cells and production in three of four (it loses only Jun→Jul on
the HMS-confounded Truth1, by 0.007). Under the neutral truth it wins both directions by wide margins, and it
stabilises the hard direction (Jun→Jul detection-only falls to 0.888; recovery+geo holds 0.934). Within
the available window, the geographic prior transports across time and does not hurt.
Deployment status: GATED HIDDEN SHADOW — nothing surfaced. The recovery+geo scores are written to a
hidden additive table event_caution_recovered_geo; a companion canary_meta row carries
surface_allowed=0 and a DO_NOT_SURFACE_* marker file states the block. No OLDGABE map, API,
tile, or user-facing output reads this table, and the daemon does not modify any production tier/score/confidence.
Surfacing is blocked until BOTH: (1) an off-season validation window passes, and
(2) a held-out region validation passes. The geographic prior is precisely the component most exposed
to seasonal and regional shift, and the entire evaluation to date is a two-month summer window (June+July 2026) with
spatial overlap. It must be re-measured on the neutral polar truth in those regimes before any surfacing is considered.
Claim status: DEFENSIBLE (measured, out-of-fold, survives the neutral truth), with an explicit
generalisation bound.
Literature alignment. Verdict: AGREEMENT. Wildfire occurrence is strongly conditioned by static
geography — fuel, topography and climate — which is the basis of published fire-susceptibility / fire-danger mapping
[4][5]; a learned location prior recovering much of that structure is
expected. Our de-confounding design (measure the geographic lift against a truth that does not depend on the reporting
channel) follows the standard caution that fire records carry reporting/observation bias
[6].
Where we are careful / diverge: we deliberately do not ship the geo model, because a
lat/lon prior fit on a single summer season can encode that season's/region's coverage and will not transport
unexamined. We hold it as a hidden shadow behind an explicit off-season + held-out-region gate.
What this does not claim: that the geographic prior generalises across seasons or regions (untested
— no off-season or out-of-region data exists yet); that raw lat/lon is a substitute for a real fuel/terrain
layer (the climatology supplement shows further headroom if one is ingested).
Next research test: extend the window to an off-season month and a held-out region; re-grade
recovery+geo on the neutral polar truth in each; ingest an independent fuel/land-cover/terrain layer (e.g. public
land-cover / elevation) and measure marginal lift over raw lat/lon.
References: [3] · [4] · [5] · [6]
Public data sources: as entry 0001 (NASA FIRMS, NOAA GOES-ABI, NIFC/IRWIN & perimeters).
Reproducible from the public feeds plus the stated features, label rules, model spec and CV protocol.
Statistical reporting: all model numbers are out-of-fold (5-fold GroupKFold, 0.2° cells); the
geographic lift is reported against both a reporting-biased and a reporting-independent truth; the temporal-transport
split is shown in both directions; the summer-only generalisation bound is stated as load-bearing, not a footnote.
entry 0002