Research · Calibration

Model performance
& validation.

All metrics on this page are read directly from the daily pipeline output. Spatial backtests use county-level GroupKFold holdout; the persistence baseline gives a lower bound on skill. The current public forecast covers 252 of 341 California beaches (74%).

Registry winner

xgb-undersample-ensemble-curated-v0

This is the model that won the promotion gate — not necessarily the one that computed today's forecast. A serve-time router splits traffic by how old each beach's last lab sample is: beaches sampled within 3 days are scored by the ensemble, beaches 5+ days stale by the offset model (per-beach baseline + staleness augmentation), with a blend in between.

Public release eligible

Discrimination & calibration scores.

Production AUCPR

0.8094

holdout test set

Production Brier

0.0570

lower is better

Temporal val AUCPR

0.8068

time-blocked CV

Temporal val Brier

0.0956

time-blocked CV

Sensitivity @ 87% spec

83.7%

Searcy 2018 benchmark

Regime note— these holdout scores are measured on days a lab sample was taken (fresh history). The public forecast is issued daily, mostly on days whose last lab result is already several days old — a harder regime. See the methodology for how served-day performance is tracked.

AUCPR baseline— the persistence model (yesterday's result as today's prediction) scores 0.5015 at the county holdout level and 0.8241 at the beach holdout level in the current pipeline snapshot. The production model must clear that margin before promotion. These are read live from the daily pipeline output — they were previously hardcoded at 0.172 / 0.241, which understated the baseline by roughly 3× and so overstated the model's margin over it.

Agreement with active advisories.

Each day the served forecast is checked against the beaches under an active health advisory or closure — the operational ground truth. A false negative is an advised beach the forecast did not flag; those are tracked with priority because they are the unsafe direction.

Agreement rate

100.0%

forecast vs advisory

False negatives

missed advisories

False positives

over-warnings

Active advisories

risk threshold 0.2

audited 2026-09-15

Performance on unseen geography.

County GroupKFold holdout: each fold withholds all beaches from one county. Tests generalisation to locations not seen during training.

ModelAUCPRBrierFoldsRowsBase rate

spatial beach persistence0.82410.1582151224257.8%

spatial county persistence0.50150.138266956620.1%

spatial beach xgb undersample ensemble0.96120.0936151224257.8%

spatial county xgb undersample ensemble0.74770.098566956620.1%

spatial beach hist gbm0.95970.0966151224257.8%

spatial county hist gbm0.66670.123166956620.1%

spatial beach hist gbm positive persistence guard0.85340.1348151224257.8%

spatial county hist gbm positive persistence guard0.60330.127066956620.1%

spatial beach hist gbm persistence blend0.95770.1071151224257.8%

spatial county hist gbm persistence blend0.70480.100066956620.1%

spatial beach xgb undersample offset0.92860.1944151224257.8%

spatial county xgb undersample offset0.61880.121766956620.1%

Models evaluated each CI run.

All candidates must clear the block-bootstrap AUCPR gate to be eligible for production. The temporal validation winner becomes production if it also clears the spatial backtest promotion policy.

logistic-curated-v0logistic-coastal-cells-curated-v0logistic-hierarchical-curated-v0hist-gbm-curated-v0hist-gbm-positive-persistence-guard-curated-v0xgb-undersample-ensemble-curated-v0 ★stacked-ensemble-curated-v0

Full evaluation protocol

Technical Methodology

Spatial CV protocol, promotion gates, and feature rationale.

Read methodology →