Knowable pool (artists enter only from their first release/play/chart/list/NMF), pooled out-of-sample recalibration (Platt vs isotonic; the site uses pooled Platt per channel, isotonic any channel), release-timing baseline, 8-week bootstrap blocks, surge-rule precision/recall vs the lists with a constant grid (docs/ADDCHECK.md); backtests restructured to score all variants with calibration tables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-16 00:34:30 +00:00
co-authored by Claude Opus 5.5
parent 94abfa0255
commit 37b3029e02
12 changed files with 767 additions and 339 deletions
+12
View File
@@ -97,3 +97,15 @@ airplay model rank [--h 4] [--max-h 12] [--top 50] the ranking for h = 1..12 -
- **Recalibration.** The add model's probabilities get a Platt map fitted honestly: an inner model trained on windows closed before the last 6 origins predicts those origins.
- **NMF.** SuperFridayChart artist pages give every NMF placement per artist since 2022 (`nmf_artist_tracks`), including songs DR never played. Track pages give the countries, fetched for placements inside the modelling window first. Before a track page is read, NMF Denmark is recognised by its follower count. `airplay model nmf` compares DR adds after releases on NMF Denmark, on NMF elsewhere and not on NMF, raw and with controls.
- **Bauer.** The site shows each artist's plays per Bauer station over the last 7 days, with songs. Bauer has no lists or history, so Bauer adds join the model once the daily logs cover a few weeks.
## Update 16 Nov 2026: honest backtests
Following the external evaluation (`Airplay-Predictor: implementation evaluation`):
- **Leak test** (`airplay model leakcheck`): rebuilds every feature from a copy of the data cut at the origin week (`Panel::truncated`) and reports any feature that changes. It found the wave leak (the surge rule confirms an add with the following week). Fixed: "weeks since last add" counts only adds up to t − 1, and training windows close a week before the origin. The test now passes for all features.
- **Knowable pool:** an artist enters training, calibration, baselines and test rows only from their first knowable week: first release, DR play, chart entry, DR list, archived play or NMF placement (`Panel::first_known`). 7,219 artists are knowable at the start of the panel, 7,451 by the end, and 485 never.
- **Pre-/post-release:** both backtests report the model with the window's NMF placement blended from the artist's history (pre-release, as on the site) and with it known (post-release).
- **Release-timing baseline:** add rate by release in the window × weeks since the last release; for release events, by weeks since the previous release. It is weak: the models beat it by +0.4 per channel.
- **Recalibration:** pooled out-of-sample Platt and isotonic (`calib.rs`) are compared with the 6-origin Platt. For the release-event model (the site), pooled Platt is best per channel and isotonic for any channel, so the site uses those (`AddModel::fit_events_pooled`), fitted on rolling out-of-sample forecasts over the whole panel. For the window model, the 6-origin Platt is marginally best. All variants still over-predict above about 0.5.
- **Intervals:** block bootstrap over 8-week blocks.
- **Surge rule vs the lists** (`airplay model addcheck --tune`, docs/ADDCHECK.md): on list pairs at most 8 weeks apart, surge has recall 68% and precision ≥ 25% (F1 0.36), against onset-only 55% / 22%. A grid over the constants moves F1 only within noise, so the defaults stay.