Knowable pool (artists enter only from their first release/play/chart/list/NMF), pooled out-of-sample recalibration (Platt vs isotonic; the site uses pooled Platt per channel, isotonic any channel), release-timing baseline, 8-week bootstrap blocks, surge-rule precision/recall vs the lists with a constant grid (docs/ADDCHECK.md); backtests restructured to score all variants with calibration tables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-16 00:34:30 +00:00
co-authored by Claude Opus 5.5
parent 94abfa0255
commit 37b3029e02
12 changed files with 767 additions and 339 deletions
+87 -47
View File
@@ -1,57 +1,97 @@
# Backtest: if the artist releases a new song, will DR add it?
Events: an artist in the panel releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; the new song's NMF Denmark placement is blended from the artist's history. 2302 release events scored, 130 (5.6%) got an add on some DR channel.
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 2140 release events scored, 120 (5.6%) got an add on some DR channel.
| Model | Log-loss, any channel | PR-AUC, any channel | Log-loss, per channel | PR-AUC, per channel |
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.2073 | 0.115 | 0.0666 | 0.059 |
| rate by last-12-weeks level | 0.1838 | 0.215 | 0.0491 | 0.202 |
| event model (pre-release: NMF from history) | 0.1854 | 0.248 | 0.0445 | 0.314 |
| event model (post-release: NMF known) | 0.1814 | 0.280 | 0.0422 | 0.325 |
| rate by in-rotation-now | 0.0666 | 0.061 | 0.2070 | 0.113 |
| rate by last-12-weeks level | 0.0478 | 0.216 | 0.1805 | 0.221 |
| release timing only | 0.0719 | 0.014 | 0.2166 | 0.068 |
| model, calibrated on the last 6 origins (pre-release) | 0.0431 | 0.341 | 0.1881 | 0.243 |
| model, raw (pre-release) | 0.0437 | 0.326 | 0.1860 | 0.252 |
| model + pooled Platt (pre-release) | 0.0425 | 0.337 | 0.1916 | 0.221 |
| model + pooled isotonic (pre-release) | 0.0439 | 0.308 | 0.1871 | 0.243 |
| model, NMF known (post-release) | 0.0413 | 0.346 | 0.1853 | 0.266 |
Skill of the event model, any channel: 0.106 (0.081 to 0.178) vs in-rotation-now, -0.008 (-0.051 to 0.068) vs last-12-weeks level.
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.098 (+0.054 to +0.162) | +0.400 (+0.313 to +0.440) | -0.042 (-0.068 to +0.041) | +0.132 (+0.128 to +0.223) |
| model + pooled Platt (pre-release) | +0.111 (+0.054 to +0.181) | +0.408 (+0.330 to +0.440) | -0.061 (-0.091 to -0.022) | +0.116 (+0.097 to +0.160) |
| model + pooled isotonic (pre-release) | +0.083 (+0.054 to +0.187) | +0.390 (+0.333 to +0.438) | -0.037 (-0.072 to +0.031) | +0.136 (+0.121 to +0.206) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1936 / 0.018 / 0.034 | 1897 / 0.034 / 0.033 | 1844 / 0.027 / 0.027 |
| 0.1–0.2 | 124 / 0.137 / 0.234 | 154 / 0.141 / 0.201 | 124 / 0.143 / 0.202 |
| 0.2–0.3 | 40 / 0.245 / 0.325 | 56 / 0.247 / 0.286 | 90 / 0.251 / 0.200 |
| 0.3–0.4 | 18 / 0.349 / 0.278 | 20 / 0.355 / 0.300 | 40 / 0.350 / 0.250 |
| 0.4–0.5 | 13 / 0.442 / 0.385 | 7 / 0.463 / 0.429 | 23 / 0.449 / 0.435 |
| 0.5–0.6 | 5 / 0.534 / 0.400 | 3 / 0.536 / 0.333 | 8 / 0.571 / 0.500 |
| 0.6–0.7 | 1 / 0.606 / 0.000 | 2 / 0.625 / 0.000 | 3 / 0.637 / 0.333 |
| 0.7–0.8 | 3 / 0.732 / 0.333 | — | 2 / 0.730 / 0.000 |
| 0.8–0.9 | — | 1 / 0.823 / 1.000 | 5 / 0.824 / 0.600 |
| 0.9–1.0 | — | — | 1 / 0.911 / 0.000 |
Calibration, per channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 29258 / 0.004 / 0.007 | 29241 / 0.004 / 0.007 | 29307 / 0.004 / 0.007 |
| 0.1–0.2 | 342 / 0.140 / 0.208 | 347 / 0.143 / 0.182 | 328 / 0.143 / 0.235 |
| 0.2–0.3 | 136 / 0.246 / 0.206 | 136 / 0.252 / 0.287 | 122 / 0.248 / 0.189 |
| 0.3–0.4 | 87 / 0.347 / 0.207 | 102 / 0.335 / 0.118 | 81 / 0.344 / 0.185 |
| 0.4–0.5 | 66 / 0.447 / 0.500 | 44 / 0.450 / 0.500 | 35 / 0.449 / 0.429 |
| 0.5–0.6 | 10 / 0.527 / 0.000 | 28 / 0.538 / 0.714 | 25 / 0.551 / 0.760 |
| 0.6–0.7 | 8 / 0.675 / 0.125 | 5 / 0.655 / 0.400 | 4 / 0.652 / 0.250 |
| 0.7–0.8 | 42 / 0.745 / 0.762 | 42 / 0.757 / 0.714 | 35 / 0.751 / 0.886 |
| 0.8–0.9 | 2 / 0.845 / 1.000 | 6 / 0.817 / 0.500 | 2 / 0.845 / 1.000 |
| 0.9–1.0 | 9 / 0.912 / 1.000 | 9 / 0.912 / 1.000 | 21 / 0.954 / 0.476 |
| Any-channel feature (all events) | Coefficient |
|---|---|
| intercept | -3.665 |
| log DR plays t | +0.557 |
| log DR plays 4w | +0.507 |
| log DR plays 12w | -0.363 |
| log max DR plays 12w | +0.249 |
| in rotation anywhere now | -0.476 |
| station groups in rotation now (/5) | -0.252 |
| current run anywhere (/12) | -1.197 |
| weeks since in rotation anywhere (/12) | -0.514 |
| Track Top-40 score | -0.282 |
| Airplay Top-20 score | -0.313 |
| chart entry recency 6w | +0.713 |
| chart entry recency 26w | +0.477 |
| log Track Top-40 hits | -0.303 |
| log Airplay Top-20 hits | -0.118 |
| Danish | +0.580 |
| group | +0.043 |
| female | -0.064 |
| career years (/50) | +0.433 |
| debut unknown | +0.530 |
| intercept | -3.768 |
| log DR plays t | +0.560 |
| log DR plays 4w | +0.510 |
| log DR plays 12w | -0.409 |
| log max DR plays 12w | +0.318 |
| in rotation anywhere now | -0.474 |
| station groups in rotation now (/5) | -0.249 |
| current run anywhere (/12) | -1.173 |
| weeks since in rotation anywhere (/12) | -0.489 |
| Track Top-40 score | -0.265 |
| Airplay Top-20 score | -0.317 |
| chart entry recency 6w | +0.685 |
| chart entry recency 26w | +0.501 |
| log Track Top-40 hits | -0.291 |
| log Airplay Top-20 hits | -0.120 |
| Danish | +0.548 |
| group | +0.013 |
| female | -0.075 |
| career years (/50) | +0.422 |
| debut unknown | +0.435 |
| Christmas in window | +0.000 |
| discography known | +0.000 |
| release recency 4w | +0.243 |
| release recency 12w | -0.814 |
| log releases last 52w | -0.224 |
| releases in the window | +0.000 |
| releases in window x not in rotation now | +0.476 |
| DR: weeks since last add 13-25 | -0.054 |
| DR: last add 26+ weeks ago | +0.497 |
| DR: no add in the panel yet | -0.684 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.463 |
| discography known | -0.000 |
| release recency 4w | +0.216 |
| release recency 12w | -0.783 |
| log releases last 52w | -0.189 |
| releases in the window | -0.000 |
| releases in window x not in rotation now | +0.474 |
| DR: weeks since last add 13-25 | -0.050 |
| DR: last add 26+ weeks ago | +0.502 |
| DR: no add in the panel yet | -0.677 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.465 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.144 |
| Airplay Top-20: last new song 2+ years ago | -0.107 |
| log archive plays per 100 sampled hours (2012-21) | +0.532 |
| log DR lists with the artist before t (2023-) | +0.108 |
| DR list recency 26w | -0.955 |
| DR lists known before t at all | +0.572 |
| NMF Denmark in the window | +0.866 |
| NMF Denmark top 10 in the window | +0.788 |
| log NMF Denmark placements last 52w | -0.047 |
| share of releases on NMF Denmark (2 years) | -0.354 |
| season index of the window (log, IFPI 2002-24) | +0.372 |
| Airplay Top-20: last new song 2+ years ago | -0.113 |
| log archive plays per 100 sampled hours (2012-21) | +0.515 |
| log DR lists with the artist before t (2023-) | +0.105 |
| DR list recency 26w | -0.984 |
| DR lists known before t at all | +0.593 |
| NMF Denmark in the window | +0.836 |
| NMF Denmark top 10 in the window | +0.682 |
| log NMF Denmark placements last 52w | -0.067 |
| share of releases on NMF Denmark (2 years) | -0.273 |
| season index of the window (log, IFPI 2002-24) | +0.334 |