Knowable pool (artists enter only from their first release/play/chart/list/NMF), pooled out-of-sample recalibration (Platt vs isotonic; the site uses pooled Platt per channel, isotonic any channel), release-timing baseline, 8-week bootstrap blocks, surge-rule precision/recall vs the lists with a constant grid (docs/ADDCHECK.md); backtests restructured to score all variants with calibration tables

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-16 00:34:30 +00:00
co-authored by Claude Opus 5.5
parent 94abfa0255
commit 37b3029e02
12 changed files with 767 additions and 339 deletions
+23
View File
@@ -0,0 +1,23 @@
# Add definitions vs DR's own lists
Truth: artists with a song new on a list vs the previous list of the channel. Recall over all list pairs (24), and recall and precision over pairs at most 8 weeks apart (15). Precision is a lower bound: songs added and dropped between lists, or added off-list, count against it.
| Definition | Recall, all pairs | Recall, close pairs | Precision, close pairs | F1, close pairs | Adds detected | New songs |
|---|---|---|---|---|---|---|
| onset | 64% | 55% | 22% | 0.32 | 351 | 177 |
| surge [0.6, 0.5, 0.5, 0.25] | 77% | 68% | 25% | 0.36 | 420 | 177 |
Surge constants tried (floor × τ, jump × τ, jump × baseline, held next week), best F1 on close pairs:
| Constants | Recall | Precision | F1 |
|---|---|---|---|
| [0.4, 0.3, 1.0, 0.5] | 65% | 26% | 0.369 |
| [0.4, 0.3, 0.5, 0.5] | 68% | 25% | 0.368 |
| [0.6, 0.3, 1.0, 0.5] | 64% | 26% | 0.368 |
| [0.4, 0.3, 0.25, 0.5] | 69% | 25% | 0.367 |
| [0.6, 0.5, 0.25, 0.25] | 70% | 25% | 0.365 |
| [0.6, 0.3, 0.5, 0.5] | 67% | 25% | 0.365 |
| [0.6, 0.5, 0.25, 0.5] | 68% | 25% | 0.365 |
| [0.6, 0.3, 0.25, 0.5] | 69% | 25% | 0.364 |
The grid's best F1 (0.369) is within noise of the current constants (0.36), so the defaults stay. Surge beats onset-only on both recall and precision.
+189 -150
View File
@@ -1,193 +1,232 @@
# Backtest: new adds
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.
## Window starting h = 0 weeks ahead (17 forecast origins)
## Window starting h = 0 weeks ahead (16 forecast origins)
Base rates: per channel 0.43% of artist-channels get an add in the window; on any channel 2.4% of artists.
Base rates: per channel 0.46% of artist-channels get an add in the window; on any channel 2.5% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0255 | 0.032 | 0.1058 | 0.067 |
| rate by last-12-weeks level | 0.0177 | 0.147 | 0.0909 | 0.133 |
| logit (pre-release: NMF from history) | 0.0147 | 0.258 | 0.0800 | 0.251 |
| logit (post-release: NMF known) | 0.0145 | 0.266 | 0.0799 | 0.259 |
| rate by in-rotation-now | 0.0271 | 0.032 | 0.1111 | 0.069 |
| rate by last-12-weeks level | 0.0188 | 0.148 | 0.0956 | 0.134 |
| release timing only | 0.0274 | 0.010 | 0.1144 | 0.040 |
| model, calibrated on the last 6 origins (pre-release) | 0.0156 | 0.263 | 0.0837 | 0.254 |
| model, raw (pre-release) | 0.0155 | 0.263 | 0.0854 | 0.246 |
| model + pooled Platt (pre-release) | 0.0156 | 0.262 | 0.0855 | 0.246 |
| model + pooled isotonic (pre-release) | 0.0157 | 0.254 | 0.0859 | 0.244 |
| model, NMF known (post-release) | 0.0153 | 0.270 | 0.0836 | 0.263 |
Skill of the logit (90% block-bootstrap interval over origins):
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.424 (0.413 to 0.437) | 0.171 (0.168 to 0.178) |
| any channel | 0.244 (0.234 to 0.257) | 0.120 (0.115 to 0.127) |
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.174 (+0.167 to +0.176) | +0.432 (+0.412 to +0.445) | +0.124 (+0.123 to +0.133) | +0.269 (+0.242 to +0.289) |
| model + pooled Platt (pre-release) | +0.172 (+0.164 to +0.172) | +0.431 (+0.409 to +0.444) | +0.105 (+0.091 to +0.112) | +0.253 (+0.213 to +0.272) |
| model + pooled isotonic (pre-release) | +0.168 (+0.160 to +0.168) | +0.428 (+0.405 to +0.444) | +0.102 (+0.086 to +0.112) | +0.250 (+0.209 to +0.272) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 112229 / 0.010 / 0.013 | 111004 / 0.012 / 0.012 | 111041 / 0.011 / 0.012 |
| 0.1–0.2 | 2985 / 0.142 / 0.196 | 3319 / 0.142 / 0.155 | 2508 / 0.146 / 0.149 |
| 0.2–0.3 | 1360 / 0.245 / 0.263 | 1636 / 0.245 / 0.249 | 2058 / 0.249 / 0.218 |
| 0.3–0.4 | 799 / 0.346 / 0.328 | 998 / 0.347 / 0.305 | 1765 / 0.339 / 0.301 |
| 0.4–0.5 | 450 / 0.447 / 0.360 | 626 / 0.445 / 0.321 | 489 / 0.442 / 0.309 |
| 0.5–0.6 | 251 / 0.546 / 0.347 | 355 / 0.548 / 0.293 | 247 / 0.546 / 0.352 |
| 0.6–0.7 | 162 / 0.647 / 0.407 | 215 / 0.646 / 0.344 | 152 / 0.646 / 0.421 |
| 0.7–0.8 | 93 / 0.746 / 0.505 | 149 / 0.740 / 0.483 | 73 / 0.740 / 0.479 |
| 0.8–0.9 | 52 / 0.841 / 0.596 | 74 / 0.837 / 0.554 | 43 / 0.837 / 0.558 |
| 0.9–1.0 | 11 / 0.924 / 0.182 | 16 / 0.924 / 0.438 | 16 / 0.932 / 0.500 |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1868823 | 0.002 | 0.002 |
| 0.1–0.2 | 9096 | 0.142 | 0.160 |
| 0.2–0.3 | 3991 | 0.247 | 0.230 |
| 0.3–0.4 | 2858 | 0.344 | 0.215 |
| 0.4–0.5 | 1615 | 0.448 | 0.310 |
| 0.5–0.6 | 1081 | 0.544 | 0.457 |
| 0.6–0.7 | 728 | 0.646 | 0.475 |
| 0.7–0.8 | 429 | 0.749 | 0.441 |
| 0.8–0.9 | 104 | 0.834 | 0.346 |
| 0.9–1.0 | 43 | 0.917 | 0.930 |
| 0.0–0.1 | 1638964 / 0.002 / 0.002 | 1639990 / 0.002 / 0.002 | 1637740 / 0.002 / 0.002 |
| 0.1–0.2 | 8541 / 0.142 / 0.165 | 7975 / 0.141 / 0.176 | 9301 / 0.135 / 0.149 |
| 0.2–0.3 | 3740 / 0.248 / 0.229 | 3586 / 0.247 / 0.231 | 4805 / 0.251 / 0.229 |
| 0.3–0.4 | 2586 / 0.344 / 0.216 | 2464 / 0.345 / 0.211 | 3432 / 0.338 / 0.297 |
| 0.4–0.5 | 1570 / 0.449 / 0.336 | 1465 / 0.449 / 0.374 | 939 / 0.448 / 0.376 |
| 0.5–0.6 | 984 / 0.545 / 0.450 | 966 / 0.547 / 0.443 | 588 / 0.548 / 0.371 |
| 0.6–0.7 | 621 / 0.645 / 0.473 | 549 / 0.643 / 0.461 | 328 / 0.644 / 0.424 |
| 0.7–0.8 | 351 / 0.751 / 0.464 | 358 / 0.751 / 0.455 | 207 / 0.756 / 0.560 |
| 0.8–0.9 | 91 / 0.835 / 0.352 | 90 / 0.834 / 0.489 | 148 / 0.853 / 0.507 |
| 0.9–1.0 | 40 / 0.914 / 0.950 | 45 / 0.928 / 0.689 | — |
## Window starting h = 4 weeks ahead (13 forecast origins)
## Window starting h = 4 weeks ahead (12 forecast origins)
Base rates: per channel 0.44% of artist-channels get an add in the window; on any channel 2.4% of artists.
Base rates: per channel 0.47% of artist-channels get an add in the window; on any channel 2.5% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0263 | 0.032 | 0.1051 | 0.083 |
| rate by last-12-weeks level | 0.0193 | 0.114 | 0.0937 | 0.136 |
| logit (pre-release: NMF from history) | 0.0176 | 0.166 | 0.0857 | 0.208 |
| logit (post-release: NMF known) | 0.0172 | 0.175 | 0.0856 | 0.218 |
| rate by in-rotation-now | 0.0277 | 0.033 | 0.1102 | 0.084 |
| rate by last-12-weeks level | 0.0204 | 0.117 | 0.0984 | 0.137 |
| release timing only | 0.0286 | 0.010 | 0.1146 | 0.043 |
| model, calibrated on the last 6 origins (pre-release) | 0.0186 | 0.166 | 0.0898 | 0.208 |
| model, raw (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model + pooled Platt (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model + pooled isotonic (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model, NMF known (post-release) | 0.0183 | 0.174 | 0.0896 | 0.219 |
Skill of the logit (90% block-bootstrap interval over origins):
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.330 (0.315 to 0.337) | 0.091 (0.065 to 0.106) |
| any channel | 0.185 (0.174 to 0.194) | 0.086 (0.081 to 0.090) |
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.088 (+0.084 to +0.095) | +0.217 (+0.207 to +0.221) |
| model + pooled Platt (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
| model + pooled isotonic (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 84532 / 0.011 / 0.015 | 83207 / 0.013 / 0.013 | 83207 / 0.013 / 0.013 |
| 0.1–0.2 | 2353 / 0.140 / 0.178 | 2764 / 0.142 / 0.156 | 2764 / 0.142 / 0.156 |
| 0.2–0.3 | 904 / 0.242 / 0.304 | 1271 / 0.245 / 0.238 | 1271 / 0.245 / 0.238 |
| 0.3–0.4 | 479 / 0.347 / 0.311 | 717 / 0.346 / 0.262 | 717 / 0.346 / 0.262 |
| 0.4–0.5 | 221 / 0.446 / 0.385 | 359 / 0.445 / 0.304 | 359 / 0.445 / 0.304 |
| 0.5–0.6 | 124 / 0.544 / 0.306 | 188 / 0.547 / 0.372 | 188 / 0.547 / 0.372 |
| 0.6–0.7 | 83 / 0.640 / 0.325 | 142 / 0.641 / 0.345 | 142 / 0.641 / 0.345 |
| 0.7–0.8 | 30 / 0.750 / 0.333 | 66 / 0.749 / 0.273 | 66 / 0.749 / 0.273 |
| 0.8–0.9 | 4 / 0.830 / 0.000 | 16 / 0.835 / 0.312 | 16 / 0.835 / 0.312 |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1428760 | 0.002 | 0.003 |
| 0.1–0.2 | 7389 | 0.141 | 0.129 |
| 0.2–0.3 | 3960 | 0.247 | 0.186 |
| 0.3–0.4 | 2024 | 0.344 | 0.199 |
| 0.4–0.5 | 1134 | 0.444 | 0.283 |
| 0.5–0.6 | 604 | 0.546 | 0.224 |
| 0.6–0.7 | 394 | 0.643 | 0.459 |
| 0.7–0.8 | 83 | 0.732 | 0.277 |
| 0.8–0.9 | 3 | 0.833 | 1.000 |
| 0.9–1.0 | 1 | 0.907 | 1.000 |
| 0.0–0.1 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 |
| 0.1–0.2 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 |
| 0.2–0.3 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 |
| 0.3–0.4 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 |
| 0.4–0.5 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 |
| 0.5–0.6 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 |
| 0.6–0.7 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 |
| 0.7–0.8 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 |
| 0.8–0.9 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 |
| 0.9–1.0 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 |
## Coefficients, window starting now (h = 0), all closed windows
| Any-channel feature | Coefficient |
|---|---|
| intercept | -4.213 |
| log DR plays t | +0.883 |
| intercept | -4.141 |
| log DR plays t | +0.882 |
| log DR plays 4w | +0.253 |
| log DR plays 12w | -0.263 |
| log max DR plays 12w | +0.594 |
| in rotation anywhere now | -1.148 |
| station groups in rotation now (/5) | +0.297 |
| current run anywhere (/12) | -1.212 |
| weeks since in rotation anywhere (/12) | -0.749 |
| log DR plays 12w | -0.255 |
| log max DR plays 12w | +0.583 |
| in rotation anywhere now | -1.155 |
| station groups in rotation now (/5) | +0.320 |
| current run anywhere (/12) | -1.215 |
| weeks since in rotation anywhere (/12) | -0.748 |
| Track Top-40 score | +0.427 |
| Airplay Top-20 score | -1.332 |
| chart entry recency 6w | +1.210 |
| chart entry recency 26w | -0.653 |
| log Track Top-40 hits | -0.200 |
| log Airplay Top-20 hits | -0.069 |
| Danish | +0.462 |
| group | -0.240 |
| female | -0.102 |
| career years (/50) | +0.210 |
| debut unknown | +0.497 |
| Airplay Top-20 score | -1.336 |
| chart entry recency 6w | +1.203 |
| chart entry recency 26w | -0.646 |
| log Track Top-40 hits | -0.201 |
| log Airplay Top-20 hits | -0.070 |
| Danish | +0.470 |
| group | -0.253 |
| female | -0.097 |
| career years (/50) | +0.222 |
| debut unknown | +0.489 |
| Christmas in window | +0.000 |
| discography known | -0.325 |
| release recency 4w | +0.440 |
| release recency 12w | +0.670 |
| log releases last 52w | -0.188 |
| releases in the window | -0.218 |
| releases in window x not in rotation now | +0.854 |
| discography known | -0.392 |
| release recency 4w | +0.435 |
| release recency 12w | +0.680 |
| log releases last 52w | -0.186 |
| releases in the window | -0.216 |
| releases in window x not in rotation now | +0.836 |
| DR: weeks since last add 13-25 | +0.095 |
| DR: last add 26+ weeks ago | -0.642 |
| DR: no add in the panel yet | -0.847 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.279 |
| Airplay Top-20: 0.5-1.25x own usual gap | +0.231 |
| Airplay Top-20: last new song 2+ years ago | -0.529 |
| log archive plays per 100 sampled hours (2012-21) | +0.288 |
| DR: no add in the panel yet | -0.845 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.278 |
| Airplay Top-20: 0.5-1.25x own usual gap | +0.229 |
| Airplay Top-20: last new song 2+ years ago | -0.527 |
| log archive plays per 100 sampled hours (2012-21) | +0.289 |
| log DR lists with the artist before t (2023-) | +0.209 |
| DR list recency 26w | -0.929 |
| DR lists known before t at all | +0.040 |
| NMF Denmark in the window | +0.941 |
| NMF Denmark top 10 in the window | +0.416 |
| log NMF Denmark placements last 52w | -0.071 |
| share of releases on NMF Denmark (2 years) | -0.415 |
| season index of the window (log, IFPI 2002-24) | +0.403 |
| DR list recency 26w | -0.924 |
| DR lists known before t at all | +0.039 |
| NMF Denmark in the window | +0.940 |
| NMF Denmark top 10 in the window | +0.408 |
| log NMF Denmark placements last 52w | -0.068 |
| share of releases on NMF Denmark (2 years) | -0.417 |
| season index of the window (log, IFPI 2002-24) | +0.412 |
| Per-channel feature | Coefficient |
|---|---|
| intercept | -7.101 |
| log plays t | +1.122 |
| log plays t-1 | +0.142 |
| intercept | -7.117 |
| log plays t | +1.106 |
| log plays t-1 | +0.141 |
| log mean plays 4w | +0.113 |
| log mean plays 12w | -0.862 |
| log max plays 12w | +1.014 |
| log other channels t | +0.332 |
| log other channels 4w | -0.100 |
| log P3 t | +0.085 |
| weeks since played (/12) | -0.437 |
| Track Top-40 score | -0.127 |
| Airplay Top-20 score | -1.373 |
| chart entry recency 6w | +0.625 |
| chart entry recency 26w | -0.512 |
| log Track Top-40 hits | +0.257 |
| log Airplay Top-20 hits | -0.395 |
| Danish | +0.162 |
| group | -0.629 |
| female | +0.127 |
| career years (/50) | -1.500 |
| debut unknown | -0.289 |
| log mean plays 12w | -0.871 |
| log max plays 12w | +1.022 |
| log other channels t | +0.328 |
| log other channels 4w | -0.095 |
| log P3 t | +0.087 |
| weeks since played (/12) | -0.509 |
| Track Top-40 score | -0.132 |
| Airplay Top-20 score | -1.376 |
| chart entry recency 6w | +0.616 |
| chart entry recency 26w | -0.506 |
| log Track Top-40 hits | +0.258 |
| log Airplay Top-20 hits | -0.398 |
| Danish | +0.171 |
| group | -0.636 |
| female | +0.136 |
| career years (/50) | -1.498 |
| debut unknown | -0.333 |
| Christmas target week | +0.000 |
| Danish x P3 | +0.072 |
| Danish x P4 | +0.085 |
| Danish x P6 | +0.154 |
| ch p4kbh | +1.100 |
| ch p4sjaelland | +0.968 |
| ch p4fyn | +0.948 |
| ch p4syd | +0.976 |
| ch p4trekanten | +0.992 |
| ch p4vest | +1.078 |
| ch p4aarhus | +0.934 |
| ch p4nord | +1.002 |
| ch p4bornholm | +0.986 |
| ch p4esbjerg | +0.968 |
| Danish x P3 | +0.064 |
| Danish x P4 | +0.058 |
| Danish x P6 | +0.111 |
| ch p4kbh | +1.094 |
| ch p4sjaelland | +0.966 |
| ch p4fyn | +0.946 |
| ch p4syd | +0.975 |
| ch p4trekanten | +0.985 |
| ch p4vest | +1.083 |
| ch p4aarhus | +0.944 |
| ch p4nord | +1.006 |
| ch p4bornholm | +0.995 |
| ch p4esbjerg | +0.972 |
| ch p5 | +2.081 |
| ch p6beat | +1.295 |
| ch p8jazz | +2.330 |
| in rotation on channel now | +0.358 |
| in rotation anywhere now | -0.338 |
| station groups in rotation now (/5) | +0.043 |
| run on channel = 1 week | +0.663 |
| run = 2 weeks | +0.441 |
| run = 3-5 weeks | -0.055 |
| run >= 6 weeks | -0.691 |
| near threshold (>= half of tau, not in rotation) | +1.376 |
| plays / tau (capped 3, /3) | -3.407 |
| weeks since in rotation on channel (/12) | -0.300 |
| discography known | -0.069 |
| release recency 4w | +1.033 |
| release recency 12w | -0.008 |
| log releases last 52w | -0.003 |
| releases in the window | -0.143 |
| releases in window x not in rotation now | +0.782 |
| DR: weeks since last add 13-25 | +0.107 |
| DR: last add 26+ weeks ago | -0.085 |
| DR: no add in the panel yet | -0.486 |
| ch p6beat | +1.296 |
| ch p8jazz | +2.312 |
| in rotation on channel now | +0.350 |
| in rotation anywhere now | -0.334 |
| station groups in rotation now (/5) | +0.041 |
| run on channel = 1 week | +0.660 |
| run = 2 weeks | +0.439 |
| run = 3-5 weeks | -0.056 |
| run >= 6 weeks | -0.693 |
| near threshold (>= half of tau, not in rotation) | +1.381 |
| plays / tau (capped 3, /3) | -3.363 |
| weeks since in rotation on channel (/12) | -0.302 |
| discography known | -0.052 |
| release recency 4w | +1.017 |
| release recency 12w | +0.014 |
| log releases last 52w | +0.009 |
| releases in the window | -0.136 |
| releases in window x not in rotation now | +0.755 |
| DR: weeks since last add 13-25 | +0.106 |
| DR: last add 26+ weeks ago | -0.081 |
| DR: no add in the panel yet | -0.483 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.194 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.009 |
| Airplay Top-20: last new song 2+ years ago | -0.417 |
| log archive plays per 100 sampled hours (2012-21) | +0.348 |
| log DR lists with the artist before t (2023-) | +0.028 |
| DR list recency 26w | -0.444 |
| DR lists known before t at all | +0.055 |
| NMF Denmark in the window | +0.805 |
| NMF Denmark top 10 in the window | +0.579 |
| log NMF Denmark placements last 52w | -0.077 |
| share of releases on NMF Denmark (2 years) | +0.166 |
| season index of the window (log, IFPI 2002-24) | +1.240 |
| share of the artist's DR plays on this station (before t) | +0.963 |
| log DR plays on this station (before t) | +0.049 |
| never played on this station (before t) | -2.888 |
| genre fit for this station | +0.108 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.013 |
| Airplay Top-20: last new song 2+ years ago | -0.418 |
| log archive plays per 100 sampled hours (2012-21) | +0.347 |
| log DR lists with the artist before t (2023-) | +0.023 |
| DR list recency 26w | -0.442 |
| DR lists known before t at all | +0.066 |
| NMF Denmark in the window | +0.779 |
| NMF Denmark top 10 in the window | +0.580 |
| log NMF Denmark placements last 52w | -0.089 |
| share of releases on NMF Denmark (2 years) | +0.212 |
| season index of the window (log, IFPI 2002-24) | +1.211 |
| share of the artist's DR plays on this station (before t) | +0.986 |
| log DR plays on this station (before t) | +0.050 |
| never played on this station (before t) | -2.825 |
| genre fit for this station | +0.091 |
+87 -47
View File
@@ -1,57 +1,97 @@
# Backtest: if the artist releases a new song, will DR add it?
Events: an artist in the panel releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; the new song's NMF Denmark placement is blended from the artist's history. 2302 release events scored, 130 (5.6%) got an add on some DR channel.
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 2140 release events scored, 120 (5.6%) got an add on some DR channel.
| Model | Log-loss, any channel | PR-AUC, any channel | Log-loss, per channel | PR-AUC, per channel |
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.2073 | 0.115 | 0.0666 | 0.059 |
| rate by last-12-weeks level | 0.1838 | 0.215 | 0.0491 | 0.202 |
| event model (pre-release: NMF from history) | 0.1854 | 0.248 | 0.0445 | 0.314 |
| event model (post-release: NMF known) | 0.1814 | 0.280 | 0.0422 | 0.325 |
| rate by in-rotation-now | 0.0666 | 0.061 | 0.2070 | 0.113 |
| rate by last-12-weeks level | 0.0478 | 0.216 | 0.1805 | 0.221 |
| release timing only | 0.0719 | 0.014 | 0.2166 | 0.068 |
| model, calibrated on the last 6 origins (pre-release) | 0.0431 | 0.341 | 0.1881 | 0.243 |
| model, raw (pre-release) | 0.0437 | 0.326 | 0.1860 | 0.252 |
| model + pooled Platt (pre-release) | 0.0425 | 0.337 | 0.1916 | 0.221 |
| model + pooled isotonic (pre-release) | 0.0439 | 0.308 | 0.1871 | 0.243 |
| model, NMF known (post-release) | 0.0413 | 0.346 | 0.1853 | 0.266 |
Skill of the event model, any channel: 0.106 (0.081 to 0.178) vs in-rotation-now, -0.008 (-0.051 to 0.068) vs last-12-weeks level.
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.098 (+0.054 to +0.162) | +0.400 (+0.313 to +0.440) | -0.042 (-0.068 to +0.041) | +0.132 (+0.128 to +0.223) |
| model + pooled Platt (pre-release) | +0.111 (+0.054 to +0.181) | +0.408 (+0.330 to +0.440) | -0.061 (-0.091 to -0.022) | +0.116 (+0.097 to +0.160) |
| model + pooled isotonic (pre-release) | +0.083 (+0.054 to +0.187) | +0.390 (+0.333 to +0.438) | -0.037 (-0.072 to +0.031) | +0.136 (+0.121 to +0.206) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1936 / 0.018 / 0.034 | 1897 / 0.034 / 0.033 | 1844 / 0.027 / 0.027 |
| 0.1–0.2 | 124 / 0.137 / 0.234 | 154 / 0.141 / 0.201 | 124 / 0.143 / 0.202 |
| 0.2–0.3 | 40 / 0.245 / 0.325 | 56 / 0.247 / 0.286 | 90 / 0.251 / 0.200 |
| 0.3–0.4 | 18 / 0.349 / 0.278 | 20 / 0.355 / 0.300 | 40 / 0.350 / 0.250 |
| 0.4–0.5 | 13 / 0.442 / 0.385 | 7 / 0.463 / 0.429 | 23 / 0.449 / 0.435 |
| 0.5–0.6 | 5 / 0.534 / 0.400 | 3 / 0.536 / 0.333 | 8 / 0.571 / 0.500 |
| 0.6–0.7 | 1 / 0.606 / 0.000 | 2 / 0.625 / 0.000 | 3 / 0.637 / 0.333 |
| 0.7–0.8 | 3 / 0.732 / 0.333 | — | 2 / 0.730 / 0.000 |
| 0.8–0.9 | — | 1 / 0.823 / 1.000 | 5 / 0.824 / 0.600 |
| 0.9–1.0 | — | — | 1 / 0.911 / 0.000 |
Calibration, per channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 29258 / 0.004 / 0.007 | 29241 / 0.004 / 0.007 | 29307 / 0.004 / 0.007 |
| 0.1–0.2 | 342 / 0.140 / 0.208 | 347 / 0.143 / 0.182 | 328 / 0.143 / 0.235 |
| 0.2–0.3 | 136 / 0.246 / 0.206 | 136 / 0.252 / 0.287 | 122 / 0.248 / 0.189 |
| 0.3–0.4 | 87 / 0.347 / 0.207 | 102 / 0.335 / 0.118 | 81 / 0.344 / 0.185 |
| 0.4–0.5 | 66 / 0.447 / 0.500 | 44 / 0.450 / 0.500 | 35 / 0.449 / 0.429 |
| 0.5–0.6 | 10 / 0.527 / 0.000 | 28 / 0.538 / 0.714 | 25 / 0.551 / 0.760 |
| 0.6–0.7 | 8 / 0.675 / 0.125 | 5 / 0.655 / 0.400 | 4 / 0.652 / 0.250 |
| 0.7–0.8 | 42 / 0.745 / 0.762 | 42 / 0.757 / 0.714 | 35 / 0.751 / 0.886 |
| 0.8–0.9 | 2 / 0.845 / 1.000 | 6 / 0.817 / 0.500 | 2 / 0.845 / 1.000 |
| 0.9–1.0 | 9 / 0.912 / 1.000 | 9 / 0.912 / 1.000 | 21 / 0.954 / 0.476 |
| Any-channel feature (all events) | Coefficient |
|---|---|
| intercept | -3.665 |
| log DR plays t | +0.557 |
| log DR plays 4w | +0.507 |
| log DR plays 12w | -0.363 |
| log max DR plays 12w | +0.249 |
| in rotation anywhere now | -0.476 |
| station groups in rotation now (/5) | -0.252 |
| current run anywhere (/12) | -1.197 |
| weeks since in rotation anywhere (/12) | -0.514 |
| Track Top-40 score | -0.282 |
| Airplay Top-20 score | -0.313 |
| chart entry recency 6w | +0.713 |
| chart entry recency 26w | +0.477 |
| log Track Top-40 hits | -0.303 |
| log Airplay Top-20 hits | -0.118 |
| Danish | +0.580 |
| group | +0.043 |
| female | -0.064 |
| career years (/50) | +0.433 |
| debut unknown | +0.530 |
| intercept | -3.768 |
| log DR plays t | +0.560 |
| log DR plays 4w | +0.510 |
| log DR plays 12w | -0.409 |
| log max DR plays 12w | +0.318 |
| in rotation anywhere now | -0.474 |
| station groups in rotation now (/5) | -0.249 |
| current run anywhere (/12) | -1.173 |
| weeks since in rotation anywhere (/12) | -0.489 |
| Track Top-40 score | -0.265 |
| Airplay Top-20 score | -0.317 |
| chart entry recency 6w | +0.685 |
| chart entry recency 26w | +0.501 |
| log Track Top-40 hits | -0.291 |
| log Airplay Top-20 hits | -0.120 |
| Danish | +0.548 |
| group | +0.013 |
| female | -0.075 |
| career years (/50) | +0.422 |
| debut unknown | +0.435 |
| Christmas in window | +0.000 |
| discography known | +0.000 |
| release recency 4w | +0.243 |
| release recency 12w | -0.814 |
| log releases last 52w | -0.224 |
| releases in the window | +0.000 |
| releases in window x not in rotation now | +0.476 |
| DR: weeks since last add 13-25 | -0.054 |
| DR: last add 26+ weeks ago | +0.497 |
| DR: no add in the panel yet | -0.684 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.463 |
| discography known | -0.000 |
| release recency 4w | +0.216 |
| release recency 12w | -0.783 |
| log releases last 52w | -0.189 |
| releases in the window | -0.000 |
| releases in window x not in rotation now | +0.474 |
| DR: weeks since last add 13-25 | -0.050 |
| DR: last add 26+ weeks ago | +0.502 |
| DR: no add in the panel yet | -0.677 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.465 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.144 |
| Airplay Top-20: last new song 2+ years ago | -0.107 |
| log archive plays per 100 sampled hours (2012-21) | +0.532 |
| log DR lists with the artist before t (2023-) | +0.108 |
| DR list recency 26w | -0.955 |
| DR lists known before t at all | +0.572 |
| NMF Denmark in the window | +0.866 |
| NMF Denmark top 10 in the window | +0.788 |
| log NMF Denmark placements last 52w | -0.047 |
| share of releases on NMF Denmark (2 years) | -0.354 |
| season index of the window (log, IFPI 2002-24) | +0.372 |
| Airplay Top-20: last new song 2+ years ago | -0.113 |
| log archive plays per 100 sampled hours (2012-21) | +0.515 |
| log DR lists with the artist before t (2023-) | +0.105 |
| DR list recency 26w | -0.984 |
| DR lists known before t at all | +0.593 |
| NMF Denmark in the window | +0.836 |
| NMF Denmark top 10 in the window | +0.682 |
| log NMF Denmark placements last 52w | -0.067 |
| share of releases on NMF Denmark (2 years) | -0.273 |
| season index of the window (log, IFPI 2002-24) | +0.334 |
+12
View File
@@ -97,3 +97,15 @@ airplay model rank [--h 4] [--max-h 12] [--top 50] the ranking for h = 1..12 -
- **Recalibration.** The add model's probabilities get a Platt map fitted honestly: an inner model trained on windows closed before the last 6 origins predicts those origins.
- **NMF.** SuperFridayChart artist pages give every NMF placement per artist since 2022 (`nmf_artist_tracks`), including songs DR never played. Track pages give the countries, fetched for placements inside the modelling window first. Before a track page is read, NMF Denmark is recognised by its follower count. `airplay model nmf` compares DR adds after releases on NMF Denmark, on NMF elsewhere and not on NMF, raw and with controls.
- **Bauer.** The site shows each artist's plays per Bauer station over the last 7 days, with songs. Bauer has no lists or history, so Bauer adds join the model once the daily logs cover a few weeks.
## Update 16 Nov 2026: honest backtests
Following the external evaluation (`Airplay-Predictor: implementation evaluation`):
- **Leak test** (`airplay model leakcheck`): rebuilds every feature from a copy of the data cut at the origin week (`Panel::truncated`) and reports any feature that changes. It found the wave leak (the surge rule confirms an add with the following week). Fixed: "weeks since last add" counts only adds up to t − 1, and training windows close a week before the origin. The test now passes for all features.
- **Knowable pool:** an artist enters training, calibration, baselines and test rows only from their first knowable week: first release, DR play, chart entry, DR list, archived play or NMF placement (`Panel::first_known`). 7,219 artists are knowable at the start of the panel, 7,451 by the end, and 485 never.
- **Pre-/post-release:** both backtests report the model with the window's NMF placement blended from the artist's history (pre-release, as on the site) and with it known (post-release).
- **Release-timing baseline:** add rate by release in the window × weeks since the last release; for release events, by weeks since the previous release. It is weak: the models beat it by +0.4 per channel.
- **Recalibration:** pooled out-of-sample Platt and isotonic (`calib.rs`) are compared with the 6-origin Platt. For the release-event model (the site), pooled Platt is best per channel and isotonic for any channel, so the site uses those (`AddModel::fit_events_pooled`), fitted on rolling out-of-sample forecasts over the whole panel. For the window model, the 6-origin Platt is marginally best. All variants still over-predict above about 0.5.
- **Intervals:** block bootstrap over 8-week blocks.
- **Surge rule vs the lists** (`airplay model addcheck --tune`, docs/ADDCHECK.md): on list pairs at most 8 weeks apart, surge has recall 68% and precision ≥ 25% (F1 0.36), against onset-only 55% / 22%. A grid over the constants moves F1 only within noise, so the defaults stay.