Files
Airplay-Predictor/docs/BACKTEST.md
T

5.5 KiB
Raw Blame History

Backtest: DR artist × channel × week

Panel: 2329 artists × 14 channels × 51 complete weeks (2025-11-17 to 2026-11-02). Forecast origins from week 27; regressions refitted every 1 origin(s).

"In rotation" = at least τ plays in the week: p3 7, p4kbh 5, p4sjaelland 5, p4fyn 5, p4syd 5, p4trekanten 5, p4vest 5, p4aarhus 5, p4nord 5, p4bornholm 5, p4esbjerg 5, p5 4, p6beat 5, p8jazz 3.

h = 1 week(s) ahead (24 forecast weeks)

Model Log-loss, all Log-loss, adds Brier, adds PR-AUC, adds MAE plays Spearman NDCG@50
persistence 0.0683 0.0488 0.0085 0.013 0.28 0.829 0.967
popularity 0.0470 0.0287 0.0080 0.256 0.35 0.864 0.888
kalman 0.0385 0.0256 0.0065 0.377 0.32 0.874 0.957
nb 0.0389 0.0260 0.0066 0.355 0.28 0.859 0.965
logit 0.0363 0.0241 0.0064 0.367 0.28 0.859 0.965

Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):

Model vs persistence vs popularity
kalman 0.475 (0.451 to 0.496) 0.109 (0.063 to 0.141)
nb 0.466 (0.445 to 0.486) 0.094 (0.051 to 0.123)
logit 0.505 (0.489 to 0.520) 0.160 (0.129 to 0.184)

Calibration of the NB's add probabilities (rows not in rotation at the origin):

Predicted Rows Mean predicted Observed
0.0–0.1 716676 0.003 0.002
0.1–0.2 15028 0.143 0.088
0.2–0.3 6884 0.244 0.219
0.3–0.4 3000 0.343 0.391
0.4–0.5 1355 0.441 0.578
0.5–0.6 521 0.540 0.551
0.6–0.7 197 0.650 0.482
0.7–0.8 108 0.748 0.509
0.8–0.9 95 0.847 0.211
0.9–1.0 31 0.941 0.258

h = 4 week(s) ahead (21 forecast weeks)

Model Log-loss, all Log-loss, adds Brier, adds PR-AUC, adds MAE plays Spearman NDCG@50
persistence 0.0773 0.0560 0.0100 0.014 0.36 0.810 0.908
popularity 0.0563 0.0350 0.0091 0.246 0.41 0.841 0.834
kalman 0.0536 0.0346 0.0081 0.347 0.43 0.849 0.895
nb 0.0543 0.0371 0.0081 0.319 0.37 0.835 0.909
logit 0.0484 0.0319 0.0079 0.325 0.37 0.835 0.909

Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):

Model vs persistence vs popularity
kalman 0.383 (0.333 to 0.430) 0.012 (-0.045 to 0.048)
nb 0.337 (0.254 to 0.407) -0.061 (-0.165 to 0.010)
logit 0.430 (0.398 to 0.462) 0.088 (0.056 to 0.102)

Calibration of the NB's add probabilities (rows not in rotation at the origin):

Predicted Rows Mean predicted Observed
0.0–0.1 622067 0.003 0.002
0.1–0.2 15752 0.144 0.083
0.2–0.3 7411 0.245 0.203
0.3–0.4 3454 0.343 0.359
0.4–0.5 1395 0.442 0.463
0.5–0.6 487 0.545 0.591
0.6–0.7 212 0.646 0.538
0.7–0.8 125 0.747 0.208
0.8–0.9 76 0.844 0.579
0.9–1.0 21 0.933 0.667

h = 8 week(s) ahead (17 forecast weeks)

Model Log-loss, all Log-loss, adds Brier, adds PR-AUC, adds MAE plays Spearman NDCG@50
persistence 0.0874 0.0648 0.0119 0.017 0.45 0.779 0.837
popularity 0.0679 0.0445 0.0107 0.236 0.46 0.815 0.784
kalman 0.0703 0.0473 0.0105 0.293 0.57 0.815 0.825
nb 0.0774 0.0570 0.0103 0.270 0.48 0.797 0.828
logit 0.0634 0.0432 0.0101 0.262 0.48 0.797 0.828

Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):

Model vs persistence vs popularity
kalman 0.271 (0.219 to 0.305) -0.064 (-0.100 to -0.046)
nb 0.120 (-0.012 to 0.209) -0.283 (-0.418 to -0.196)
logit 0.334 (0.289 to 0.366) 0.028 (-0.001 to 0.043)

Calibration of the NB's add probabilities (rows not in rotation at the origin):

Predicted Rows Mean predicted Observed
0.0–0.1 497574 0.004 0.004
0.1–0.2 15374 0.143 0.071
0.2–0.3 7616 0.244 0.160
0.3–0.4 4057 0.343 0.263
0.4–0.5 1630 0.441 0.426
0.5–0.6 493 0.544 0.519
0.6–0.7 264 0.637 0.523
0.7–0.8 79 0.751 0.456
0.8–0.9 63 0.837 0.476
0.9–1.0 15 0.922 0.733

NB coefficients, h = 4, all weeks (alpha = 0.116)

Feature Coefficient
intercept -0.694
log plays t +0.380
log plays t-1 +0.140
log mean plays 4w +0.447
log mean plays 12w -0.190
log max plays 12w +0.420
log other channels t -0.024
log other channels 4w +0.045
log P3 t +0.020
weeks since played (/12) -3.147
Track Top-40 score -0.018
Airplay Top-20 score -0.397
chart entry recency 6w +0.425
chart entry recency 26w -0.275
log Track Top-40 hits +0.029
log Airplay Top-20 hits +0.019
Danish -0.013
group -0.091
female +0.022
career years (/50) -0.269
debut unknown -0.080
Christmas target week +0.000
Danish x P3 +0.007
Danish x P4 +0.039
Danish x P6 +0.020
ch p4kbh -0.043
ch p4sjaelland -0.039
ch p4fyn -0.046
ch p4syd -0.040
ch p4trekanten -0.044
ch p4vest -0.041
ch p4aarhus -0.040
ch p4nord -0.037
ch p4bornholm -0.043
ch p4esbjerg -0.041
ch p5 +0.092
ch p6beat +0.044
ch p8jazz +0.085