148 lines
5.5 KiB
Markdown
148 lines
5.5 KiB
Markdown
# Backtest: DR artist × channel × week
|
||
|
||
Panel: 3246 artists × 14 channels × 51 complete weeks (2025-11-17 to 2026-11-02). Forecast origins from week 27; regressions refitted every 2 origin(s).
|
||
|
||
"In rotation" = at least τ plays in the week: p3 23, p4kbh 15, p4sjaelland 14, p4fyn 14, p4syd 14, p4trekanten 14, p4vest 13, p4aarhus 14, p4nord 15, p4bornholm 14, p4esbjerg 14, p5 10, p6beat 16, p8jazz 7.
|
||
|
||
## h = 1 week(s) ahead (24 forecast weeks)
|
||
|
||
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|
||
|---|---|---|---|---|---|---|---|
|
||
| persistence | 0.0217 | 0.0159 | 0.0023 | 0.003 | 0.27 | 0.826 | 0.967 |
|
||
| popularity | 0.0216 | 0.0110 | 0.0028 | 0.170 | 0.31 | 0.864 | 0.886 |
|
||
| kalman | 0.0156 | 0.0088 | 0.0019 | 0.316 | 0.30 | 0.872 | 0.951 |
|
||
| nb | 0.0131 | 0.0080 | 0.0017 | 0.330 | 0.27 | 0.856 | 0.960 |
|
||
| logit | 0.0113 | 0.0066 | 0.0017 | 0.359 | 0.27 | 0.856 | 0.960 |
|
||
|
||
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
|
||
|
||
| Model | vs persistence | vs popularity |
|
||
|---|---|---|
|
||
| kalman | 0.447 (0.376 to 0.510) | 0.204 (0.136 to 0.266) |
|
||
| nb | 0.497 (0.439 to 0.550) | 0.274 (0.204 to 0.337) |
|
||
| logit | 0.582 (0.554 to 0.614) | 0.398 (0.368 to 0.436) |
|
||
|
||
Calibration of the NB's add probabilities (rows not in rotation at the origin):
|
||
|
||
| Predicted | Rows | Mean predicted | Observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 1067535 | 0.000 | 0.001 |
|
||
| 0.1–0.2 | 2694 | 0.146 | 0.129 |
|
||
| 0.2–0.3 | 1559 | 0.247 | 0.246 |
|
||
| 0.3–0.4 | 1113 | 0.347 | 0.340 |
|
||
| 0.4–0.5 | 697 | 0.449 | 0.420 |
|
||
| 0.5–0.6 | 444 | 0.543 | 0.482 |
|
||
| 0.6–0.7 | 212 | 0.649 | 0.448 |
|
||
| 0.7–0.8 | 96 | 0.740 | 0.417 |
|
||
| 0.8–0.9 | 63 | 0.840 | 0.508 |
|
||
| 0.9–1.0 | 19 | 0.950 | 0.421 |
|
||
|
||
## h = 4 week(s) ahead (21 forecast weeks)
|
||
|
||
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|
||
|---|---|---|---|---|---|---|---|
|
||
| persistence | 0.0318 | 0.0242 | 0.0037 | 0.005 | 0.33 | 0.811 | 0.907 |
|
||
| popularity | 0.0284 | 0.0168 | 0.0039 | 0.145 | 0.36 | 0.845 | 0.829 |
|
||
| kalman | 0.0283 | 0.0196 | 0.0033 | 0.227 | 0.39 | 0.850 | 0.887 |
|
||
| nb | 0.0257 | 0.0186 | 0.0031 | 0.251 | 0.34 | 0.835 | 0.898 |
|
||
| logit | 0.0198 | 0.0127 | 0.0029 | 0.298 | 0.34 | 0.835 | 0.898 |
|
||
|
||
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
|
||
|
||
| Model | vs persistence | vs popularity |
|
||
|---|---|---|
|
||
| kalman | 0.191 (0.046 to 0.325) | -0.170 (-0.329 to -0.038) |
|
||
| nb | 0.235 (0.093 to 0.362) | -0.107 (-0.283 to 0.030) |
|
||
| logit | 0.477 (0.420 to 0.536) | 0.244 (0.175 to 0.293) |
|
||
|
||
Calibration of the NB's add probabilities (rows not in rotation at the origin):
|
||
|
||
| Predicted | Rows | Mean predicted | Observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 931278 | 0.001 | 0.002 |
|
||
| 0.1–0.2 | 3415 | 0.143 | 0.094 |
|
||
| 0.2–0.3 | 2051 | 0.247 | 0.207 |
|
||
| 0.3–0.4 | 1566 | 0.348 | 0.287 |
|
||
| 0.4–0.5 | 1010 | 0.444 | 0.413 |
|
||
| 0.5–0.6 | 528 | 0.545 | 0.443 |
|
||
| 0.6–0.7 | 235 | 0.641 | 0.306 |
|
||
| 0.7–0.8 | 94 | 0.748 | 0.521 |
|
||
| 0.8–0.9 | 35 | 0.839 | 0.457 |
|
||
| 0.9–1.0 | 3 | 0.927 | 0.000 |
|
||
|
||
## h = 8 week(s) ahead (17 forecast weeks)
|
||
|
||
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|
||
|---|---|---|---|---|---|---|---|
|
||
| persistence | 0.0433 | 0.0344 | 0.0055 | 0.007 | 0.40 | 0.783 | 0.833 |
|
||
| popularity | 0.0359 | 0.0241 | 0.0052 | 0.152 | 0.40 | 0.823 | 0.777 |
|
||
| kalman | 0.0427 | 0.0330 | 0.0050 | 0.195 | 0.51 | 0.821 | 0.816 |
|
||
| nb | 0.0418 | 0.0330 | 0.0048 | 0.233 | 0.43 | 0.804 | 0.812 |
|
||
| logit | 0.0301 | 0.0208 | 0.0046 | 0.280 | 0.43 | 0.804 | 0.812 |
|
||
|
||
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
|
||
|
||
| Model | vs persistence | vs popularity |
|
||
|---|---|---|
|
||
| kalman | 0.040 (-0.078 to 0.167) | -0.367 (-0.490 to -0.223) |
|
||
| nb | 0.042 (-0.097 to 0.159) | -0.365 (-0.522 to -0.228) |
|
||
| logit | 0.394 (0.348 to 0.433) | 0.137 (0.089 to 0.173) |
|
||
|
||
Calibration of the NB's add probabilities (rows not in rotation at the origin):
|
||
|
||
| Predicted | Rows | Mean predicted | Observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 751167 | 0.001 | 0.003 |
|
||
| 0.1–0.2 | 4090 | 0.145 | 0.084 |
|
||
| 0.2–0.3 | 2546 | 0.248 | 0.209 |
|
||
| 0.3–0.4 | 1673 | 0.348 | 0.289 |
|
||
| 0.4–0.5 | 976 | 0.443 | 0.416 |
|
||
| 0.5–0.6 | 429 | 0.544 | 0.445 |
|
||
| 0.6–0.7 | 208 | 0.646 | 0.409 |
|
||
| 0.7–0.8 | 76 | 0.740 | 0.434 |
|
||
| 0.8–0.9 | 25 | 0.828 | 0.880 |
|
||
|
||
## NB coefficients, h = 4, all weeks (alpha = 0.134)
|
||
|
||
| Feature | Coefficient |
|
||
|---|---|
|
||
| intercept | -0.787 |
|
||
| log plays t | +0.313 |
|
||
| log plays t-1 | +0.140 |
|
||
| log mean plays 4w | +0.480 |
|
||
| log mean plays 12w | -0.057 |
|
||
| log max plays 12w | +0.368 |
|
||
| log other channels t | -0.018 |
|
||
| log other channels 4w | +0.044 |
|
||
| log P3 t | +0.014 |
|
||
| weeks since played (/12) | -3.164 |
|
||
| Track Top-40 score | -0.021 |
|
||
| Airplay Top-20 score | -0.418 |
|
||
| chart entry recency 6w | +0.539 |
|
||
| chart entry recency 26w | -0.383 |
|
||
| log Track Top-40 hits | +0.032 |
|
||
| log Airplay Top-20 hits | +0.005 |
|
||
| Danish | -0.007 |
|
||
| group | -0.053 |
|
||
| female | +0.021 |
|
||
| career years (/50) | -0.229 |
|
||
| debut unknown | -0.059 |
|
||
| Christmas target week | +0.000 |
|
||
| Danish x P3 | -0.001 |
|
||
| Danish x P4 | +0.020 |
|
||
| Danish x P6 | +0.007 |
|
||
| ch p4kbh | -0.013 |
|
||
| ch p4sjaelland | -0.009 |
|
||
| ch p4fyn | -0.015 |
|
||
| ch p4syd | -0.010 |
|
||
| ch p4trekanten | -0.012 |
|
||
| ch p4vest | -0.010 |
|
||
| ch p4aarhus | -0.010 |
|
||
| ch p4nord | -0.007 |
|
||
| ch p4bornholm | -0.013 |
|
||
| ch p4esbjerg | -0.011 |
|
||
| ch p5 | +0.118 |
|
||
| ch p6beat | +0.066 |
|
||
| ch p8jazz | +0.103 |
|
||
|