Files
Airplay-Predictor/docs/BACKTEST.md
T

149 lines
5.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backtest: DR artist × channel × week
Panel: 2329 artists × 14 channels × 51 complete weeks (2025-11-17 to 2026-11-02). Forecast origins from week 27; regressions refitted every 1 origin(s).
"In rotation" = at least τ plays in the week: p3 7, p4kbh 5, p4sjaelland 5, p4fyn 5, p4syd 5, p4trekanten 5, p4vest 5, p4aarhus 5, p4nord 5, p4bornholm 5, p4esbjerg 5, p5 4, p6beat 5, p8jazz 3.
## h = 1 week(s) ahead (24 forecast weeks)
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|---|---|---|---|---|---|---|---|
| persistence | 0.0683 | 0.0488 | 0.0085 | 0.013 | 0.28 | 0.829 | 0.967 |
| popularity | 0.0470 | 0.0287 | 0.0080 | 0.256 | 0.35 | 0.864 | 0.888 |
| kalman | 0.0385 | 0.0256 | 0.0065 | 0.377 | 0.32 | 0.874 | 0.957 |
| nb | 0.0389 | 0.0260 | 0.0066 | 0.355 | 0.28 | 0.859 | 0.965 |
| logit | 0.0363 | 0.0241 | 0.0064 | 0.367 | 0.28 | 0.859 | 0.965 |
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
| Model | vs persistence | vs popularity |
|---|---|---|
| kalman | 0.475 (0.451 to 0.496) | 0.109 (0.063 to 0.141) |
| nb | 0.466 (0.445 to 0.486) | 0.094 (0.051 to 0.123) |
| logit | 0.505 (0.489 to 0.520) | 0.160 (0.129 to 0.184) |
Calibration of the NB's add probabilities (rows not in rotation at the origin):
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 716676 | 0.003 | 0.002 |
| 0.1–0.2 | 15028 | 0.143 | 0.088 |
| 0.2–0.3 | 6884 | 0.244 | 0.219 |
| 0.3–0.4 | 3000 | 0.343 | 0.391 |
| 0.4–0.5 | 1355 | 0.441 | 0.578 |
| 0.5–0.6 | 521 | 0.540 | 0.551 |
| 0.6–0.7 | 197 | 0.650 | 0.482 |
| 0.7–0.8 | 108 | 0.748 | 0.509 |
| 0.8–0.9 | 95 | 0.847 | 0.211 |
| 0.9–1.0 | 31 | 0.941 | 0.258 |
## h = 4 week(s) ahead (21 forecast weeks)
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|---|---|---|---|---|---|---|---|
| persistence | 0.0773 | 0.0560 | 0.0100 | 0.014 | 0.36 | 0.810 | 0.908 |
| popularity | 0.0563 | 0.0350 | 0.0091 | 0.246 | 0.41 | 0.841 | 0.834 |
| kalman | 0.0536 | 0.0346 | 0.0081 | 0.347 | 0.43 | 0.849 | 0.895 |
| nb | 0.0543 | 0.0371 | 0.0081 | 0.319 | 0.37 | 0.835 | 0.909 |
| logit | 0.0484 | 0.0319 | 0.0079 | 0.325 | 0.37 | 0.835 | 0.909 |
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
| Model | vs persistence | vs popularity |
|---|---|---|
| kalman | 0.383 (0.333 to 0.430) | 0.012 (-0.045 to 0.048) |
| nb | 0.337 (0.254 to 0.407) | -0.061 (-0.165 to 0.010) |
| logit | 0.430 (0.398 to 0.462) | 0.088 (0.056 to 0.102) |
Calibration of the NB's add probabilities (rows not in rotation at the origin):
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 622067 | 0.003 | 0.002 |
| 0.1–0.2 | 15752 | 0.144 | 0.083 |
| 0.2–0.3 | 7411 | 0.245 | 0.203 |
| 0.3–0.4 | 3454 | 0.343 | 0.359 |
| 0.4–0.5 | 1395 | 0.442 | 0.463 |
| 0.5–0.6 | 487 | 0.545 | 0.591 |
| 0.6–0.7 | 212 | 0.646 | 0.538 |
| 0.7–0.8 | 125 | 0.747 | 0.208 |
| 0.8–0.9 | 76 | 0.844 | 0.579 |
| 0.9–1.0 | 21 | 0.933 | 0.667 |
## h = 8 week(s) ahead (17 forecast weeks)
| Model | Log-loss, all | Log-loss, adds | Brier, adds | PR-AUC, adds | MAE plays | Spearman | NDCG@50 |
|---|---|---|---|---|---|---|---|
| persistence | 0.0874 | 0.0648 | 0.0119 | 0.017 | 0.45 | 0.779 | 0.837 |
| popularity | 0.0679 | 0.0445 | 0.0107 | 0.236 | 0.46 | 0.815 | 0.784 |
| kalman | 0.0703 | 0.0473 | 0.0105 | 0.293 | 0.57 | 0.815 | 0.825 |
| nb | 0.0774 | 0.0570 | 0.0103 | 0.270 | 0.48 | 0.797 | 0.828 |
| logit | 0.0634 | 0.0432 | 0.0101 | 0.262 | 0.48 | 0.797 | 0.828 |
Skill on adds (1 − log-loss ÷ baseline log-loss; 90% block-bootstrap interval over weeks):
| Model | vs persistence | vs popularity |
|---|---|---|
| kalman | 0.271 (0.219 to 0.305) | -0.064 (-0.100 to -0.046) |
| nb | 0.120 (-0.012 to 0.209) | -0.283 (-0.418 to -0.196) |
| logit | 0.334 (0.289 to 0.366) | 0.028 (-0.001 to 0.043) |
Calibration of the NB's add probabilities (rows not in rotation at the origin):
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 497574 | 0.004 | 0.004 |
| 0.1–0.2 | 15374 | 0.143 | 0.071 |
| 0.2–0.3 | 7616 | 0.244 | 0.160 |
| 0.3–0.4 | 4057 | 0.343 | 0.263 |
| 0.4–0.5 | 1630 | 0.441 | 0.426 |
| 0.5–0.6 | 493 | 0.544 | 0.519 |
| 0.6–0.7 | 264 | 0.637 | 0.523 |
| 0.7–0.8 | 79 | 0.751 | 0.456 |
| 0.8–0.9 | 63 | 0.837 | 0.476 |
| 0.9–1.0 | 15 | 0.922 | 0.733 |
## NB coefficients, h = 4, all weeks (alpha = 0.116)
| Feature | Coefficient |
|---|---|
| intercept | -0.694 |
| log plays t | +0.380 |
| log plays t-1 | +0.140 |
| log mean plays 4w | +0.447 |
| log mean plays 12w | -0.190 |
| log max plays 12w | +0.420 |
| log other channels t | -0.024 |
| log other channels 4w | +0.045 |
| log P3 t | +0.020 |
| weeks since played (/12) | -3.147 |
| Track Top-40 score | -0.018 |
| Airplay Top-20 score | -0.397 |
| chart entry recency 6w | +0.425 |
| chart entry recency 26w | -0.275 |
| log Track Top-40 hits | +0.029 |
| log Airplay Top-20 hits | +0.019 |
| Danish | -0.013 |
| group | -0.091 |
| female | +0.022 |
| career years (/50) | -0.269 |
| debut unknown | -0.080 |
| Christmas target week | +0.000 |
| Danish x P3 | +0.007 |
| Danish x P4 | +0.039 |
| Danish x P6 | +0.020 |
| ch p4kbh | -0.043 |
| ch p4sjaelland | -0.039 |
| ch p4fyn | -0.046 |
| ch p4syd | -0.040 |
| ch p4trekanten | -0.044 |
| ch p4vest | -0.041 |
| ch p4aarhus | -0.040 |
| ch p4nord | -0.037 |
| ch p4bornholm | -0.043 |
| ch p4esbjerg | -0.041 |
| ch p5 | +0.092 |
| ch p6beat | +0.044 |
| ch p8jazz | +0.085 |