Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

143 lines
4.8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backtest: new adds
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.
## Window starting h = 0 weeks ahead (17 forecast origins)
Base rates: per channel 0.75% of artist-channels get an add in the window; on any channel 4.2% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0440 | 0.012 | 0.1758 | 0.042 |
| rate by last-12-weeks level | 0.0328 | 0.099 | 0.1656 | 0.083 |
| logit | 0.0273 | 0.230 | 0.1563 | 0.156 |
Skill of the logit (90% block-bootstrap interval over origins):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.381 (0.361 to 0.396) | 0.168 (0.148 to 0.181) |
| any channel | 0.111 (0.099 to 0.120) | 0.056 (0.040 to 0.068) |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 760136 | 0.003 | 0.004 |
| 0.1–0.2 | 4598 | 0.148 | 0.180 |
| 0.2–0.3 | 3797 | 0.246 | 0.219 |
| 0.3–0.4 | 1815 | 0.345 | 0.297 |
| 0.4–0.5 | 1080 | 0.444 | 0.359 |
| 0.5–0.6 | 715 | 0.548 | 0.365 |
| 0.6–0.7 | 252 | 0.643 | 0.476 |
| 0.7–0.8 | 149 | 0.742 | 0.470 |
| 0.8–0.9 | 6 | 0.809 | 0.000 |
## Window starting h = 4 weeks ahead (13 forecast origins)
Base rates: per channel 0.77% of artist-channels get an add in the window; on any channel 4.3% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0438 | 0.019 | 0.1752 | 0.053 |
| rate by last-12-weeks level | 0.0347 | 0.091 | 0.1689 | 0.078 |
| logit | 0.0326 | 0.139 | 0.1629 | 0.127 |
Skill of the logit (90% block-bootstrap interval over origins):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.257 (0.242 to 0.276) | 0.062 (0.043 to 0.077) |
| any channel | 0.070 (0.063 to 0.075) | 0.035 (0.020 to 0.046) |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 579503 | 0.004 | 0.005 |
| 0.1–0.2 | 6709 | 0.143 | 0.118 |
| 0.2–0.3 | 2781 | 0.243 | 0.193 |
| 0.3–0.4 | 1213 | 0.341 | 0.232 |
| 0.4–0.5 | 375 | 0.442 | 0.291 |
| 0.5–0.6 | 131 | 0.542 | 0.198 |
| 0.6–0.7 | 42 | 0.637 | 0.571 |
| 0.7–0.8 | 18 | 0.721 | 0.667 |
## Coefficients, window starting now (h = 0), all closed windows
| Any-channel feature | Coefficient |
|---|---|
| intercept | -1.503 |
| log DR plays t | +0.669 |
| log DR plays 4w | +0.119 |
| log DR plays 12w | -0.428 |
| log max DR plays 12w | +0.140 |
| in rotation anywhere now | -2.146 |
| channels in rotation now (/14) | -2.288 |
| current run anywhere (/12) | -1.783 |
| weeks since in rotation anywhere (/12) | -2.397 |
| Track Top-40 score | +0.248 |
| Airplay Top-20 score | +0.210 |
| chart entry recency 6w | +0.482 |
| chart entry recency 26w | -0.328 |
| log Track Top-40 hits | +0.002 |
| log Airplay Top-20 hits | -0.280 |
| Danish | +0.500 |
| group | -0.468 |
| female | -0.142 |
| career years (/50) | -0.250 |
| debut unknown | +0.129 |
| Christmas in window | +0.000 |
| Per-channel feature | Coefficient |
|---|---|
| intercept | -6.792 |
| log plays t | +0.665 |
| log plays t-1 | +0.129 |
| log mean plays 4w | +0.169 |
| log mean plays 12w | -0.662 |
| log max plays 12w | +1.391 |
| log other channels t | +0.257 |
| log other channels 4w | -0.214 |
| log P3 t | +0.174 |
| weeks since played (/12) | -0.733 |
| Track Top-40 score | -0.563 |
| Airplay Top-20 score | -0.416 |
| chart entry recency 6w | +0.160 |
| chart entry recency 26w | +0.288 |
| log Track Top-40 hits | +0.260 |
| log Airplay Top-20 hits | -0.324 |
| Danish | +0.295 |
| group | -0.840 |
| female | +0.007 |
| career years (/50) | -1.415 |
| debut unknown | -0.130 |
| Christmas target week | +0.000 |
| Danish x P3 | -0.176 |
| Danish x P4 | +0.154 |
| Danish x P6 | -0.213 |
| ch p4kbh | +0.236 |
| ch p4sjaelland | +0.256 |
| ch p4fyn | +0.235 |
| ch p4syd | +0.297 |
| ch p4trekanten | +0.272 |
| ch p4vest | +0.361 |
| ch p4aarhus | +0.297 |
| ch p4nord | +0.186 |
| ch p4bornholm | +0.278 |
| ch p4esbjerg | +0.290 |
| ch p5 | +1.468 |
| ch p6beat | +0.871 |
| ch p8jazz | +1.960 |
| in rotation on channel now | -1.352 |
| in rotation anywhere now | +0.104 |
| channels in rotation now (/14) | -0.674 |
| run on channel = 1 week | +0.944 |
| run = 2 weeks | +0.352 |
| run = 3-5 weeks | -1.570 |
| run >= 6 weeks | -1.078 |
| near threshold (>= half of tau, not in rotation) | +1.389 |
| plays / tau (capped 3, /3) | -0.664 |
| weeks since in rotation on channel (/12) | +0.421 |