Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

169 lines
5.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backtest: new adds
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.
## Window starting h = 0 weeks ahead (17 forecast origins)
Base rates: per channel 0.36% of artist-channels get an add in the window; on any channel 2.0% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0234 | 0.006 | 0.0994 | 0.023 |
| rate by last-12-weeks level | 0.0162 | 0.097 | 0.0863 | 0.073 |
| logit | 0.0130 | 0.245 | 0.0747 | 0.199 |
Skill of the logit (90% block-bootstrap interval over origins):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.444 (0.427 to 0.458) | 0.196 (0.186 to 0.199) |
| any channel | 0.249 (0.230 to 0.265) | 0.134 (0.124 to 0.142) |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 1669178 | 0.002 | 0.002 |
| 0.1–0.2 | 5763 | 0.143 | 0.193 |
| 0.2–0.3 | 2302 | 0.244 | 0.232 |
| 0.3–0.4 | 1408 | 0.345 | 0.294 |
| 0.4–0.5 | 872 | 0.441 | 0.336 |
| 0.5–0.6 | 583 | 0.547 | 0.453 |
| 0.6–0.7 | 306 | 0.653 | 0.461 |
| 0.7–0.8 | 189 | 0.739 | 0.339 |
| 0.8–0.9 | 152 | 0.839 | 0.658 |
| 0.9–1.0 | 3 | 0.902 | 1.000 |
## Window starting h = 4 weeks ahead (13 forecast origins)
Base rates: per channel 0.37% of artist-channels get an add in the window; on any channel 2.0% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0232 | 0.014 | 0.0979 | 0.031 |
| rate by last-12-weeks level | 0.0174 | 0.087 | 0.0887 | 0.066 |
| logit | 0.0160 | 0.134 | 0.0805 | 0.142 |
Skill of the logit (90% block-bootstrap interval over origins):
| | vs in-rotation-now | vs last-12-weeks level |
|---|---|---|
| per channel | 0.313 (0.305 to 0.323) | 0.081 (0.057 to 0.096) |
| any channel | 0.178 (0.162 to 0.190) | 0.092 (0.084 to 0.098) |
Calibration, per channel:
| Predicted | Rows | Mean predicted | Observed |
|---|---|---|---|
| 0.0–0.1 | 1275342 | 0.002 | 0.002 |
| 0.1–0.2 | 6075 | 0.141 | 0.120 |
| 0.2–0.3 | 2066 | 0.244 | 0.169 |
| 0.3–0.4 | 812 | 0.342 | 0.197 |
| 0.4–0.5 | 509 | 0.447 | 0.287 |
| 0.5–0.6 | 332 | 0.539 | 0.455 |
| 0.6–0.7 | 110 | 0.650 | 0.273 |
| 0.7–0.8 | 36 | 0.737 | 0.278 |
| 0.8–0.9 | 2 | 0.841 | 0.000 |
## Coefficients, window starting now (h = 0), all closed windows
| Any-channel feature | Coefficient |
|---|---|
| intercept | -5.107 |
| log DR plays t | +0.979 |
| log DR plays 4w | +0.245 |
| log DR plays 12w | -0.284 |
| log max DR plays 12w | +0.420 |
| in rotation anywhere now | -1.359 |
| station groups in rotation now (/5) | -3.411 |
| current run anywhere (/12) | -1.907 |
| weeks since in rotation anywhere (/12) | -0.348 |
| Track Top-40 score | +0.147 |
| Airplay Top-20 score | -0.633 |
| chart entry recency 6w | +1.325 |
| chart entry recency 26w | -1.443 |
| log Track Top-40 hits | -0.083 |
| log Airplay Top-20 hits | -0.057 |
| Danish | +0.505 |
| group | -0.363 |
| female | -0.169 |
| career years (/50) | +0.108 |
| debut unknown | +0.514 |
| Christmas in window | +0.000 |
| discography known | +0.877 |
| release recency 4w | +0.345 |
| release recency 12w | +0.630 |
| log releases last 52w | -0.191 |
| releases in the window | -0.425 |
| releases in window x not in rotation now | +1.334 |
| DR: weeks since last add 13-25 | -0.015 |
| DR: last add 26+ weeks ago | -0.511 |
| DR: no add in the panel yet | -0.998 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.255 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.106 |
| Airplay Top-20: last new song 2+ years ago | -0.548 |
| Per-channel feature | Coefficient |
|---|---|
| intercept | -8.129 |
| log plays t | +0.772 |
| log plays t-1 | +0.119 |
| log mean plays 4w | +0.121 |
| log mean plays 12w | -0.700 |
| log max plays 12w | +1.336 |
| log other channels t | +0.210 |
| log other channels 4w | -0.129 |
| log P3 t | +0.121 |
| weeks since played (/12) | -1.250 |
| Track Top-40 score | -0.315 |
| Airplay Top-20 score | -0.715 |
| chart entry recency 6w | +0.400 |
| chart entry recency 26w | -0.160 |
| log Track Top-40 hits | +0.273 |
| log Airplay Top-20 hits | -0.269 |
| Danish | +0.222 |
| group | -0.661 |
| female | +0.019 |
| career years (/50) | -1.152 |
| debut unknown | -0.069 |
| Christmas target week | +0.000 |
| Danish x P3 | +0.010 |
| Danish x P4 | +0.343 |
| Danish x P6 | -0.044 |
| ch p4kbh | +0.293 |
| ch p4sjaelland | +0.344 |
| ch p4fyn | +0.357 |
| ch p4syd | +0.428 |
| ch p4trekanten | +0.408 |
| ch p4vest | +0.465 |
| ch p4aarhus | +0.378 |
| ch p4nord | +0.230 |
| ch p4bornholm | +0.373 |
| ch p4esbjerg | +0.398 |
| ch p5 | +1.665 |
| ch p6beat | +1.017 |
| ch p8jazz | +2.213 |
| in rotation on channel now | -1.017 |
| in rotation anywhere now | +0.188 |
| station groups in rotation now (/5) | -1.490 |
| run on channel = 1 week | +0.936 |
| run = 2 weeks | +0.398 |
| run = 3-5 weeks | -1.398 |
| run >= 6 weeks | -0.954 |
| near threshold (>= half of tau, not in rotation) | +1.429 |
| plays / tau (capped 3, /3) | -0.704 |
| weeks since in rotation on channel (/12) | +0.497 |
| discography known | +0.741 |
| release recency 4w | +0.924 |
| release recency 12w | +0.021 |
| log releases last 52w | -0.185 |
| releases in the window | +0.226 |
| releases in window x not in rotation now | +0.822 |
| DR: weeks since last add 13-25 | +0.070 |
| DR: last add 26+ weeks ago | +0.247 |
| DR: no add in the panel yet | -0.105 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.171 |
| Airplay Top-20: 0.5-1.25x own usual gap | +0.032 |
| Airplay Top-20: last new song 2+ years ago | -0.254 |