Files

233 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backtest: new adds
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.
## Window starting h = 0 weeks ahead (16 forecast origins)
Base rates: per channel 0.46% of artist-channels get an add in the window; on any channel 2.5% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0271 | 0.032 | 0.1111 | 0.069 |
| rate by last-12-weeks level | 0.0188 | 0.148 | 0.0956 | 0.134 |
| release timing only | 0.0274 | 0.010 | 0.1144 | 0.040 |
| model, calibrated on the last 6 origins (pre-release) | 0.0156 | 0.263 | 0.0837 | 0.254 |
| model, raw (pre-release) | 0.0155 | 0.263 | 0.0854 | 0.246 |
| model + pooled Platt (pre-release) | 0.0156 | 0.262 | 0.0855 | 0.246 |
| model + pooled isotonic (pre-release) | 0.0157 | 0.254 | 0.0859 | 0.244 |
| model, NMF known (post-release) | 0.0153 | 0.270 | 0.0836 | 0.263 |
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.174 (+0.167 to +0.176) | +0.432 (+0.412 to +0.445) | +0.124 (+0.123 to +0.133) | +0.269 (+0.242 to +0.289) |
| model + pooled Platt (pre-release) | +0.172 (+0.164 to +0.172) | +0.431 (+0.409 to +0.444) | +0.105 (+0.091 to +0.112) | +0.253 (+0.213 to +0.272) |
| model + pooled isotonic (pre-release) | +0.168 (+0.160 to +0.168) | +0.428 (+0.405 to +0.444) | +0.102 (+0.086 to +0.112) | +0.250 (+0.209 to +0.272) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 112229 / 0.010 / 0.013 | 111004 / 0.012 / 0.012 | 111041 / 0.011 / 0.012 |
| 0.1–0.2 | 2985 / 0.142 / 0.196 | 3319 / 0.142 / 0.155 | 2508 / 0.146 / 0.149 |
| 0.2–0.3 | 1360 / 0.245 / 0.263 | 1636 / 0.245 / 0.249 | 2058 / 0.249 / 0.218 |
| 0.3–0.4 | 799 / 0.346 / 0.328 | 998 / 0.347 / 0.305 | 1765 / 0.339 / 0.301 |
| 0.4–0.5 | 450 / 0.447 / 0.360 | 626 / 0.445 / 0.321 | 489 / 0.442 / 0.309 |
| 0.5–0.6 | 251 / 0.546 / 0.347 | 355 / 0.548 / 0.293 | 247 / 0.546 / 0.352 |
| 0.6–0.7 | 162 / 0.647 / 0.407 | 215 / 0.646 / 0.344 | 152 / 0.646 / 0.421 |
| 0.7–0.8 | 93 / 0.746 / 0.505 | 149 / 0.740 / 0.483 | 73 / 0.740 / 0.479 |
| 0.8–0.9 | 52 / 0.841 / 0.596 | 74 / 0.837 / 0.554 | 43 / 0.837 / 0.558 |
| 0.9–1.0 | 11 / 0.924 / 0.182 | 16 / 0.924 / 0.438 | 16 / 0.932 / 0.500 |
Calibration, per channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1638964 / 0.002 / 0.002 | 1639990 / 0.002 / 0.002 | 1637740 / 0.002 / 0.002 |
| 0.1–0.2 | 8541 / 0.142 / 0.165 | 7975 / 0.141 / 0.176 | 9301 / 0.135 / 0.149 |
| 0.2–0.3 | 3740 / 0.248 / 0.229 | 3586 / 0.247 / 0.231 | 4805 / 0.251 / 0.229 |
| 0.3–0.4 | 2586 / 0.344 / 0.216 | 2464 / 0.345 / 0.211 | 3432 / 0.338 / 0.297 |
| 0.4–0.5 | 1570 / 0.449 / 0.336 | 1465 / 0.449 / 0.374 | 939 / 0.448 / 0.376 |
| 0.5–0.6 | 984 / 0.545 / 0.450 | 966 / 0.547 / 0.443 | 588 / 0.548 / 0.371 |
| 0.6–0.7 | 621 / 0.645 / 0.473 | 549 / 0.643 / 0.461 | 328 / 0.644 / 0.424 |
| 0.7–0.8 | 351 / 0.751 / 0.464 | 358 / 0.751 / 0.455 | 207 / 0.756 / 0.560 |
| 0.8–0.9 | 91 / 0.835 / 0.352 | 90 / 0.834 / 0.489 | 148 / 0.853 / 0.507 |
| 0.9–1.0 | 40 / 0.914 / 0.950 | 45 / 0.928 / 0.689 | — |
## Window starting h = 4 weeks ahead (12 forecast origins)
Base rates: per channel 0.47% of artist-channels get an add in the window; on any channel 2.5% of artists.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0277 | 0.033 | 0.1102 | 0.084 |
| rate by last-12-weeks level | 0.0204 | 0.117 | 0.0984 | 0.137 |
| release timing only | 0.0286 | 0.010 | 0.1146 | 0.043 |
| model, calibrated on the last 6 origins (pre-release) | 0.0186 | 0.166 | 0.0898 | 0.208 |
| model, raw (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model + pooled Platt (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model + pooled isotonic (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
| model, NMF known (post-release) | 0.0183 | 0.174 | 0.0896 | 0.219 |
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.088 (+0.084 to +0.095) | +0.217 (+0.207 to +0.221) |
| model + pooled Platt (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
| model + pooled isotonic (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 84532 / 0.011 / 0.015 | 83207 / 0.013 / 0.013 | 83207 / 0.013 / 0.013 |
| 0.1–0.2 | 2353 / 0.140 / 0.178 | 2764 / 0.142 / 0.156 | 2764 / 0.142 / 0.156 |
| 0.2–0.3 | 904 / 0.242 / 0.304 | 1271 / 0.245 / 0.238 | 1271 / 0.245 / 0.238 |
| 0.3–0.4 | 479 / 0.347 / 0.311 | 717 / 0.346 / 0.262 | 717 / 0.346 / 0.262 |
| 0.4–0.5 | 221 / 0.446 / 0.385 | 359 / 0.445 / 0.304 | 359 / 0.445 / 0.304 |
| 0.5–0.6 | 124 / 0.544 / 0.306 | 188 / 0.547 / 0.372 | 188 / 0.547 / 0.372 |
| 0.6–0.7 | 83 / 0.640 / 0.325 | 142 / 0.641 / 0.345 | 142 / 0.641 / 0.345 |
| 0.7–0.8 | 30 / 0.750 / 0.333 | 66 / 0.749 / 0.273 | 66 / 0.749 / 0.273 |
| 0.8–0.9 | 4 / 0.830 / 0.000 | 16 / 0.835 / 0.312 | 16 / 0.835 / 0.312 |
Calibration, per channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 |
| 0.1–0.2 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 |
| 0.2–0.3 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 |
| 0.3–0.4 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 |
| 0.4–0.5 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 |
| 0.5–0.6 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 |
| 0.6–0.7 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 |
| 0.7–0.8 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 |
| 0.8–0.9 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 |
| 0.9–1.0 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 |
## Coefficients, window starting now (h = 0), all closed windows
| Any-channel feature | Coefficient |
|---|---|
| intercept | -4.141 |
| log DR plays t | +0.882 |
| log DR plays 4w | +0.253 |
| log DR plays 12w | -0.255 |
| log max DR plays 12w | +0.583 |
| in rotation anywhere now | -1.155 |
| station groups in rotation now (/5) | +0.320 |
| current run anywhere (/12) | -1.215 |
| weeks since in rotation anywhere (/12) | -0.748 |
| Track Top-40 score | +0.427 |
| Airplay Top-20 score | -1.336 |
| chart entry recency 6w | +1.203 |
| chart entry recency 26w | -0.646 |
| log Track Top-40 hits | -0.201 |
| log Airplay Top-20 hits | -0.070 |
| Danish | +0.470 |
| group | -0.253 |
| female | -0.097 |
| career years (/50) | +0.222 |
| debut unknown | +0.489 |
| Christmas in window | +0.000 |
| discography known | -0.392 |
| release recency 4w | +0.435 |
| release recency 12w | +0.680 |
| log releases last 52w | -0.186 |
| releases in the window | -0.216 |
| releases in window x not in rotation now | +0.836 |
| DR: weeks since last add 13-25 | +0.095 |
| DR: last add 26+ weeks ago | -0.642 |
| DR: no add in the panel yet | -0.845 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.278 |
| Airplay Top-20: 0.5-1.25x own usual gap | +0.229 |
| Airplay Top-20: last new song 2+ years ago | -0.527 |
| log archive plays per 100 sampled hours (2012-21) | +0.289 |
| log DR lists with the artist before t (2023-) | +0.209 |
| DR list recency 26w | -0.924 |
| DR lists known before t at all | +0.039 |
| NMF Denmark in the window | +0.940 |
| NMF Denmark top 10 in the window | +0.408 |
| log NMF Denmark placements last 52w | -0.068 |
| share of releases on NMF Denmark (2 years) | -0.417 |
| season index of the window (log, IFPI 2002-24) | +0.412 |
| Per-channel feature | Coefficient |
|---|---|
| intercept | -7.117 |
| log plays t | +1.106 |
| log plays t-1 | +0.141 |
| log mean plays 4w | +0.113 |
| log mean plays 12w | -0.871 |
| log max plays 12w | +1.022 |
| log other channels t | +0.328 |
| log other channels 4w | -0.095 |
| log P3 t | +0.087 |
| weeks since played (/12) | -0.509 |
| Track Top-40 score | -0.132 |
| Airplay Top-20 score | -1.376 |
| chart entry recency 6w | +0.616 |
| chart entry recency 26w | -0.506 |
| log Track Top-40 hits | +0.258 |
| log Airplay Top-20 hits | -0.398 |
| Danish | +0.171 |
| group | -0.636 |
| female | +0.136 |
| career years (/50) | -1.498 |
| debut unknown | -0.333 |
| Christmas target week | +0.000 |
| Danish x P3 | +0.064 |
| Danish x P4 | +0.058 |
| Danish x P6 | +0.111 |
| ch p4kbh | +1.094 |
| ch p4sjaelland | +0.966 |
| ch p4fyn | +0.946 |
| ch p4syd | +0.975 |
| ch p4trekanten | +0.985 |
| ch p4vest | +1.083 |
| ch p4aarhus | +0.944 |
| ch p4nord | +1.006 |
| ch p4bornholm | +0.995 |
| ch p4esbjerg | +0.972 |
| ch p5 | +2.081 |
| ch p6beat | +1.296 |
| ch p8jazz | +2.312 |
| in rotation on channel now | +0.350 |
| in rotation anywhere now | -0.334 |
| station groups in rotation now (/5) | +0.041 |
| run on channel = 1 week | +0.660 |
| run = 2 weeks | +0.439 |
| run = 3-5 weeks | -0.056 |
| run >= 6 weeks | -0.693 |
| near threshold (>= half of tau, not in rotation) | +1.381 |
| plays / tau (capped 3, /3) | -3.363 |
| weeks since in rotation on channel (/12) | -0.302 |
| discography known | -0.052 |
| release recency 4w | +1.017 |
| release recency 12w | +0.014 |
| log releases last 52w | +0.009 |
| releases in the window | -0.136 |
| releases in window x not in rotation now | +0.755 |
| DR: weeks since last add 13-25 | +0.106 |
| DR: last add 26+ weeks ago | -0.081 |
| DR: no add in the panel yet | -0.483 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.194 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.013 |
| Airplay Top-20: last new song 2+ years ago | -0.418 |
| log archive plays per 100 sampled hours (2012-21) | +0.347 |
| log DR lists with the artist before t (2023-) | +0.023 |
| DR list recency 26w | -0.442 |
| DR lists known before t at all | +0.066 |
| NMF Denmark in the window | +0.779 |
| NMF Denmark top 10 in the window | +0.580 |
| log NMF Denmark placements last 52w | -0.089 |
| share of releases on NMF Denmark (2 years) | +0.212 |
| season index of the window (log, IFPI 2002-24) | +1.211 |
| share of the artist's DR plays on this station (before t) | +0.986 |
| log DR plays on this station (before t) | +0.050 |
| never played on this station (before t) | -2.825 |
| genre fit for this station | +0.091 |