233 lines
12 KiB
Markdown
233 lines
12 KiB
Markdown
# Backtest: new adds
|
||
|
||
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.
|
||
|
||
## Window starting h = 0 weeks ahead (16 forecast origins)
|
||
|
||
Base rates: per channel 0.46% of artist-channels get an add in the window; on any channel 2.5% of artists.
|
||
|
||
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||
|---|---|---|---|---|
|
||
| rate by in-rotation-now | 0.0271 | 0.032 | 0.1111 | 0.069 |
|
||
| rate by last-12-weeks level | 0.0188 | 0.148 | 0.0956 | 0.134 |
|
||
| release timing only | 0.0274 | 0.010 | 0.1144 | 0.040 |
|
||
| model, calibrated on the last 6 origins (pre-release) | 0.0156 | 0.263 | 0.0837 | 0.254 |
|
||
| model, raw (pre-release) | 0.0155 | 0.263 | 0.0854 | 0.246 |
|
||
| model + pooled Platt (pre-release) | 0.0156 | 0.262 | 0.0855 | 0.246 |
|
||
| model + pooled isotonic (pre-release) | 0.0157 | 0.254 | 0.0859 | 0.244 |
|
||
| model, NMF known (post-release) | 0.0153 | 0.270 | 0.0836 | 0.263 |
|
||
|
||
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||
|
||
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||
|---|---|---|---|---|
|
||
| model, calibrated on the last 6 origins (pre-release) | +0.174 (+0.167 to +0.176) | +0.432 (+0.412 to +0.445) | +0.124 (+0.123 to +0.133) | +0.269 (+0.242 to +0.289) |
|
||
| model + pooled Platt (pre-release) | +0.172 (+0.164 to +0.172) | +0.431 (+0.409 to +0.444) | +0.105 (+0.091 to +0.112) | +0.253 (+0.213 to +0.272) |
|
||
| model + pooled isotonic (pre-release) | +0.168 (+0.160 to +0.168) | +0.428 (+0.405 to +0.444) | +0.102 (+0.086 to +0.112) | +0.250 (+0.209 to +0.272) |
|
||
|
||
Calibration, any channel:
|
||
|
||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 112229 / 0.010 / 0.013 | 111004 / 0.012 / 0.012 | 111041 / 0.011 / 0.012 |
|
||
| 0.1–0.2 | 2985 / 0.142 / 0.196 | 3319 / 0.142 / 0.155 | 2508 / 0.146 / 0.149 |
|
||
| 0.2–0.3 | 1360 / 0.245 / 0.263 | 1636 / 0.245 / 0.249 | 2058 / 0.249 / 0.218 |
|
||
| 0.3–0.4 | 799 / 0.346 / 0.328 | 998 / 0.347 / 0.305 | 1765 / 0.339 / 0.301 |
|
||
| 0.4–0.5 | 450 / 0.447 / 0.360 | 626 / 0.445 / 0.321 | 489 / 0.442 / 0.309 |
|
||
| 0.5–0.6 | 251 / 0.546 / 0.347 | 355 / 0.548 / 0.293 | 247 / 0.546 / 0.352 |
|
||
| 0.6–0.7 | 162 / 0.647 / 0.407 | 215 / 0.646 / 0.344 | 152 / 0.646 / 0.421 |
|
||
| 0.7–0.8 | 93 / 0.746 / 0.505 | 149 / 0.740 / 0.483 | 73 / 0.740 / 0.479 |
|
||
| 0.8–0.9 | 52 / 0.841 / 0.596 | 74 / 0.837 / 0.554 | 43 / 0.837 / 0.558 |
|
||
| 0.9–1.0 | 11 / 0.924 / 0.182 | 16 / 0.924 / 0.438 | 16 / 0.932 / 0.500 |
|
||
|
||
Calibration, per channel:
|
||
|
||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 1638964 / 0.002 / 0.002 | 1639990 / 0.002 / 0.002 | 1637740 / 0.002 / 0.002 |
|
||
| 0.1–0.2 | 8541 / 0.142 / 0.165 | 7975 / 0.141 / 0.176 | 9301 / 0.135 / 0.149 |
|
||
| 0.2–0.3 | 3740 / 0.248 / 0.229 | 3586 / 0.247 / 0.231 | 4805 / 0.251 / 0.229 |
|
||
| 0.3–0.4 | 2586 / 0.344 / 0.216 | 2464 / 0.345 / 0.211 | 3432 / 0.338 / 0.297 |
|
||
| 0.4–0.5 | 1570 / 0.449 / 0.336 | 1465 / 0.449 / 0.374 | 939 / 0.448 / 0.376 |
|
||
| 0.5–0.6 | 984 / 0.545 / 0.450 | 966 / 0.547 / 0.443 | 588 / 0.548 / 0.371 |
|
||
| 0.6–0.7 | 621 / 0.645 / 0.473 | 549 / 0.643 / 0.461 | 328 / 0.644 / 0.424 |
|
||
| 0.7–0.8 | 351 / 0.751 / 0.464 | 358 / 0.751 / 0.455 | 207 / 0.756 / 0.560 |
|
||
| 0.8–0.9 | 91 / 0.835 / 0.352 | 90 / 0.834 / 0.489 | 148 / 0.853 / 0.507 |
|
||
| 0.9–1.0 | 40 / 0.914 / 0.950 | 45 / 0.928 / 0.689 | — |
|
||
|
||
## Window starting h = 4 weeks ahead (12 forecast origins)
|
||
|
||
Base rates: per channel 0.47% of artist-channels get an add in the window; on any channel 2.5% of artists.
|
||
|
||
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||
|---|---|---|---|---|
|
||
| rate by in-rotation-now | 0.0277 | 0.033 | 0.1102 | 0.084 |
|
||
| rate by last-12-weeks level | 0.0204 | 0.117 | 0.0984 | 0.137 |
|
||
| release timing only | 0.0286 | 0.010 | 0.1146 | 0.043 |
|
||
| model, calibrated on the last 6 origins (pre-release) | 0.0186 | 0.166 | 0.0898 | 0.208 |
|
||
| model, raw (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||
| model + pooled Platt (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||
| model + pooled isotonic (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||
| model, NMF known (post-release) | 0.0183 | 0.174 | 0.0896 | 0.219 |
|
||
|
||
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||
|
||
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||
|---|---|---|---|---|
|
||
| model, calibrated on the last 6 origins (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.088 (+0.084 to +0.095) | +0.217 (+0.207 to +0.221) |
|
||
| model + pooled Platt (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
|
||
| model + pooled isotonic (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
|
||
|
||
Calibration, any channel:
|
||
|
||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 84532 / 0.011 / 0.015 | 83207 / 0.013 / 0.013 | 83207 / 0.013 / 0.013 |
|
||
| 0.1–0.2 | 2353 / 0.140 / 0.178 | 2764 / 0.142 / 0.156 | 2764 / 0.142 / 0.156 |
|
||
| 0.2–0.3 | 904 / 0.242 / 0.304 | 1271 / 0.245 / 0.238 | 1271 / 0.245 / 0.238 |
|
||
| 0.3–0.4 | 479 / 0.347 / 0.311 | 717 / 0.346 / 0.262 | 717 / 0.346 / 0.262 |
|
||
| 0.4–0.5 | 221 / 0.446 / 0.385 | 359 / 0.445 / 0.304 | 359 / 0.445 / 0.304 |
|
||
| 0.5–0.6 | 124 / 0.544 / 0.306 | 188 / 0.547 / 0.372 | 188 / 0.547 / 0.372 |
|
||
| 0.6–0.7 | 83 / 0.640 / 0.325 | 142 / 0.641 / 0.345 | 142 / 0.641 / 0.345 |
|
||
| 0.7–0.8 | 30 / 0.750 / 0.333 | 66 / 0.749 / 0.273 | 66 / 0.749 / 0.273 |
|
||
| 0.8–0.9 | 4 / 0.830 / 0.000 | 16 / 0.835 / 0.312 | 16 / 0.835 / 0.312 |
|
||
|
||
Calibration, per channel:
|
||
|
||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||
|---|---|---|---|
|
||
| 0.0–0.1 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 |
|
||
| 0.1–0.2 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 |
|
||
| 0.2–0.3 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 |
|
||
| 0.3–0.4 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 |
|
||
| 0.4–0.5 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 |
|
||
| 0.5–0.6 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 |
|
||
| 0.6–0.7 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 |
|
||
| 0.7–0.8 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 |
|
||
| 0.8–0.9 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 |
|
||
| 0.9–1.0 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 |
|
||
|
||
## Coefficients, window starting now (h = 0), all closed windows
|
||
|
||
| Any-channel feature | Coefficient |
|
||
|---|---|
|
||
| intercept | -4.141 |
|
||
| log DR plays t | +0.882 |
|
||
| log DR plays 4w | +0.253 |
|
||
| log DR plays 12w | -0.255 |
|
||
| log max DR plays 12w | +0.583 |
|
||
| in rotation anywhere now | -1.155 |
|
||
| station groups in rotation now (/5) | +0.320 |
|
||
| current run anywhere (/12) | -1.215 |
|
||
| weeks since in rotation anywhere (/12) | -0.748 |
|
||
| Track Top-40 score | +0.427 |
|
||
| Airplay Top-20 score | -1.336 |
|
||
| chart entry recency 6w | +1.203 |
|
||
| chart entry recency 26w | -0.646 |
|
||
| log Track Top-40 hits | -0.201 |
|
||
| log Airplay Top-20 hits | -0.070 |
|
||
| Danish | +0.470 |
|
||
| group | -0.253 |
|
||
| female | -0.097 |
|
||
| career years (/50) | +0.222 |
|
||
| debut unknown | +0.489 |
|
||
| Christmas in window | +0.000 |
|
||
| discography known | -0.392 |
|
||
| release recency 4w | +0.435 |
|
||
| release recency 12w | +0.680 |
|
||
| log releases last 52w | -0.186 |
|
||
| releases in the window | -0.216 |
|
||
| releases in window x not in rotation now | +0.836 |
|
||
| DR: weeks since last add 13-25 | +0.095 |
|
||
| DR: last add 26+ weeks ago | -0.642 |
|
||
| DR: no add in the panel yet | -0.845 |
|
||
| Airplay Top-20: last new song 13-25 weeks ago | +0.278 |
|
||
| Airplay Top-20: 0.5-1.25x own usual gap | +0.229 |
|
||
| Airplay Top-20: last new song 2+ years ago | -0.527 |
|
||
| log archive plays per 100 sampled hours (2012-21) | +0.289 |
|
||
| log DR lists with the artist before t (2023-) | +0.209 |
|
||
| DR list recency 26w | -0.924 |
|
||
| DR lists known before t at all | +0.039 |
|
||
| NMF Denmark in the window | +0.940 |
|
||
| NMF Denmark top 10 in the window | +0.408 |
|
||
| log NMF Denmark placements last 52w | -0.068 |
|
||
| share of releases on NMF Denmark (2 years) | -0.417 |
|
||
| season index of the window (log, IFPI 2002-24) | +0.412 |
|
||
|
||
| Per-channel feature | Coefficient |
|
||
|---|---|
|
||
| intercept | -7.117 |
|
||
| log plays t | +1.106 |
|
||
| log plays t-1 | +0.141 |
|
||
| log mean plays 4w | +0.113 |
|
||
| log mean plays 12w | -0.871 |
|
||
| log max plays 12w | +1.022 |
|
||
| log other channels t | +0.328 |
|
||
| log other channels 4w | -0.095 |
|
||
| log P3 t | +0.087 |
|
||
| weeks since played (/12) | -0.509 |
|
||
| Track Top-40 score | -0.132 |
|
||
| Airplay Top-20 score | -1.376 |
|
||
| chart entry recency 6w | +0.616 |
|
||
| chart entry recency 26w | -0.506 |
|
||
| log Track Top-40 hits | +0.258 |
|
||
| log Airplay Top-20 hits | -0.398 |
|
||
| Danish | +0.171 |
|
||
| group | -0.636 |
|
||
| female | +0.136 |
|
||
| career years (/50) | -1.498 |
|
||
| debut unknown | -0.333 |
|
||
| Christmas target week | +0.000 |
|
||
| Danish x P3 | +0.064 |
|
||
| Danish x P4 | +0.058 |
|
||
| Danish x P6 | +0.111 |
|
||
| ch p4kbh | +1.094 |
|
||
| ch p4sjaelland | +0.966 |
|
||
| ch p4fyn | +0.946 |
|
||
| ch p4syd | +0.975 |
|
||
| ch p4trekanten | +0.985 |
|
||
| ch p4vest | +1.083 |
|
||
| ch p4aarhus | +0.944 |
|
||
| ch p4nord | +1.006 |
|
||
| ch p4bornholm | +0.995 |
|
||
| ch p4esbjerg | +0.972 |
|
||
| ch p5 | +2.081 |
|
||
| ch p6beat | +1.296 |
|
||
| ch p8jazz | +2.312 |
|
||
| in rotation on channel now | +0.350 |
|
||
| in rotation anywhere now | -0.334 |
|
||
| station groups in rotation now (/5) | +0.041 |
|
||
| run on channel = 1 week | +0.660 |
|
||
| run = 2 weeks | +0.439 |
|
||
| run = 3-5 weeks | -0.056 |
|
||
| run >= 6 weeks | -0.693 |
|
||
| near threshold (>= half of tau, not in rotation) | +1.381 |
|
||
| plays / tau (capped 3, /3) | -3.363 |
|
||
| weeks since in rotation on channel (/12) | -0.302 |
|
||
| discography known | -0.052 |
|
||
| release recency 4w | +1.017 |
|
||
| release recency 12w | +0.014 |
|
||
| log releases last 52w | +0.009 |
|
||
| releases in the window | -0.136 |
|
||
| releases in window x not in rotation now | +0.755 |
|
||
| DR: weeks since last add 13-25 | +0.106 |
|
||
| DR: last add 26+ weeks ago | -0.081 |
|
||
| DR: no add in the panel yet | -0.483 |
|
||
| Airplay Top-20: last new song 13-25 weeks ago | +0.194 |
|
||
| Airplay Top-20: 0.5-1.25x own usual gap | -0.013 |
|
||
| Airplay Top-20: last new song 2+ years ago | -0.418 |
|
||
| log archive plays per 100 sampled hours (2012-21) | +0.347 |
|
||
| log DR lists with the artist before t (2023-) | +0.023 |
|
||
| DR list recency 26w | -0.442 |
|
||
| DR lists known before t at all | +0.066 |
|
||
| NMF Denmark in the window | +0.779 |
|
||
| NMF Denmark top 10 in the window | +0.580 |
|
||
| log NMF Denmark placements last 52w | -0.089 |
|
||
| share of releases on NMF Denmark (2 years) | +0.212 |
|
||
| season index of the window (log, IFPI 2002-24) | +1.211 |
|
||
| share of the artist's DR plays on this station (before t) | +0.986 |
|
||
| log DR plays on this station (before t) | +0.050 |
|
||
| never played on this station (before t) | -2.825 |
|
||
| genre fit for this station | +0.091 |
|
||
|