Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

12 KiB
Raw Blame History

Backtest: new adds

An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.

Window starting h = 0 weeks ahead (16 forecast origins)

Base rates: per channel 0.46% of artist-channels get an add in the window; on any channel 2.5% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0271 0.032 0.1111 0.069
rate by last-12-weeks level 0.0188 0.148 0.0956 0.134
release timing only 0.0274 0.010 0.1144 0.040
model, calibrated on the last 6 origins (pre-release) 0.0156 0.263 0.0837 0.254
model, raw (pre-release) 0.0155 0.263 0.0854 0.246
model + pooled Platt (pre-release) 0.0156 0.262 0.0855 0.246
model + pooled isotonic (pre-release) 0.0157 0.254 0.0859 0.244
model, NMF known (post-release) 0.0153 0.270 0.0836 0.263

Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):

Model Per channel vs last-12-weeks Per channel vs release timing Any channel vs last-12-weeks Any channel vs release timing
model, calibrated on the last 6 origins (pre-release) +0.174 (+0.167 to +0.176) +0.432 (+0.412 to +0.445) +0.124 (+0.123 to +0.133) +0.269 (+0.242 to +0.289)
model + pooled Platt (pre-release) +0.172 (+0.164 to +0.172) +0.431 (+0.409 to +0.444) +0.105 (+0.091 to +0.112) +0.253 (+0.213 to +0.272)
model + pooled isotonic (pre-release) +0.168 (+0.160 to +0.168) +0.428 (+0.405 to +0.444) +0.102 (+0.086 to +0.112) +0.250 (+0.209 to +0.272)

Calibration, any channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 112229 / 0.010 / 0.013 111004 / 0.012 / 0.012 111041 / 0.011 / 0.012
0.1–0.2 2985 / 0.142 / 0.196 3319 / 0.142 / 0.155 2508 / 0.146 / 0.149
0.2–0.3 1360 / 0.245 / 0.263 1636 / 0.245 / 0.249 2058 / 0.249 / 0.218
0.3–0.4 799 / 0.346 / 0.328 998 / 0.347 / 0.305 1765 / 0.339 / 0.301
0.4–0.5 450 / 0.447 / 0.360 626 / 0.445 / 0.321 489 / 0.442 / 0.309
0.5–0.6 251 / 0.546 / 0.347 355 / 0.548 / 0.293 247 / 0.546 / 0.352
0.6–0.7 162 / 0.647 / 0.407 215 / 0.646 / 0.344 152 / 0.646 / 0.421
0.7–0.8 93 / 0.746 / 0.505 149 / 0.740 / 0.483 73 / 0.740 / 0.479
0.8–0.9 52 / 0.841 / 0.596 74 / 0.837 / 0.554 43 / 0.837 / 0.558
0.9–1.0 11 / 0.924 / 0.182 16 / 0.924 / 0.438 16 / 0.932 / 0.500

Calibration, per channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 1638964 / 0.002 / 0.002 1639990 / 0.002 / 0.002 1637740 / 0.002 / 0.002
0.1–0.2 8541 / 0.142 / 0.165 7975 / 0.141 / 0.176 9301 / 0.135 / 0.149
0.2–0.3 3740 / 0.248 / 0.229 3586 / 0.247 / 0.231 4805 / 0.251 / 0.229
0.3–0.4 2586 / 0.344 / 0.216 2464 / 0.345 / 0.211 3432 / 0.338 / 0.297
0.4–0.5 1570 / 0.449 / 0.336 1465 / 0.449 / 0.374 939 / 0.448 / 0.376
0.5–0.6 984 / 0.545 / 0.450 966 / 0.547 / 0.443 588 / 0.548 / 0.371
0.6–0.7 621 / 0.645 / 0.473 549 / 0.643 / 0.461 328 / 0.644 / 0.424
0.7–0.8 351 / 0.751 / 0.464 358 / 0.751 / 0.455 207 / 0.756 / 0.560
0.8–0.9 91 / 0.835 / 0.352 90 / 0.834 / 0.489 148 / 0.853 / 0.507
0.9–1.0 40 / 0.914 / 0.950 45 / 0.928 / 0.689 —

Window starting h = 4 weeks ahead (12 forecast origins)

Base rates: per channel 0.47% of artist-channels get an add in the window; on any channel 2.5% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0277 0.033 0.1102 0.084
rate by last-12-weeks level 0.0204 0.117 0.0984 0.137
release timing only 0.0286 0.010 0.1146 0.043
model, calibrated on the last 6 origins (pre-release) 0.0186 0.166 0.0898 0.208
model, raw (pre-release) 0.0186 0.166 0.0908 0.200
model + pooled Platt (pre-release) 0.0186 0.166 0.0908 0.200
model + pooled isotonic (pre-release) 0.0186 0.166 0.0908 0.200
model, NMF known (post-release) 0.0183 0.174 0.0896 0.219

Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):

Model Per channel vs last-12-weeks Per channel vs release timing Any channel vs last-12-weeks Any channel vs release timing
model, calibrated on the last 6 origins (pre-release) +0.086 (+0.065 to +0.080) +0.349 (+0.334 to +0.345) +0.088 (+0.084 to +0.095) +0.217 (+0.207 to +0.221)
model + pooled Platt (pre-release) +0.086 (+0.065 to +0.080) +0.349 (+0.334 to +0.345) +0.077 (+0.072 to +0.080) +0.208 (+0.195 to +0.208)
model + pooled isotonic (pre-release) +0.086 (+0.065 to +0.080) +0.349 (+0.334 to +0.345) +0.077 (+0.072 to +0.080) +0.208 (+0.195 to +0.208)

Calibration, any channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 84532 / 0.011 / 0.015 83207 / 0.013 / 0.013 83207 / 0.013 / 0.013
0.1–0.2 2353 / 0.140 / 0.178 2764 / 0.142 / 0.156 2764 / 0.142 / 0.156
0.2–0.3 904 / 0.242 / 0.304 1271 / 0.245 / 0.238 1271 / 0.245 / 0.238
0.3–0.4 479 / 0.347 / 0.311 717 / 0.346 / 0.262 717 / 0.346 / 0.262
0.4–0.5 221 / 0.446 / 0.385 359 / 0.445 / 0.304 359 / 0.445 / 0.304
0.5–0.6 124 / 0.544 / 0.306 188 / 0.547 / 0.372 188 / 0.547 / 0.372
0.6–0.7 83 / 0.640 / 0.325 142 / 0.641 / 0.345 142 / 0.641 / 0.345
0.7–0.8 30 / 0.750 / 0.333 66 / 0.749 / 0.273 66 / 0.749 / 0.273
0.8–0.9 4 / 0.830 / 0.000 16 / 0.835 / 0.312 16 / 0.835 / 0.312

Calibration, per channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 1228298 / 0.002 / 0.003 1228298 / 0.002 / 0.003 1228298 / 0.002 / 0.003
0.1–0.2 6683 / 0.141 / 0.129 6683 / 0.141 / 0.129 6683 / 0.141 / 0.129
0.2–0.3 3587 / 0.247 / 0.188 3587 / 0.247 / 0.188 3587 / 0.247 / 0.188
0.3–0.4 1731 / 0.345 / 0.207 1731 / 0.345 / 0.207 1731 / 0.345 / 0.207
0.4–0.5 956 / 0.443 / 0.285 956 / 0.443 / 0.285 956 / 0.443 / 0.285
0.5–0.6 571 / 0.548 / 0.256 571 / 0.548 / 0.256 571 / 0.548 / 0.256
0.6–0.7 333 / 0.647 / 0.441 333 / 0.647 / 0.441 333 / 0.647 / 0.441
0.7–0.8 55 / 0.734 / 0.255 55 / 0.734 / 0.255 55 / 0.734 / 0.255
0.8–0.9 5 / 0.820 / 0.400 5 / 0.820 / 0.400 5 / 0.820 / 0.400
0.9–1.0 1 / 0.905 / 1.000 1 / 0.905 / 1.000 1 / 0.905 / 1.000

Coefficients, window starting now (h = 0), all closed windows

Any-channel feature Coefficient
intercept -4.141
log DR plays t +0.882
log DR plays 4w +0.253
log DR plays 12w -0.255
log max DR plays 12w +0.583
in rotation anywhere now -1.155
station groups in rotation now (/5) +0.320
current run anywhere (/12) -1.215
weeks since in rotation anywhere (/12) -0.748
Track Top-40 score +0.427
Airplay Top-20 score -1.336
chart entry recency 6w +1.203
chart entry recency 26w -0.646
log Track Top-40 hits -0.201
log Airplay Top-20 hits -0.070
Danish +0.470
group -0.253
female -0.097
career years (/50) +0.222
debut unknown +0.489
Christmas in window +0.000
discography known -0.392
release recency 4w +0.435
release recency 12w +0.680
log releases last 52w -0.186
releases in the window -0.216
releases in window x not in rotation now +0.836
DR: weeks since last add 13-25 +0.095
DR: last add 26+ weeks ago -0.642
DR: no add in the panel yet -0.845
Airplay Top-20: last new song 13-25 weeks ago +0.278
Airplay Top-20: 0.5-1.25x own usual gap +0.229
Airplay Top-20: last new song 2+ years ago -0.527
log archive plays per 100 sampled hours (2012-21) +0.289
log DR lists with the artist before t (2023-) +0.209
DR list recency 26w -0.924
DR lists known before t at all +0.039
NMF Denmark in the window +0.940
NMF Denmark top 10 in the window +0.408
log NMF Denmark placements last 52w -0.068
share of releases on NMF Denmark (2 years) -0.417
season index of the window (log, IFPI 2002-24) +0.412
Per-channel feature Coefficient
intercept -7.117
log plays t +1.106
log plays t-1 +0.141
log mean plays 4w +0.113
log mean plays 12w -0.871
log max plays 12w +1.022
log other channels t +0.328
log other channels 4w -0.095
log P3 t +0.087
weeks since played (/12) -0.509
Track Top-40 score -0.132
Airplay Top-20 score -1.376
chart entry recency 6w +0.616
chart entry recency 26w -0.506
log Track Top-40 hits +0.258
log Airplay Top-20 hits -0.398
Danish +0.171
group -0.636
female +0.136
career years (/50) -1.498
debut unknown -0.333
Christmas target week +0.000
Danish x P3 +0.064
Danish x P4 +0.058
Danish x P6 +0.111
ch p4kbh +1.094
ch p4sjaelland +0.966
ch p4fyn +0.946
ch p4syd +0.975
ch p4trekanten +0.985
ch p4vest +1.083
ch p4aarhus +0.944
ch p4nord +1.006
ch p4bornholm +0.995
ch p4esbjerg +0.972
ch p5 +2.081
ch p6beat +1.296
ch p8jazz +2.312
in rotation on channel now +0.350
in rotation anywhere now -0.334
station groups in rotation now (/5) +0.041
run on channel = 1 week +0.660
run = 2 weeks +0.439
run = 3-5 weeks -0.056
run >= 6 weeks -0.693
near threshold (>= half of tau, not in rotation) +1.381
plays / tau (capped 3, /3) -3.363
weeks since in rotation on channel (/12) -0.302
discography known -0.052
release recency 4w +1.017
release recency 12w +0.014
log releases last 52w +0.009
releases in the window -0.136
releases in window x not in rotation now +0.755
DR: weeks since last add 13-25 +0.106
DR: last add 26+ weeks ago -0.081
DR: no add in the panel yet -0.483
Airplay Top-20: last new song 13-25 weeks ago +0.194
Airplay Top-20: 0.5-1.25x own usual gap -0.013
Airplay Top-20: last new song 2+ years ago -0.418
log archive plays per 100 sampled hours (2012-21) +0.347
log DR lists with the artist before t (2023-) +0.023
DR list recency 26w -0.442
DR lists known before t at all +0.066
NMF Denmark in the window +0.779
NMF Denmark top 10 in the window +0.580
log NMF Denmark placements last 52w -0.089
share of releases on NMF Denmark (2 years) +0.212
season index of the window (log, IFPI 2002-24) +1.211
share of the artist's DR plays on this station (before t) +0.986
log DR plays on this station (before t) +0.050
never played on this station (before t) -2.825
genre fit for this station +0.091