Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

5.9 KiB
Raw Blame History

Backtest: new adds

An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.

Window starting h = 0 weeks ahead (17 forecast origins)

Base rates: per channel 0.36% of artist-channels get an add in the window; on any channel 2.0% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0234 0.006 0.0994 0.023
rate by last-12-weeks level 0.0162 0.097 0.0863 0.073
logit 0.0130 0.245 0.0747 0.199

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.444 (0.427 to 0.458) 0.196 (0.186 to 0.199)
any channel 0.249 (0.230 to 0.265) 0.134 (0.124 to 0.142)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1669178 0.002 0.002
0.1–0.2 5763 0.143 0.193
0.2–0.3 2302 0.244 0.232
0.3–0.4 1408 0.345 0.294
0.4–0.5 872 0.441 0.336
0.5–0.6 583 0.547 0.453
0.6–0.7 306 0.653 0.461
0.7–0.8 189 0.739 0.339
0.8–0.9 152 0.839 0.658
0.9–1.0 3 0.902 1.000

Window starting h = 4 weeks ahead (13 forecast origins)

Base rates: per channel 0.37% of artist-channels get an add in the window; on any channel 2.0% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0232 0.014 0.0979 0.031
rate by last-12-weeks level 0.0174 0.087 0.0887 0.066
logit 0.0160 0.134 0.0805 0.142

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.313 (0.305 to 0.323) 0.081 (0.057 to 0.096)
any channel 0.178 (0.162 to 0.190) 0.092 (0.084 to 0.098)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1275342 0.002 0.002
0.1–0.2 6075 0.141 0.120
0.2–0.3 2066 0.244 0.169
0.3–0.4 812 0.342 0.197
0.4–0.5 509 0.447 0.287
0.5–0.6 332 0.539 0.455
0.6–0.7 110 0.650 0.273
0.7–0.8 36 0.737 0.278
0.8–0.9 2 0.841 0.000

Coefficients, window starting now (h = 0), all closed windows

Any-channel feature Coefficient
intercept -5.107
log DR plays t +0.979
log DR plays 4w +0.245
log DR plays 12w -0.284
log max DR plays 12w +0.420
in rotation anywhere now -1.359
station groups in rotation now (/5) -3.411
current run anywhere (/12) -1.907
weeks since in rotation anywhere (/12) -0.348
Track Top-40 score +0.147
Airplay Top-20 score -0.633
chart entry recency 6w +1.325
chart entry recency 26w -1.443
log Track Top-40 hits -0.083
log Airplay Top-20 hits -0.057
Danish +0.505
group -0.363
female -0.169
career years (/50) +0.108
debut unknown +0.514
Christmas in window +0.000
discography known +0.877
release recency 4w +0.345
release recency 12w +0.630
log releases last 52w -0.191
releases in the window -0.425
releases in window x not in rotation now +1.334
DR: weeks since last add 13-25 -0.015
DR: last add 26+ weeks ago -0.511
DR: no add in the panel yet -0.998
Airplay Top-20: last new song 13-25 weeks ago +0.255
Airplay Top-20: 0.5-1.25x own usual gap -0.106
Airplay Top-20: last new song 2+ years ago -0.548
Per-channel feature Coefficient
intercept -8.129
log plays t +0.772
log plays t-1 +0.119
log mean plays 4w +0.121
log mean plays 12w -0.700
log max plays 12w +1.336
log other channels t +0.210
log other channels 4w -0.129
log P3 t +0.121
weeks since played (/12) -1.250
Track Top-40 score -0.315
Airplay Top-20 score -0.715
chart entry recency 6w +0.400
chart entry recency 26w -0.160
log Track Top-40 hits +0.273
log Airplay Top-20 hits -0.269
Danish +0.222
group -0.661
female +0.019
career years (/50) -1.152
debut unknown -0.069
Christmas target week +0.000
Danish x P3 +0.010
Danish x P4 +0.343
Danish x P6 -0.044
ch p4kbh +0.293
ch p4sjaelland +0.344
ch p4fyn +0.357
ch p4syd +0.428
ch p4trekanten +0.408
ch p4vest +0.465
ch p4aarhus +0.378
ch p4nord +0.230
ch p4bornholm +0.373
ch p4esbjerg +0.398
ch p5 +1.665
ch p6beat +1.017
ch p8jazz +2.213
in rotation on channel now -1.017
in rotation anywhere now +0.188
station groups in rotation now (/5) -1.490
run on channel = 1 week +0.936
run = 2 weeks +0.398
run = 3-5 weeks -1.398
run >= 6 weeks -0.954
near threshold (>= half of tau, not in rotation) +1.429
plays / tau (capped 3, /3) -0.704
weeks since in rotation on channel (/12) +0.497
discography known +0.741
release recency 4w +0.924
release recency 12w +0.021
log releases last 52w -0.185
releases in the window +0.226
releases in window x not in rotation now +0.822
DR: weeks since last add 13-25 +0.070
DR: last add 26+ weeks ago +0.247
DR: no add in the panel yet -0.105
Airplay Top-20: last new song 13-25 weeks ago +0.171
Airplay Top-20: 0.5-1.25x own usual gap +0.032
Airplay Top-20: last new song 2+ years ago -0.254