Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

7.2 KiB
Raw Blame History

Backtest: new adds

An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.

Window starting h = 0 weeks ahead (17 forecast origins)

Base rates: per channel 0.43% of artist-channels get an add in the window; on any channel 2.4% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0255 0.032 0.1058 0.067
rate by last-12-weeks level 0.0177 0.147 0.0909 0.133
logit (pre-release: NMF from history) 0.0147 0.258 0.0800 0.251
logit (post-release: NMF known) 0.0145 0.266 0.0799 0.259

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.424 (0.413 to 0.437) 0.171 (0.168 to 0.178)
any channel 0.244 (0.234 to 0.257) 0.120 (0.115 to 0.127)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1868823 0.002 0.002
0.1–0.2 9096 0.142 0.160
0.2–0.3 3991 0.247 0.230
0.3–0.4 2858 0.344 0.215
0.4–0.5 1615 0.448 0.310
0.5–0.6 1081 0.544 0.457
0.6–0.7 728 0.646 0.475
0.7–0.8 429 0.749 0.441
0.8–0.9 104 0.834 0.346
0.9–1.0 43 0.917 0.930

Window starting h = 4 weeks ahead (13 forecast origins)

Base rates: per channel 0.44% of artist-channels get an add in the window; on any channel 2.4% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0263 0.032 0.1051 0.083
rate by last-12-weeks level 0.0193 0.114 0.0937 0.136
logit (pre-release: NMF from history) 0.0176 0.166 0.0857 0.208
logit (post-release: NMF known) 0.0172 0.175 0.0856 0.218

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.330 (0.315 to 0.337) 0.091 (0.065 to 0.106)
any channel 0.185 (0.174 to 0.194) 0.086 (0.081 to 0.090)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1428760 0.002 0.003
0.1–0.2 7389 0.141 0.129
0.2–0.3 3960 0.247 0.186
0.3–0.4 2024 0.344 0.199
0.4–0.5 1134 0.444 0.283
0.5–0.6 604 0.546 0.224
0.6–0.7 394 0.643 0.459
0.7–0.8 83 0.732 0.277
0.8–0.9 3 0.833 1.000
0.9–1.0 1 0.907 1.000

Coefficients, window starting now (h = 0), all closed windows

Any-channel feature Coefficient
intercept -4.213
log DR plays t +0.883
log DR plays 4w +0.253
log DR plays 12w -0.263
log max DR plays 12w +0.594
in rotation anywhere now -1.148
station groups in rotation now (/5) +0.297
current run anywhere (/12) -1.212
weeks since in rotation anywhere (/12) -0.749
Track Top-40 score +0.427
Airplay Top-20 score -1.332
chart entry recency 6w +1.210
chart entry recency 26w -0.653
log Track Top-40 hits -0.200
log Airplay Top-20 hits -0.069
Danish +0.462
group -0.240
female -0.102
career years (/50) +0.210
debut unknown +0.497
Christmas in window +0.000
discography known -0.325
release recency 4w +0.440
release recency 12w +0.670
log releases last 52w -0.188
releases in the window -0.218
releases in window x not in rotation now +0.854
DR: weeks since last add 13-25 +0.095
DR: last add 26+ weeks ago -0.642
DR: no add in the panel yet -0.847
Airplay Top-20: last new song 13-25 weeks ago +0.279
Airplay Top-20: 0.5-1.25x own usual gap +0.231
Airplay Top-20: last new song 2+ years ago -0.529
log archive plays per 100 sampled hours (2012-21) +0.288
log DR lists with the artist before t (2023-) +0.209
DR list recency 26w -0.929
DR lists known before t at all +0.040
NMF Denmark in the window +0.941
NMF Denmark top 10 in the window +0.416
log NMF Denmark placements last 52w -0.071
share of releases on NMF Denmark (2 years) -0.415
season index of the window (log, IFPI 2002-24) +0.403
Per-channel feature Coefficient
intercept -7.101
log plays t +1.122
log plays t-1 +0.142
log mean plays 4w +0.113
log mean plays 12w -0.862
log max plays 12w +1.014
log other channels t +0.332
log other channels 4w -0.100
log P3 t +0.085
weeks since played (/12) -0.437
Track Top-40 score -0.127
Airplay Top-20 score -1.373
chart entry recency 6w +0.625
chart entry recency 26w -0.512
log Track Top-40 hits +0.257
log Airplay Top-20 hits -0.395
Danish +0.162
group -0.629
female +0.127
career years (/50) -1.500
debut unknown -0.289
Christmas target week +0.000
Danish x P3 +0.072
Danish x P4 +0.085
Danish x P6 +0.154
ch p4kbh +1.100
ch p4sjaelland +0.968
ch p4fyn +0.948
ch p4syd +0.976
ch p4trekanten +0.992
ch p4vest +1.078
ch p4aarhus +0.934
ch p4nord +1.002
ch p4bornholm +0.986
ch p4esbjerg +0.968
ch p5 +2.081
ch p6beat +1.295
ch p8jazz +2.330
in rotation on channel now +0.358
in rotation anywhere now -0.338
station groups in rotation now (/5) +0.043
run on channel = 1 week +0.663
run = 2 weeks +0.441
run = 3-5 weeks -0.055
run >= 6 weeks -0.691
near threshold (>= half of tau, not in rotation) +1.376
plays / tau (capped 3, /3) -3.407
weeks since in rotation on channel (/12) -0.300
discography known -0.069
release recency 4w +1.033
release recency 12w -0.008
log releases last 52w -0.003
releases in the window -0.143
releases in window x not in rotation now +0.782
DR: weeks since last add 13-25 +0.107
DR: last add 26+ weeks ago -0.085
DR: no add in the panel yet -0.486
Airplay Top-20: last new song 13-25 weeks ago +0.194
Airplay Top-20: 0.5-1.25x own usual gap -0.009
Airplay Top-20: last new song 2+ years ago -0.417
log archive plays per 100 sampled hours (2012-21) +0.348
log DR lists with the artist before t (2023-) +0.028
DR list recency 26w -0.444
DR lists known before t at all +0.055
NMF Denmark in the window +0.805
NMF Denmark top 10 in the window +0.579
log NMF Denmark placements last 52w -0.077
share of releases on NMF Denmark (2 years) +0.166
season index of the window (log, IFPI 2002-24) +1.240
share of the artist's DR plays on this station (before t) +0.963
log DR plays on this station (before t) +0.049
never played on this station (before t) -2.888
genre fit for this station +0.108