Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

4.8 KiB
Raw Blame History

Backtest: new adds

An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.

Window starting h = 0 weeks ahead (17 forecast origins)

Base rates: per channel 0.75% of artist-channels get an add in the window; on any channel 4.2% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0440 0.012 0.1758 0.042
rate by last-12-weeks level 0.0328 0.099 0.1656 0.083
logit 0.0273 0.230 0.1563 0.156

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.381 (0.361 to 0.396) 0.168 (0.148 to 0.181)
any channel 0.111 (0.099 to 0.120) 0.056 (0.040 to 0.068)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 760136 0.003 0.004
0.1–0.2 4598 0.148 0.180
0.2–0.3 3797 0.246 0.219
0.3–0.4 1815 0.345 0.297
0.4–0.5 1080 0.444 0.359
0.5–0.6 715 0.548 0.365
0.6–0.7 252 0.643 0.476
0.7–0.8 149 0.742 0.470
0.8–0.9 6 0.809 0.000

Window starting h = 4 weeks ahead (13 forecast origins)

Base rates: per channel 0.77% of artist-channels get an add in the window; on any channel 4.3% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0438 0.019 0.1752 0.053
rate by last-12-weeks level 0.0347 0.091 0.1689 0.078
logit 0.0326 0.139 0.1629 0.127

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.257 (0.242 to 0.276) 0.062 (0.043 to 0.077)
any channel 0.070 (0.063 to 0.075) 0.035 (0.020 to 0.046)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 579503 0.004 0.005
0.1–0.2 6709 0.143 0.118
0.2–0.3 2781 0.243 0.193
0.3–0.4 1213 0.341 0.232
0.4–0.5 375 0.442 0.291
0.5–0.6 131 0.542 0.198
0.6–0.7 42 0.637 0.571
0.7–0.8 18 0.721 0.667

Coefficients, window starting now (h = 0), all closed windows

Any-channel feature Coefficient
intercept -1.503
log DR plays t +0.669
log DR plays 4w +0.119
log DR plays 12w -0.428
log max DR plays 12w +0.140
in rotation anywhere now -2.146
channels in rotation now (/14) -2.288
current run anywhere (/12) -1.783
weeks since in rotation anywhere (/12) -2.397
Track Top-40 score +0.248
Airplay Top-20 score +0.210
chart entry recency 6w +0.482
chart entry recency 26w -0.328
log Track Top-40 hits +0.002
log Airplay Top-20 hits -0.280
Danish +0.500
group -0.468
female -0.142
career years (/50) -0.250
debut unknown +0.129
Christmas in window +0.000
Per-channel feature Coefficient
intercept -6.792
log plays t +0.665
log plays t-1 +0.129
log mean plays 4w +0.169
log mean plays 12w -0.662
log max plays 12w +1.391
log other channels t +0.257
log other channels 4w -0.214
log P3 t +0.174
weeks since played (/12) -0.733
Track Top-40 score -0.563
Airplay Top-20 score -0.416
chart entry recency 6w +0.160
chart entry recency 26w +0.288
log Track Top-40 hits +0.260
log Airplay Top-20 hits -0.324
Danish +0.295
group -0.840
female +0.007
career years (/50) -1.415
debut unknown -0.130
Christmas target week +0.000
Danish x P3 -0.176
Danish x P4 +0.154
Danish x P6 -0.213
ch p4kbh +0.236
ch p4sjaelland +0.256
ch p4fyn +0.235
ch p4syd +0.297
ch p4trekanten +0.272
ch p4vest +0.361
ch p4aarhus +0.297
ch p4nord +0.186
ch p4bornholm +0.278
ch p4esbjerg +0.290
ch p5 +1.468
ch p6beat +0.871
ch p8jazz +1.960
in rotation on channel now -1.352
in rotation anywhere now +0.104
channels in rotation now (/14) -0.674
run on channel = 1 week +0.944
run = 2 weeks +0.352
run = 3-5 weeks -1.570
run >= 6 weeks -1.078
near threshold (>= half of tau, not in rotation) +1.389
plays / tau (capped 3, /3) -0.664
weeks since in rotation on channel (/12) +0.421