Files
Airplay-Predictor/docs/BACKTEST-ADDS.md
T

6.6 KiB
Raw Blame History

Backtest: new adds

An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.

Window starting h = 0 weeks ahead (17 forecast origins)

Base rates: per channel 0.32% of artist-channels get an add in the window; on any channel 1.8% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0214 0.006 0.0913 0.021
rate by last-12-weeks level 0.0147 0.093 0.0781 0.072
logit 0.0119 0.229 0.0679 0.204

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.441 (0.427 to 0.451) 0.187 (0.174 to 0.191)
any channel 0.256 (0.238 to 0.275) 0.130 (0.120 to 0.141)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1876920 0.001 0.002
0.1–0.2 6259 0.142 0.211
0.2–0.3 2324 0.243 0.272
0.3–0.4 933 0.342 0.279
0.4–0.5 710 0.450 0.304
0.5–0.6 613 0.550 0.303
0.6–0.7 445 0.647 0.353
0.7–0.8 283 0.748 0.502
0.8–0.9 237 0.844 0.422
0.9–1.0 44 0.921 0.545

Window starting h = 4 weeks ahead (13 forecast origins)

Base rates: per channel 0.33% of artist-channels get an add in the window; on any channel 1.8% of artists.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0213 0.014 0.0896 0.031
rate by last-12-weeks level 0.0158 0.084 0.0801 0.066
logit 0.0147 0.134 0.0744 0.143

Skill of the logit (90% block-bootstrap interval over origins):

vs in-rotation-now vs last-12-weeks level
per channel 0.310 (0.302 to 0.316) 0.070 (0.042 to 0.087)
any channel 0.170 (0.152 to 0.186) 0.071 (0.064 to 0.080)

Calibration, per channel:

Predicted Rows Mean predicted Observed
0.0–0.1 1434699 0.002 0.002
0.1–0.2 5169 0.142 0.147
0.2–0.3 2108 0.243 0.164
0.3–0.4 980 0.344 0.198
0.4–0.5 621 0.446 0.203
0.5–0.6 404 0.544 0.223
0.6–0.7 255 0.644 0.475
0.7–0.8 74 0.751 0.203
0.8–0.9 42 0.837 0.310

Coefficients, window starting now (h = 0), all closed windows

Any-channel feature Coefficient
intercept -4.804
log DR plays t +0.988
log DR plays 4w +0.335
log DR plays 12w -0.407
log max DR plays 12w +0.696
in rotation anywhere now -1.287
station groups in rotation now (/5) -3.848
current run anywhere (/12) -1.865
weeks since in rotation anywhere (/12) -0.272
Track Top-40 score +0.074
Airplay Top-20 score -0.687
chart entry recency 6w +1.656
chart entry recency 26w -1.565
log Track Top-40 hits -0.209
log Airplay Top-20 hits -0.197
Danish +0.499
group -0.341
female -0.140
career years (/50) +0.297
debut unknown +0.622
Christmas in window +0.000
discography known -0.354
release recency 4w +0.199
release recency 12w +0.894
log releases last 52w -0.200
releases in the window -1.019
releases in window x not in rotation now +1.573
DR: weeks since last add 13-25 +0.099
DR: last add 26+ weeks ago -0.438
DR: no add in the panel yet -0.948
Airplay Top-20: last new song 13-25 weeks ago +0.299
Airplay Top-20: 0.5-1.25x own usual gap -0.018
Airplay Top-20: last new song 2+ years ago -0.529
log archive plays per 100 sampled hours (2012-21) +0.366
log DR lists with the artist before t (2023-) +0.332
DR list recency 26w -0.621
DR lists known before t at all -0.166
NMF Denmark in the window +0.985
NMF Denmark top 10 in the window +0.444
log NMF Denmark placements last 52w -0.079
share of releases on NMF Denmark (2 years) -0.623
Per-channel feature Coefficient
intercept -7.632
log plays t +0.821
log plays t-1 +0.119
log mean plays 4w -0.008
log mean plays 12w -0.531
log max plays 12w +1.318
log other channels t +0.235
log other channels 4w -0.087
log P3 t +0.037
weeks since played (/12) -1.762
Track Top-40 score -0.273
Airplay Top-20 score -0.757
chart entry recency 6w +0.747
chart entry recency 26w -0.459
log Track Top-40 hits +0.226
log Airplay Top-20 hits -0.351
Danish +0.235
group -0.650
female +0.012
career years (/50) -1.128
debut unknown -0.056
Christmas target week +0.000
Danish x P3 -0.051
Danish x P4 +0.302
Danish x P6 +0.004
ch p4kbh +0.695
ch p4sjaelland +0.514
ch p4fyn +0.499
ch p4syd +0.544
ch p4trekanten +0.581
ch p4vest +0.611
ch p4aarhus +0.526
ch p4nord +0.587
ch p4bornholm +0.536
ch p4esbjerg +0.555
ch p5 +1.820
ch p6beat +1.118
ch p8jazz +2.402
in rotation on channel now -0.952
in rotation anywhere now +0.118
station groups in rotation now (/5) -1.127
run on channel = 1 week +0.893
run = 2 weeks +0.413
run = 3-5 weeks -1.287
run >= 6 weeks -0.971
near threshold (>= half of tau, not in rotation) +1.421
plays / tau (capped 3, /3) -0.967
weeks since in rotation on channel (/12) +0.425
discography known -0.042
release recency 4w +0.840
release recency 12w +0.180
log releases last 52w -0.059
releases in the window -0.478
releases in window x not in rotation now +1.007
DR: weeks since last add 13-25 +0.007
DR: last add 26+ weeks ago +0.310
DR: no add in the panel yet -0.072
Airplay Top-20: last new song 13-25 weeks ago +0.084
Airplay Top-20: 0.5-1.25x own usual gap +0.103
Airplay Top-20: last new song 2+ years ago -0.240
log archive plays per 100 sampled hours (2012-21) +0.179
log DR lists with the artist before t (2023-) +0.175
DR list recency 26w -0.254
DR lists known before t at all -0.234
NMF Denmark in the window +0.853
NMF Denmark top 10 in the window +0.479
log NMF Denmark placements last 52w -0.122
share of releases on NMF Denmark (2 years) -0.189