Files
Airplay-Predictor/docs/BACKTEST-EVENTS.md
T

5.6 KiB
Raw Blame History

Backtest: if the artist releases a new song, will DR add it?

Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within 2 week(s) from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 3193 release events scored, 63 (2.0%) got an add on some DR channel.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0245 0.015 0.0941 0.062
rate by last-12-weeks level 0.0174 0.082 0.0738 0.140
release timing only 0.0267 0.003 0.0982 0.021
model, calibrated on the last 6 origins (pre-release) 0.0183 0.080 0.0748 0.170
model, raw (pre-release) 0.0182 0.091 0.0696 0.233
model + pooled Platt (pre-release) 0.0177 0.094 0.0704 0.224
model + pooled isotonic (pre-release) 0.0176 0.091 0.0706 0.235
model, NMF known (post-release) 0.0184 0.085 0.0750 0.163

Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):

Model Per channel vs last-12-weeks Per channel vs release timing Any channel vs last-12-weeks Any channel vs release timing
model, calibrated on the last 6 origins (pre-release) -0.053 (-0.131 to +0.002) +0.313 (+0.257 to +0.378) -0.013 (-0.017 to +0.048) +0.239 (+0.226 to +0.288)
model + pooled Platt (pre-release) -0.015 (-0.053 to +0.020) +0.338 (+0.286 to +0.399) +0.046 (-0.008 to +0.107) +0.283 (+0.234 to +0.334)
model + pooled isotonic (pre-release) -0.010 (-0.095 to +0.039) +0.342 (+0.234 to +0.430) +0.044 (-0.011 to +0.100) +0.281 (+0.223 to +0.335)

Calibration, any channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 3081 / 0.015 / 0.013 3033 / 0.011 / 0.010 2979 / 0.009 / 0.008
0.1–0.2 82 / 0.136 / 0.207 107 / 0.140 / 0.168 169 / 0.135 / 0.160
0.2–0.3 16 / 0.245 / 0.312 20 / 0.239 / 0.300 16 / 0.238 / 0.188
0.3–0.4 6 / 0.316 / 0.333 12 / 0.340 / 0.250 8 / 0.344 / 0.375
0.4–0.5 4 / 0.451 / 0.000 13 / 0.437 / 0.231 8 / 0.436 / 0.250
0.5–0.6 2 / 0.533 / 0.000 6 / 0.566 / 0.667 4 / 0.563 / 0.250
0.6–0.7 2 / 0.636 / 0.000 1 / 0.605 / 0.000 2 / 0.620 / 0.000
0.7–0.8 — — 5 / 0.775 / 0.600
0.8–0.9 — 1 / 0.818 / 0.000 2 / 0.827 / 0.500

Calibration, per channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 44463 / 0.004 / 0.003 44305 / 0.002 / 0.003 44334 / 0.003 / 0.003
0.1–0.2 125 / 0.136 / 0.128 229 / 0.137 / 0.105 278 / 0.123 / 0.090
0.2–0.3 69 / 0.259 / 0.159 86 / 0.242 / 0.093 30 / 0.245 / 0.067
0.3–0.4 21 / 0.337 / 0.048 42 / 0.346 / 0.071 35 / 0.358 / 0.057
0.4–0.5 8 / 0.456 / 0.000 28 / 0.454 / 0.143 13 / 0.448 / 0.308
0.5–0.6 11 / 0.527 / 0.000 6 / 0.540 / 0.500 6 / 0.546 / 0.500
0.6–0.7 3 / 0.658 / 0.333 4 / 0.634 / 0.250 1 / 0.697 / 0.000
0.7–0.8 2 / 0.745 / 0.000 2 / 0.770 / 0.000 5 / 0.754 / 0.200
Any-channel feature (all events) Coefficient
intercept -5.682
log DR plays t +0.672
log DR plays 4w +0.859
log DR plays 12w -0.429
log max DR plays 12w +0.201
in rotation anywhere now -0.575
station groups in rotation now (/5) -0.131
current run anywhere (/12) -1.254
weeks since in rotation anywhere (/12) -0.225
Track Top-40 score +0.602
Airplay Top-20 score -0.736
chart entry recency 6w +0.145
chart entry recency 26w -0.668
log Track Top-40 hits -0.292
log Airplay Top-20 hits -0.136
Danish +0.821
group -0.081
female -0.185
career years (/50) +0.645
debut unknown +0.056
Christmas in window +0.000
discography known -0.000
release recency 4w -0.154
release recency 12w -1.148
log releases last 52w -0.286
releases in the window -0.000
releases in window x not in rotation now +0.575
DR: weeks since last add 13-25 -0.412
DR: last add 26+ weeks ago -0.142
DR: no add in the panel yet -0.811
Airplay Top-20: last new song 13-25 weeks ago +0.833
Airplay Top-20: 0.5-1.25x own usual gap -0.222
Airplay Top-20: last new song 2+ years ago +0.215
log archive plays per 100 sampled hours (2012-21) +0.591
log DR lists with the artist before t (2023-) +0.160
DR list recency 26w -0.506
DR lists known before t at all -0.279
NMF Denmark in the window +0.923
NMF Denmark top 10 in the window +0.249
log NMF Denmark placements last 52w +0.276
share of releases on NMF Denmark (2 years) +0.002
season index of the window (log, IFPI 2002-24) -0.071