Files
Airplay-Predictor/docs/BACKTEST-EVENTS.md
T

6.0 KiB
Raw Blame History

Backtest: if the artist releases a new song, will DR add it?

Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within 2 week(s) from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 3269 release events scored, 63 (1.9%) got an add on some DR channel.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0240 0.014 0.0923 0.062
rate by last-12-weeks level 0.0170 0.082 0.0721 0.140
release timing only 0.0262 0.003 0.0964 0.020
model, calibrated on the last 6 origins (pre-release) 0.0179 0.085 0.0723 0.179
model, raw (pre-release) 0.0178 0.097 0.0676 0.245
model + pooled Platt (pre-release) 0.0172 0.098 0.0683 0.232
model + pooled isotonic (pre-release) 0.0175 0.088 0.0689 0.252
model, NMF known (post-release) 0.0180 0.090 0.0723 0.170

Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):

Model Per channel vs last-12-weeks Per channel vs release timing Any channel vs last-12-weeks Any channel vs release timing
model, calibrated on the last 6 origins (pre-release) -0.054 (-0.109 to -0.010) +0.315 (+0.264 to +0.373) -0.002 (-0.006 to +0.060) +0.250 (+0.237 to +0.299)
model + pooled Platt (pre-release) -0.010 (-0.041 to +0.026) +0.343 (+0.296 to +0.404) +0.053 (+0.000 to +0.115) +0.291 (+0.243 to +0.343)
model + pooled isotonic (pre-release) -0.028 (-0.082 to +0.022) +0.331 (+0.248 to +0.418) +0.045 (-0.000 to +0.098) +0.286 (+0.228 to +0.338)

Calibration, any channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 3159 / 0.014 / 0.012 3108 / 0.011 / 0.009 3100 / 0.009 / 0.009
0.1–0.2 80 / 0.140 / 0.212 103 / 0.140 / 0.165 105 / 0.137 / 0.152
0.2–0.3 15 / 0.259 / 0.200 23 / 0.227 / 0.217 33 / 0.227 / 0.273
0.3–0.4 10 / 0.337 / 0.400 17 / 0.345 / 0.353 9 / 0.353 / 0.222
0.4–0.5 2 / 0.440 / 0.000 8 / 0.438 / 0.250 7 / 0.445 / 0.429
0.5–0.6 1 / 0.517 / 0.000 5 / 0.534 / 0.200 4 / 0.561 / 0.000
0.6–0.7 2 / 0.658 / 0.000 4 / 0.631 / 0.750 5 / 0.644 / 0.200
0.7–0.8 — — 1 / 0.709 / 1.000
0.8–0.9 — 1 / 0.813 / 0.000 5 / 0.839 / 0.600

Calibration, per channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 45525 / 0.003 / 0.003 45392 / 0.002 / 0.003 45368 / 0.003 / 0.003
0.1–0.2 126 / 0.135 / 0.135 182 / 0.133 / 0.082 301 / 0.124 / 0.083
0.2–0.3 39 / 0.247 / 0.256 95 / 0.246 / 0.168 36 / 0.242 / 0.056
0.3–0.4 54 / 0.340 / 0.037 54 / 0.333 / 0.093 32 / 0.341 / 0.094
0.4–0.5 8 / 0.467 / 0.125 25 / 0.461 / 0.160 10 / 0.447 / 0.400
0.5–0.6 9 / 0.533 / 0.000 12 / 0.521 / 0.250 8 / 0.554 / 0.375
0.6–0.7 1 / 0.657 / 0.000 4 / 0.660 / 0.000 7 / 0.650 / 0.000
0.7–0.8 4 / 0.754 / 0.250 2 / 0.730 / 0.500 4 / 0.734 / 0.250

Quiet artists (out of rotation everywhere for 26 weeks) releasing, any channel:

Group Events Adds Rate Mean predicted (raw) Mean predicted (isotonic) Log-loss (isotonic) PR-AUC (raw)
former favourites (4+ weeks on DR 2019-24) 326 7 0.021 0.025 0.030 0.1028 0.095
other quiet artists 2616 15 0.006 0.007 0.008 0.0313 0.126
Any-channel feature (all events) Coefficient
intercept -5.988
log DR plays t +0.677
log DR plays 4w +0.897
log DR plays 12w -0.425
log max DR plays 12w +0.233
in rotation anywhere now -0.575
station groups in rotation now (/5) -0.137
current run anywhere (/12) -1.280
weeks since in rotation anywhere (/12) -0.016
Track Top-40 score +0.603
Airplay Top-20 score -0.700
chart entry recency 6w +0.172
chart entry recency 26w -0.684
log Track Top-40 hits -0.283
log Airplay Top-20 hits -0.147
Danish +0.830
group -0.078
female -0.173
career years (/50) +0.646
debut unknown +0.019
Christmas in window +0.000
discography known -0.000
release recency 4w -0.157
release recency 12w -1.118
log releases last 52w -0.285
releases in the window -0.000
releases in window x not in rotation now +0.575
DR: weeks since last add 13-25 -0.493
DR: last add 26+ weeks ago -0.099
DR: no add in the panel yet -0.708
Airplay Top-20: last new song 13-25 weeks ago +0.826
Airplay Top-20: 0.5-1.25x own usual gap -0.176
Airplay Top-20: last new song 2+ years ago +0.334
log archive plays per 100 sampled hours (2012-21) +0.618
log DR lists with the artist before t (2023-) +0.156
DR list recency 26w -0.607
DR lists known before t at all -0.210
quiet former favourite (4+ weeks on DR 2019-24, out of rotation 26w) -0.665
NMF Denmark in the window +0.943
NMF Denmark top 10 in the window +0.202
log NMF Denmark placements last 52w +0.251
share of releases on NMF Denmark (2 years) +0.020
season index of the window (log, IFPI 2002-24) -0.049