Files
Airplay-Predictor/docs/BACKTEST-EVENTS.md
T

5.8 KiB
Raw Blame History

Backtest: if the artist releases a new song, will DR add it?

Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 2140 release events scored, 120 (5.6%) got an add on some DR channel.

Model Log-loss, per channel PR-AUC, per channel Log-loss, any channel PR-AUC, any channel
rate by in-rotation-now 0.0666 0.061 0.2070 0.113
rate by last-12-weeks level 0.0478 0.216 0.1805 0.221
release timing only 0.0719 0.014 0.2166 0.068
model, calibrated on the last 6 origins (pre-release) 0.0431 0.341 0.1881 0.243
model, raw (pre-release) 0.0437 0.326 0.1860 0.252
model + pooled Platt (pre-release) 0.0425 0.337 0.1916 0.221
model + pooled isotonic (pre-release) 0.0439 0.308 0.1871 0.243
model, NMF known (post-release) 0.0413 0.346 0.1853 0.266

Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):

Model Per channel vs last-12-weeks Per channel vs release timing Any channel vs last-12-weeks Any channel vs release timing
model, calibrated on the last 6 origins (pre-release) +0.098 (+0.054 to +0.162) +0.400 (+0.313 to +0.440) -0.042 (-0.068 to +0.041) +0.132 (+0.128 to +0.223)
model + pooled Platt (pre-release) +0.111 (+0.054 to +0.181) +0.408 (+0.330 to +0.440) -0.061 (-0.091 to -0.022) +0.116 (+0.097 to +0.160)
model + pooled isotonic (pre-release) +0.083 (+0.054 to +0.187) +0.390 (+0.333 to +0.438) -0.037 (-0.072 to +0.031) +0.136 (+0.121 to +0.206)

Calibration, any channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 1936 / 0.018 / 0.034 1897 / 0.034 / 0.033 1844 / 0.027 / 0.027
0.1–0.2 124 / 0.137 / 0.234 154 / 0.141 / 0.201 124 / 0.143 / 0.202
0.2–0.3 40 / 0.245 / 0.325 56 / 0.247 / 0.286 90 / 0.251 / 0.200
0.3–0.4 18 / 0.349 / 0.278 20 / 0.355 / 0.300 40 / 0.350 / 0.250
0.4–0.5 13 / 0.442 / 0.385 7 / 0.463 / 0.429 23 / 0.449 / 0.435
0.5–0.6 5 / 0.534 / 0.400 3 / 0.536 / 0.333 8 / 0.571 / 0.500
0.6–0.7 1 / 0.606 / 0.000 2 / 0.625 / 0.000 3 / 0.637 / 0.333
0.7–0.8 3 / 0.732 / 0.333 — 2 / 0.730 / 0.000
0.8–0.9 — 1 / 0.823 / 1.000 5 / 0.824 / 0.600
0.9–1.0 — — 1 / 0.911 / 0.000

Calibration, per channel:

Predicted model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed model + pooled Platt (pre-release): rows / mean predicted / observed model + pooled isotonic (pre-release): rows / mean predicted / observed
0.0–0.1 29258 / 0.004 / 0.007 29241 / 0.004 / 0.007 29307 / 0.004 / 0.007
0.1–0.2 342 / 0.140 / 0.208 347 / 0.143 / 0.182 328 / 0.143 / 0.235
0.2–0.3 136 / 0.246 / 0.206 136 / 0.252 / 0.287 122 / 0.248 / 0.189
0.3–0.4 87 / 0.347 / 0.207 102 / 0.335 / 0.118 81 / 0.344 / 0.185
0.4–0.5 66 / 0.447 / 0.500 44 / 0.450 / 0.500 35 / 0.449 / 0.429
0.5–0.6 10 / 0.527 / 0.000 28 / 0.538 / 0.714 25 / 0.551 / 0.760
0.6–0.7 8 / 0.675 / 0.125 5 / 0.655 / 0.400 4 / 0.652 / 0.250
0.7–0.8 42 / 0.745 / 0.762 42 / 0.757 / 0.714 35 / 0.751 / 0.886
0.8–0.9 2 / 0.845 / 1.000 6 / 0.817 / 0.500 2 / 0.845 / 1.000
0.9–1.0 9 / 0.912 / 1.000 9 / 0.912 / 1.000 21 / 0.954 / 0.476
Any-channel feature (all events) Coefficient
intercept -3.768
log DR plays t +0.560
log DR plays 4w +0.510
log DR plays 12w -0.409
log max DR plays 12w +0.318
in rotation anywhere now -0.474
station groups in rotation now (/5) -0.249
current run anywhere (/12) -1.173
weeks since in rotation anywhere (/12) -0.489
Track Top-40 score -0.265
Airplay Top-20 score -0.317
chart entry recency 6w +0.685
chart entry recency 26w +0.501
log Track Top-40 hits -0.291
log Airplay Top-20 hits -0.120
Danish +0.548
group +0.013
female -0.075
career years (/50) +0.422
debut unknown +0.435
Christmas in window +0.000
discography known -0.000
release recency 4w +0.216
release recency 12w -0.783
log releases last 52w -0.189
releases in the window -0.000
releases in window x not in rotation now +0.474
DR: weeks since last add 13-25 -0.050
DR: last add 26+ weeks ago +0.502
DR: no add in the panel yet -0.677
Airplay Top-20: last new song 13-25 weeks ago +0.465
Airplay Top-20: 0.5-1.25x own usual gap -0.144
Airplay Top-20: last new song 2+ years ago -0.113
log archive plays per 100 sampled hours (2012-21) +0.515
log DR lists with the artist before t (2023-) +0.105
DR list recency 26w -0.984
DR lists known before t at all +0.593
NMF Denmark in the window +0.836
NMF Denmark top 10 in the window +0.682
log NMF Denmark placements last 52w -0.067
share of releases on NMF Denmark (2 years) -0.273
season index of the window (log, IFPI 2002-24) +0.334