Files
Airplay-Predictor/docs/BACKTEST-EVENTS.md
T

104 lines
6.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Backtest: if the artist releases a new song, will DR add it?
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within 2 week(s) from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 3269 release events scored, 63 (1.9%) got an add on some DR channel.
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|---|---|---|---|---|
| rate by in-rotation-now | 0.0240 | 0.014 | 0.0923 | 0.062 |
| rate by last-12-weeks level | 0.0170 | 0.082 | 0.0721 | 0.140 |
| release timing only | 0.0262 | 0.003 | 0.0964 | 0.020 |
| model, calibrated on the last 6 origins (pre-release) | 0.0179 | 0.085 | 0.0723 | 0.179 |
| model, raw (pre-release) | 0.0178 | 0.097 | 0.0676 | 0.245 |
| model + pooled Platt (pre-release) | 0.0172 | 0.098 | 0.0683 | 0.232 |
| model + pooled isotonic (pre-release) | 0.0175 | 0.088 | 0.0689 | 0.252 |
| model, NMF known (post-release) | 0.0180 | 0.090 | 0.0723 | 0.170 |
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|---|---|---|---|---|
| model, calibrated on the last 6 origins (pre-release) | -0.054 (-0.109 to -0.010) | +0.315 (+0.264 to +0.373) | -0.002 (-0.006 to +0.060) | +0.250 (+0.237 to +0.299) |
| model + pooled Platt (pre-release) | -0.010 (-0.041 to +0.026) | +0.343 (+0.296 to +0.404) | +0.053 (+0.000 to +0.115) | +0.291 (+0.243 to +0.343) |
| model + pooled isotonic (pre-release) | -0.028 (-0.082 to +0.022) | +0.331 (+0.248 to +0.418) | +0.045 (-0.000 to +0.098) | +0.286 (+0.228 to +0.338) |
Calibration, any channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 3159 / 0.014 / 0.012 | 3108 / 0.011 / 0.009 | 3100 / 0.009 / 0.009 |
| 0.1–0.2 | 80 / 0.140 / 0.212 | 103 / 0.140 / 0.165 | 105 / 0.137 / 0.152 |
| 0.2–0.3 | 15 / 0.259 / 0.200 | 23 / 0.227 / 0.217 | 33 / 0.227 / 0.273 |
| 0.3–0.4 | 10 / 0.337 / 0.400 | 17 / 0.345 / 0.353 | 9 / 0.353 / 0.222 |
| 0.4–0.5 | 2 / 0.440 / 0.000 | 8 / 0.438 / 0.250 | 7 / 0.445 / 0.429 |
| 0.5–0.6 | 1 / 0.517 / 0.000 | 5 / 0.534 / 0.200 | 4 / 0.561 / 0.000 |
| 0.6–0.7 | 2 / 0.658 / 0.000 | 4 / 0.631 / 0.750 | 5 / 0.644 / 0.200 |
| 0.7–0.8 | — | — | 1 / 0.709 / 1.000 |
| 0.8–0.9 | — | 1 / 0.813 / 0.000 | 5 / 0.839 / 0.600 |
Calibration, per channel:
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|---|---|---|---|
| 0.0–0.1 | 45525 / 0.003 / 0.003 | 45392 / 0.002 / 0.003 | 45368 / 0.003 / 0.003 |
| 0.1–0.2 | 126 / 0.135 / 0.135 | 182 / 0.133 / 0.082 | 301 / 0.124 / 0.083 |
| 0.2–0.3 | 39 / 0.247 / 0.256 | 95 / 0.246 / 0.168 | 36 / 0.242 / 0.056 |
| 0.3–0.4 | 54 / 0.340 / 0.037 | 54 / 0.333 / 0.093 | 32 / 0.341 / 0.094 |
| 0.4–0.5 | 8 / 0.467 / 0.125 | 25 / 0.461 / 0.160 | 10 / 0.447 / 0.400 |
| 0.5–0.6 | 9 / 0.533 / 0.000 | 12 / 0.521 / 0.250 | 8 / 0.554 / 0.375 |
| 0.6–0.7 | 1 / 0.657 / 0.000 | 4 / 0.660 / 0.000 | 7 / 0.650 / 0.000 |
| 0.7–0.8 | 4 / 0.754 / 0.250 | 2 / 0.730 / 0.500 | 4 / 0.734 / 0.250 |
Quiet artists (out of rotation everywhere for 26 weeks) releasing, any channel:
| Group | Events | Adds | Rate | Mean predicted (raw) | Mean predicted (isotonic) | Log-loss (isotonic) | PR-AUC (raw) |
|---|---|---|---|---|---|---|---|
| former favourites (4+ weeks on DR 2019-24) | 326 | 7 | 0.021 | 0.025 | 0.030 | 0.1028 | 0.095 |
| other quiet artists | 2616 | 15 | 0.006 | 0.007 | 0.008 | 0.0313 | 0.126 |
| Any-channel feature (all events) | Coefficient |
|---|---|
| intercept | -5.988 |
| log DR plays t | +0.677 |
| log DR plays 4w | +0.897 |
| log DR plays 12w | -0.425 |
| log max DR plays 12w | +0.233 |
| in rotation anywhere now | -0.575 |
| station groups in rotation now (/5) | -0.137 |
| current run anywhere (/12) | -1.280 |
| weeks since in rotation anywhere (/12) | -0.016 |
| Track Top-40 score | +0.603 |
| Airplay Top-20 score | -0.700 |
| chart entry recency 6w | +0.172 |
| chart entry recency 26w | -0.684 |
| log Track Top-40 hits | -0.283 |
| log Airplay Top-20 hits | -0.147 |
| Danish | +0.830 |
| group | -0.078 |
| female | -0.173 |
| career years (/50) | +0.646 |
| debut unknown | +0.019 |
| Christmas in window | +0.000 |
| discography known | -0.000 |
| release recency 4w | -0.157 |
| release recency 12w | -1.118 |
| log releases last 52w | -0.285 |
| releases in the window | -0.000 |
| releases in window x not in rotation now | +0.575 |
| DR: weeks since last add 13-25 | -0.493 |
| DR: last add 26+ weeks ago | -0.099 |
| DR: no add in the panel yet | -0.708 |
| Airplay Top-20: last new song 13-25 weeks ago | +0.826 |
| Airplay Top-20: 0.5-1.25x own usual gap | -0.176 |
| Airplay Top-20: last new song 2+ years ago | +0.334 |
| log archive plays per 100 sampled hours (2012-21) | +0.618 |
| log DR lists with the artist before t (2023-) | +0.156 |
| DR list recency 26w | -0.607 |
| DR lists known before t at all | -0.210 |
| quiet former favourite (4+ weeks on DR 2019-24, out of rotation 26w) | -0.665 |
| NMF Denmark in the window | +0.943 |
| NMF Denmark top 10 in the window | +0.202 |
| log NMF Denmark placements last 52w | +0.251 |
| share of releases on NMF Denmark (2 years) | +0.020 |
| season index of the window (log, IFPI 2002-24) | -0.049 |