Knowable pool (artists enter only from their first release/play/chart/list/NMF), pooled out-of-sample recalibration (Platt vs isotonic; the site uses pooled Platt per channel, isotonic any channel), release-timing baseline, 8-week bootstrap blocks, surge-rule precision/recall vs the lists with a constant grid (docs/ADDCHECK.md); backtests restructured to score all variants with calibration tables
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,23 @@
|
|||||||
|
# Add definitions vs DR's own lists
|
||||||
|
|
||||||
|
Truth: artists with a song new on a list vs the previous list of the channel. Recall over all list pairs (24), and recall and precision over pairs at most 8 weeks apart (15). Precision is a lower bound: songs added and dropped between lists, or added off-list, count against it.
|
||||||
|
|
||||||
|
| Definition | Recall, all pairs | Recall, close pairs | Precision, close pairs | F1, close pairs | Adds detected | New songs |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| onset | 64% | 55% | 22% | 0.32 | 351 | 177 |
|
||||||
|
| surge [0.6, 0.5, 0.5, 0.25] | 77% | 68% | 25% | 0.36 | 420 | 177 |
|
||||||
|
|
||||||
|
Surge constants tried (floor × τ, jump × τ, jump × baseline, held next week), best F1 on close pairs:
|
||||||
|
|
||||||
|
| Constants | Recall | Precision | F1 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| [0.4, 0.3, 1.0, 0.5] | 65% | 26% | 0.369 |
|
||||||
|
| [0.4, 0.3, 0.5, 0.5] | 68% | 25% | 0.368 |
|
||||||
|
| [0.6, 0.3, 1.0, 0.5] | 64% | 26% | 0.368 |
|
||||||
|
| [0.4, 0.3, 0.25, 0.5] | 69% | 25% | 0.367 |
|
||||||
|
| [0.6, 0.5, 0.25, 0.25] | 70% | 25% | 0.365 |
|
||||||
|
| [0.6, 0.3, 0.5, 0.5] | 67% | 25% | 0.365 |
|
||||||
|
| [0.6, 0.5, 0.25, 0.5] | 68% | 25% | 0.365 |
|
||||||
|
| [0.6, 0.3, 0.25, 0.5] | 69% | 25% | 0.364 |
|
||||||
|
|
||||||
|
The grid's best F1 (0.369) is within noise of the current constants (0.36), so the defaults stay. Surge beats onset-only on both recall and precision.
|
||||||
+189
-150
@@ -1,193 +1,232 @@
|
|||||||
# Backtest: new adds
|
# Backtest: new adds
|
||||||
|
|
||||||
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation: per channel, and on any DR channel after 4 weeks out of rotation everywhere. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed by its origin.
|
An add = the artist enters rotation (plays ≥ τ) after at least 4 weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the 8 weeks that start h weeks after the forecast origin. Origins from week 27; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.
|
||||||
|
|
||||||
## Window starting h = 0 weeks ahead (17 forecast origins)
|
## Window starting h = 0 weeks ahead (16 forecast origins)
|
||||||
|
|
||||||
Base rates: per channel 0.43% of artist-channels get an add in the window; on any channel 2.4% of artists.
|
Base rates: per channel 0.46% of artist-channels get an add in the window; on any channel 2.5% of artists.
|
||||||
|
|
||||||
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| rate by in-rotation-now | 0.0255 | 0.032 | 0.1058 | 0.067 |
|
| rate by in-rotation-now | 0.0271 | 0.032 | 0.1111 | 0.069 |
|
||||||
| rate by last-12-weeks level | 0.0177 | 0.147 | 0.0909 | 0.133 |
|
| rate by last-12-weeks level | 0.0188 | 0.148 | 0.0956 | 0.134 |
|
||||||
| logit (pre-release: NMF from history) | 0.0147 | 0.258 | 0.0800 | 0.251 |
|
| release timing only | 0.0274 | 0.010 | 0.1144 | 0.040 |
|
||||||
| logit (post-release: NMF known) | 0.0145 | 0.266 | 0.0799 | 0.259 |
|
| model, calibrated on the last 6 origins (pre-release) | 0.0156 | 0.263 | 0.0837 | 0.254 |
|
||||||
|
| model, raw (pre-release) | 0.0155 | 0.263 | 0.0854 | 0.246 |
|
||||||
|
| model + pooled Platt (pre-release) | 0.0156 | 0.262 | 0.0855 | 0.246 |
|
||||||
|
| model + pooled isotonic (pre-release) | 0.0157 | 0.254 | 0.0859 | 0.244 |
|
||||||
|
| model, NMF known (post-release) | 0.0153 | 0.270 | 0.0836 | 0.263 |
|
||||||
|
|
||||||
Skill of the logit (90% block-bootstrap interval over origins):
|
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||||||
|
|
||||||
| | vs in-rotation-now | vs last-12-weeks level |
|
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||||||
|---|---|---|
|
|---|---|---|---|---|
|
||||||
| per channel | 0.424 (0.413 to 0.437) | 0.171 (0.168 to 0.178) |
|
| model, calibrated on the last 6 origins (pre-release) | +0.174 (+0.167 to +0.176) | +0.432 (+0.412 to +0.445) | +0.124 (+0.123 to +0.133) | +0.269 (+0.242 to +0.289) |
|
||||||
| any channel | 0.244 (0.234 to 0.257) | 0.120 (0.115 to 0.127) |
|
| model + pooled Platt (pre-release) | +0.172 (+0.164 to +0.172) | +0.431 (+0.409 to +0.444) | +0.105 (+0.091 to +0.112) | +0.253 (+0.213 to +0.272) |
|
||||||
|
| model + pooled isotonic (pre-release) | +0.168 (+0.160 to +0.168) | +0.428 (+0.405 to +0.444) | +0.102 (+0.086 to +0.112) | +0.250 (+0.209 to +0.272) |
|
||||||
|
|
||||||
|
Calibration, any channel:
|
||||||
|
|
||||||
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 0.0–0.1 | 112229 / 0.010 / 0.013 | 111004 / 0.012 / 0.012 | 111041 / 0.011 / 0.012 |
|
||||||
|
| 0.1–0.2 | 2985 / 0.142 / 0.196 | 3319 / 0.142 / 0.155 | 2508 / 0.146 / 0.149 |
|
||||||
|
| 0.2–0.3 | 1360 / 0.245 / 0.263 | 1636 / 0.245 / 0.249 | 2058 / 0.249 / 0.218 |
|
||||||
|
| 0.3–0.4 | 799 / 0.346 / 0.328 | 998 / 0.347 / 0.305 | 1765 / 0.339 / 0.301 |
|
||||||
|
| 0.4–0.5 | 450 / 0.447 / 0.360 | 626 / 0.445 / 0.321 | 489 / 0.442 / 0.309 |
|
||||||
|
| 0.5–0.6 | 251 / 0.546 / 0.347 | 355 / 0.548 / 0.293 | 247 / 0.546 / 0.352 |
|
||||||
|
| 0.6–0.7 | 162 / 0.647 / 0.407 | 215 / 0.646 / 0.344 | 152 / 0.646 / 0.421 |
|
||||||
|
| 0.7–0.8 | 93 / 0.746 / 0.505 | 149 / 0.740 / 0.483 | 73 / 0.740 / 0.479 |
|
||||||
|
| 0.8–0.9 | 52 / 0.841 / 0.596 | 74 / 0.837 / 0.554 | 43 / 0.837 / 0.558 |
|
||||||
|
| 0.9–1.0 | 11 / 0.924 / 0.182 | 16 / 0.924 / 0.438 | 16 / 0.932 / 0.500 |
|
||||||
|
|
||||||
Calibration, per channel:
|
Calibration, per channel:
|
||||||
|
|
||||||
| Predicted | Rows | Mean predicted | Observed |
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 0.0–0.1 | 1868823 | 0.002 | 0.002 |
|
| 0.0–0.1 | 1638964 / 0.002 / 0.002 | 1639990 / 0.002 / 0.002 | 1637740 / 0.002 / 0.002 |
|
||||||
| 0.1–0.2 | 9096 | 0.142 | 0.160 |
|
| 0.1–0.2 | 8541 / 0.142 / 0.165 | 7975 / 0.141 / 0.176 | 9301 / 0.135 / 0.149 |
|
||||||
| 0.2–0.3 | 3991 | 0.247 | 0.230 |
|
| 0.2–0.3 | 3740 / 0.248 / 0.229 | 3586 / 0.247 / 0.231 | 4805 / 0.251 / 0.229 |
|
||||||
| 0.3–0.4 | 2858 | 0.344 | 0.215 |
|
| 0.3–0.4 | 2586 / 0.344 / 0.216 | 2464 / 0.345 / 0.211 | 3432 / 0.338 / 0.297 |
|
||||||
| 0.4–0.5 | 1615 | 0.448 | 0.310 |
|
| 0.4–0.5 | 1570 / 0.449 / 0.336 | 1465 / 0.449 / 0.374 | 939 / 0.448 / 0.376 |
|
||||||
| 0.5–0.6 | 1081 | 0.544 | 0.457 |
|
| 0.5–0.6 | 984 / 0.545 / 0.450 | 966 / 0.547 / 0.443 | 588 / 0.548 / 0.371 |
|
||||||
| 0.6–0.7 | 728 | 0.646 | 0.475 |
|
| 0.6–0.7 | 621 / 0.645 / 0.473 | 549 / 0.643 / 0.461 | 328 / 0.644 / 0.424 |
|
||||||
| 0.7–0.8 | 429 | 0.749 | 0.441 |
|
| 0.7–0.8 | 351 / 0.751 / 0.464 | 358 / 0.751 / 0.455 | 207 / 0.756 / 0.560 |
|
||||||
| 0.8–0.9 | 104 | 0.834 | 0.346 |
|
| 0.8–0.9 | 91 / 0.835 / 0.352 | 90 / 0.834 / 0.489 | 148 / 0.853 / 0.507 |
|
||||||
| 0.9–1.0 | 43 | 0.917 | 0.930 |
|
| 0.9–1.0 | 40 / 0.914 / 0.950 | 45 / 0.928 / 0.689 | — |
|
||||||
|
|
||||||
## Window starting h = 4 weeks ahead (13 forecast origins)
|
## Window starting h = 4 weeks ahead (12 forecast origins)
|
||||||
|
|
||||||
Base rates: per channel 0.44% of artist-channels get an add in the window; on any channel 2.4% of artists.
|
Base rates: per channel 0.47% of artist-channels get an add in the window; on any channel 2.5% of artists.
|
||||||
|
|
||||||
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| rate by in-rotation-now | 0.0263 | 0.032 | 0.1051 | 0.083 |
|
| rate by in-rotation-now | 0.0277 | 0.033 | 0.1102 | 0.084 |
|
||||||
| rate by last-12-weeks level | 0.0193 | 0.114 | 0.0937 | 0.136 |
|
| rate by last-12-weeks level | 0.0204 | 0.117 | 0.0984 | 0.137 |
|
||||||
| logit (pre-release: NMF from history) | 0.0176 | 0.166 | 0.0857 | 0.208 |
|
| release timing only | 0.0286 | 0.010 | 0.1146 | 0.043 |
|
||||||
| logit (post-release: NMF known) | 0.0172 | 0.175 | 0.0856 | 0.218 |
|
| model, calibrated on the last 6 origins (pre-release) | 0.0186 | 0.166 | 0.0898 | 0.208 |
|
||||||
|
| model, raw (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||||||
|
| model + pooled Platt (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||||||
|
| model + pooled isotonic (pre-release) | 0.0186 | 0.166 | 0.0908 | 0.200 |
|
||||||
|
| model, NMF known (post-release) | 0.0183 | 0.174 | 0.0896 | 0.219 |
|
||||||
|
|
||||||
Skill of the logit (90% block-bootstrap interval over origins):
|
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||||||
|
|
||||||
| | vs in-rotation-now | vs last-12-weeks level |
|
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||||||
|---|---|---|
|
|---|---|---|---|---|
|
||||||
| per channel | 0.330 (0.315 to 0.337) | 0.091 (0.065 to 0.106) |
|
| model, calibrated on the last 6 origins (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.088 (+0.084 to +0.095) | +0.217 (+0.207 to +0.221) |
|
||||||
| any channel | 0.185 (0.174 to 0.194) | 0.086 (0.081 to 0.090) |
|
| model + pooled Platt (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
|
||||||
|
| model + pooled isotonic (pre-release) | +0.086 (+0.065 to +0.080) | +0.349 (+0.334 to +0.345) | +0.077 (+0.072 to +0.080) | +0.208 (+0.195 to +0.208) |
|
||||||
|
|
||||||
|
Calibration, any channel:
|
||||||
|
|
||||||
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 0.0–0.1 | 84532 / 0.011 / 0.015 | 83207 / 0.013 / 0.013 | 83207 / 0.013 / 0.013 |
|
||||||
|
| 0.1–0.2 | 2353 / 0.140 / 0.178 | 2764 / 0.142 / 0.156 | 2764 / 0.142 / 0.156 |
|
||||||
|
| 0.2–0.3 | 904 / 0.242 / 0.304 | 1271 / 0.245 / 0.238 | 1271 / 0.245 / 0.238 |
|
||||||
|
| 0.3–0.4 | 479 / 0.347 / 0.311 | 717 / 0.346 / 0.262 | 717 / 0.346 / 0.262 |
|
||||||
|
| 0.4–0.5 | 221 / 0.446 / 0.385 | 359 / 0.445 / 0.304 | 359 / 0.445 / 0.304 |
|
||||||
|
| 0.5–0.6 | 124 / 0.544 / 0.306 | 188 / 0.547 / 0.372 | 188 / 0.547 / 0.372 |
|
||||||
|
| 0.6–0.7 | 83 / 0.640 / 0.325 | 142 / 0.641 / 0.345 | 142 / 0.641 / 0.345 |
|
||||||
|
| 0.7–0.8 | 30 / 0.750 / 0.333 | 66 / 0.749 / 0.273 | 66 / 0.749 / 0.273 |
|
||||||
|
| 0.8–0.9 | 4 / 0.830 / 0.000 | 16 / 0.835 / 0.312 | 16 / 0.835 / 0.312 |
|
||||||
|
|
||||||
Calibration, per channel:
|
Calibration, per channel:
|
||||||
|
|
||||||
| Predicted | Rows | Mean predicted | Observed |
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 0.0–0.1 | 1428760 | 0.002 | 0.003 |
|
| 0.0–0.1 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 | 1228298 / 0.002 / 0.003 |
|
||||||
| 0.1–0.2 | 7389 | 0.141 | 0.129 |
|
| 0.1–0.2 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 | 6683 / 0.141 / 0.129 |
|
||||||
| 0.2–0.3 | 3960 | 0.247 | 0.186 |
|
| 0.2–0.3 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 | 3587 / 0.247 / 0.188 |
|
||||||
| 0.3–0.4 | 2024 | 0.344 | 0.199 |
|
| 0.3–0.4 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 | 1731 / 0.345 / 0.207 |
|
||||||
| 0.4–0.5 | 1134 | 0.444 | 0.283 |
|
| 0.4–0.5 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 | 956 / 0.443 / 0.285 |
|
||||||
| 0.5–0.6 | 604 | 0.546 | 0.224 |
|
| 0.5–0.6 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 | 571 / 0.548 / 0.256 |
|
||||||
| 0.6–0.7 | 394 | 0.643 | 0.459 |
|
| 0.6–0.7 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 | 333 / 0.647 / 0.441 |
|
||||||
| 0.7–0.8 | 83 | 0.732 | 0.277 |
|
| 0.7–0.8 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 | 55 / 0.734 / 0.255 |
|
||||||
| 0.8–0.9 | 3 | 0.833 | 1.000 |
|
| 0.8–0.9 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 | 5 / 0.820 / 0.400 |
|
||||||
| 0.9–1.0 | 1 | 0.907 | 1.000 |
|
| 0.9–1.0 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 | 1 / 0.905 / 1.000 |
|
||||||
|
|
||||||
## Coefficients, window starting now (h = 0), all closed windows
|
## Coefficients, window starting now (h = 0), all closed windows
|
||||||
|
|
||||||
| Any-channel feature | Coefficient |
|
| Any-channel feature | Coefficient |
|
||||||
|---|---|
|
|---|---|
|
||||||
| intercept | -4.213 |
|
| intercept | -4.141 |
|
||||||
| log DR plays t | +0.883 |
|
| log DR plays t | +0.882 |
|
||||||
| log DR plays 4w | +0.253 |
|
| log DR plays 4w | +0.253 |
|
||||||
| log DR plays 12w | -0.263 |
|
| log DR plays 12w | -0.255 |
|
||||||
| log max DR plays 12w | +0.594 |
|
| log max DR plays 12w | +0.583 |
|
||||||
| in rotation anywhere now | -1.148 |
|
| in rotation anywhere now | -1.155 |
|
||||||
| station groups in rotation now (/5) | +0.297 |
|
| station groups in rotation now (/5) | +0.320 |
|
||||||
| current run anywhere (/12) | -1.212 |
|
| current run anywhere (/12) | -1.215 |
|
||||||
| weeks since in rotation anywhere (/12) | -0.749 |
|
| weeks since in rotation anywhere (/12) | -0.748 |
|
||||||
| Track Top-40 score | +0.427 |
|
| Track Top-40 score | +0.427 |
|
||||||
| Airplay Top-20 score | -1.332 |
|
| Airplay Top-20 score | -1.336 |
|
||||||
| chart entry recency 6w | +1.210 |
|
| chart entry recency 6w | +1.203 |
|
||||||
| chart entry recency 26w | -0.653 |
|
| chart entry recency 26w | -0.646 |
|
||||||
| log Track Top-40 hits | -0.200 |
|
| log Track Top-40 hits | -0.201 |
|
||||||
| log Airplay Top-20 hits | -0.069 |
|
| log Airplay Top-20 hits | -0.070 |
|
||||||
| Danish | +0.462 |
|
| Danish | +0.470 |
|
||||||
| group | -0.240 |
|
| group | -0.253 |
|
||||||
| female | -0.102 |
|
| female | -0.097 |
|
||||||
| career years (/50) | +0.210 |
|
| career years (/50) | +0.222 |
|
||||||
| debut unknown | +0.497 |
|
| debut unknown | +0.489 |
|
||||||
| Christmas in window | +0.000 |
|
| Christmas in window | +0.000 |
|
||||||
| discography known | -0.325 |
|
| discography known | -0.392 |
|
||||||
| release recency 4w | +0.440 |
|
| release recency 4w | +0.435 |
|
||||||
| release recency 12w | +0.670 |
|
| release recency 12w | +0.680 |
|
||||||
| log releases last 52w | -0.188 |
|
| log releases last 52w | -0.186 |
|
||||||
| releases in the window | -0.218 |
|
| releases in the window | -0.216 |
|
||||||
| releases in window x not in rotation now | +0.854 |
|
| releases in window x not in rotation now | +0.836 |
|
||||||
| DR: weeks since last add 13-25 | +0.095 |
|
| DR: weeks since last add 13-25 | +0.095 |
|
||||||
| DR: last add 26+ weeks ago | -0.642 |
|
| DR: last add 26+ weeks ago | -0.642 |
|
||||||
| DR: no add in the panel yet | -0.847 |
|
| DR: no add in the panel yet | -0.845 |
|
||||||
| Airplay Top-20: last new song 13-25 weeks ago | +0.279 |
|
| Airplay Top-20: last new song 13-25 weeks ago | +0.278 |
|
||||||
| Airplay Top-20: 0.5-1.25x own usual gap | +0.231 |
|
| Airplay Top-20: 0.5-1.25x own usual gap | +0.229 |
|
||||||
| Airplay Top-20: last new song 2+ years ago | -0.529 |
|
| Airplay Top-20: last new song 2+ years ago | -0.527 |
|
||||||
| log archive plays per 100 sampled hours (2012-21) | +0.288 |
|
| log archive plays per 100 sampled hours (2012-21) | +0.289 |
|
||||||
| log DR lists with the artist before t (2023-) | +0.209 |
|
| log DR lists with the artist before t (2023-) | +0.209 |
|
||||||
| DR list recency 26w | -0.929 |
|
| DR list recency 26w | -0.924 |
|
||||||
| DR lists known before t at all | +0.040 |
|
| DR lists known before t at all | +0.039 |
|
||||||
| NMF Denmark in the window | +0.941 |
|
| NMF Denmark in the window | +0.940 |
|
||||||
| NMF Denmark top 10 in the window | +0.416 |
|
| NMF Denmark top 10 in the window | +0.408 |
|
||||||
| log NMF Denmark placements last 52w | -0.071 |
|
| log NMF Denmark placements last 52w | -0.068 |
|
||||||
| share of releases on NMF Denmark (2 years) | -0.415 |
|
| share of releases on NMF Denmark (2 years) | -0.417 |
|
||||||
| season index of the window (log, IFPI 2002-24) | +0.403 |
|
| season index of the window (log, IFPI 2002-24) | +0.412 |
|
||||||
|
|
||||||
| Per-channel feature | Coefficient |
|
| Per-channel feature | Coefficient |
|
||||||
|---|---|
|
|---|---|
|
||||||
| intercept | -7.101 |
|
| intercept | -7.117 |
|
||||||
| log plays t | +1.122 |
|
| log plays t | +1.106 |
|
||||||
| log plays t-1 | +0.142 |
|
| log plays t-1 | +0.141 |
|
||||||
| log mean plays 4w | +0.113 |
|
| log mean plays 4w | +0.113 |
|
||||||
| log mean plays 12w | -0.862 |
|
| log mean plays 12w | -0.871 |
|
||||||
| log max plays 12w | +1.014 |
|
| log max plays 12w | +1.022 |
|
||||||
| log other channels t | +0.332 |
|
| log other channels t | +0.328 |
|
||||||
| log other channels 4w | -0.100 |
|
| log other channels 4w | -0.095 |
|
||||||
| log P3 t | +0.085 |
|
| log P3 t | +0.087 |
|
||||||
| weeks since played (/12) | -0.437 |
|
| weeks since played (/12) | -0.509 |
|
||||||
| Track Top-40 score | -0.127 |
|
| Track Top-40 score | -0.132 |
|
||||||
| Airplay Top-20 score | -1.373 |
|
| Airplay Top-20 score | -1.376 |
|
||||||
| chart entry recency 6w | +0.625 |
|
| chart entry recency 6w | +0.616 |
|
||||||
| chart entry recency 26w | -0.512 |
|
| chart entry recency 26w | -0.506 |
|
||||||
| log Track Top-40 hits | +0.257 |
|
| log Track Top-40 hits | +0.258 |
|
||||||
| log Airplay Top-20 hits | -0.395 |
|
| log Airplay Top-20 hits | -0.398 |
|
||||||
| Danish | +0.162 |
|
| Danish | +0.171 |
|
||||||
| group | -0.629 |
|
| group | -0.636 |
|
||||||
| female | +0.127 |
|
| female | +0.136 |
|
||||||
| career years (/50) | -1.500 |
|
| career years (/50) | -1.498 |
|
||||||
| debut unknown | -0.289 |
|
| debut unknown | -0.333 |
|
||||||
| Christmas target week | +0.000 |
|
| Christmas target week | +0.000 |
|
||||||
| Danish x P3 | +0.072 |
|
| Danish x P3 | +0.064 |
|
||||||
| Danish x P4 | +0.085 |
|
| Danish x P4 | +0.058 |
|
||||||
| Danish x P6 | +0.154 |
|
| Danish x P6 | +0.111 |
|
||||||
| ch p4kbh | +1.100 |
|
| ch p4kbh | +1.094 |
|
||||||
| ch p4sjaelland | +0.968 |
|
| ch p4sjaelland | +0.966 |
|
||||||
| ch p4fyn | +0.948 |
|
| ch p4fyn | +0.946 |
|
||||||
| ch p4syd | +0.976 |
|
| ch p4syd | +0.975 |
|
||||||
| ch p4trekanten | +0.992 |
|
| ch p4trekanten | +0.985 |
|
||||||
| ch p4vest | +1.078 |
|
| ch p4vest | +1.083 |
|
||||||
| ch p4aarhus | +0.934 |
|
| ch p4aarhus | +0.944 |
|
||||||
| ch p4nord | +1.002 |
|
| ch p4nord | +1.006 |
|
||||||
| ch p4bornholm | +0.986 |
|
| ch p4bornholm | +0.995 |
|
||||||
| ch p4esbjerg | +0.968 |
|
| ch p4esbjerg | +0.972 |
|
||||||
| ch p5 | +2.081 |
|
| ch p5 | +2.081 |
|
||||||
| ch p6beat | +1.295 |
|
| ch p6beat | +1.296 |
|
||||||
| ch p8jazz | +2.330 |
|
| ch p8jazz | +2.312 |
|
||||||
| in rotation on channel now | +0.358 |
|
| in rotation on channel now | +0.350 |
|
||||||
| in rotation anywhere now | -0.338 |
|
| in rotation anywhere now | -0.334 |
|
||||||
| station groups in rotation now (/5) | +0.043 |
|
| station groups in rotation now (/5) | +0.041 |
|
||||||
| run on channel = 1 week | +0.663 |
|
| run on channel = 1 week | +0.660 |
|
||||||
| run = 2 weeks | +0.441 |
|
| run = 2 weeks | +0.439 |
|
||||||
| run = 3-5 weeks | -0.055 |
|
| run = 3-5 weeks | -0.056 |
|
||||||
| run >= 6 weeks | -0.691 |
|
| run >= 6 weeks | -0.693 |
|
||||||
| near threshold (>= half of tau, not in rotation) | +1.376 |
|
| near threshold (>= half of tau, not in rotation) | +1.381 |
|
||||||
| plays / tau (capped 3, /3) | -3.407 |
|
| plays / tau (capped 3, /3) | -3.363 |
|
||||||
| weeks since in rotation on channel (/12) | -0.300 |
|
| weeks since in rotation on channel (/12) | -0.302 |
|
||||||
| discography known | -0.069 |
|
| discography known | -0.052 |
|
||||||
| release recency 4w | +1.033 |
|
| release recency 4w | +1.017 |
|
||||||
| release recency 12w | -0.008 |
|
| release recency 12w | +0.014 |
|
||||||
| log releases last 52w | -0.003 |
|
| log releases last 52w | +0.009 |
|
||||||
| releases in the window | -0.143 |
|
| releases in the window | -0.136 |
|
||||||
| releases in window x not in rotation now | +0.782 |
|
| releases in window x not in rotation now | +0.755 |
|
||||||
| DR: weeks since last add 13-25 | +0.107 |
|
| DR: weeks since last add 13-25 | +0.106 |
|
||||||
| DR: last add 26+ weeks ago | -0.085 |
|
| DR: last add 26+ weeks ago | -0.081 |
|
||||||
| DR: no add in the panel yet | -0.486 |
|
| DR: no add in the panel yet | -0.483 |
|
||||||
| Airplay Top-20: last new song 13-25 weeks ago | +0.194 |
|
| Airplay Top-20: last new song 13-25 weeks ago | +0.194 |
|
||||||
| Airplay Top-20: 0.5-1.25x own usual gap | -0.009 |
|
| Airplay Top-20: 0.5-1.25x own usual gap | -0.013 |
|
||||||
| Airplay Top-20: last new song 2+ years ago | -0.417 |
|
| Airplay Top-20: last new song 2+ years ago | -0.418 |
|
||||||
| log archive plays per 100 sampled hours (2012-21) | +0.348 |
|
| log archive plays per 100 sampled hours (2012-21) | +0.347 |
|
||||||
| log DR lists with the artist before t (2023-) | +0.028 |
|
| log DR lists with the artist before t (2023-) | +0.023 |
|
||||||
| DR list recency 26w | -0.444 |
|
| DR list recency 26w | -0.442 |
|
||||||
| DR lists known before t at all | +0.055 |
|
| DR lists known before t at all | +0.066 |
|
||||||
| NMF Denmark in the window | +0.805 |
|
| NMF Denmark in the window | +0.779 |
|
||||||
| NMF Denmark top 10 in the window | +0.579 |
|
| NMF Denmark top 10 in the window | +0.580 |
|
||||||
| log NMF Denmark placements last 52w | -0.077 |
|
| log NMF Denmark placements last 52w | -0.089 |
|
||||||
| share of releases on NMF Denmark (2 years) | +0.166 |
|
| share of releases on NMF Denmark (2 years) | +0.212 |
|
||||||
| season index of the window (log, IFPI 2002-24) | +1.240 |
|
| season index of the window (log, IFPI 2002-24) | +1.211 |
|
||||||
| share of the artist's DR plays on this station (before t) | +0.963 |
|
| share of the artist's DR plays on this station (before t) | +0.986 |
|
||||||
| log DR plays on this station (before t) | +0.049 |
|
| log DR plays on this station (before t) | +0.050 |
|
||||||
| never played on this station (before t) | -2.888 |
|
| never played on this station (before t) | -2.825 |
|
||||||
| genre fit for this station | +0.108 |
|
| genre fit for this station | +0.091 |
|
||||||
|
|
||||||
|
|||||||
+87
-47
@@ -1,57 +1,97 @@
|
|||||||
# Backtest: if the artist releases a new song, will DR add it?
|
# Backtest: if the artist releases a new song, will DR add it?
|
||||||
|
|
||||||
Events: an artist in the panel releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; the new song's NMF Denmark placement is blended from the artist's history. 2302 release events scored, 130 (5.6%) got an add on some DR channel.
|
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 2140 release events scored, 120 (5.6%) got an add on some DR channel.
|
||||||
|
|
||||||
| Model | Log-loss, any channel | PR-AUC, any channel | Log-loss, per channel | PR-AUC, per channel |
|
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| rate by in-rotation-now | 0.2073 | 0.115 | 0.0666 | 0.059 |
|
| rate by in-rotation-now | 0.0666 | 0.061 | 0.2070 | 0.113 |
|
||||||
| rate by last-12-weeks level | 0.1838 | 0.215 | 0.0491 | 0.202 |
|
| rate by last-12-weeks level | 0.0478 | 0.216 | 0.1805 | 0.221 |
|
||||||
| event model (pre-release: NMF from history) | 0.1854 | 0.248 | 0.0445 | 0.314 |
|
| release timing only | 0.0719 | 0.014 | 0.2166 | 0.068 |
|
||||||
| event model (post-release: NMF known) | 0.1814 | 0.280 | 0.0422 | 0.325 |
|
| model, calibrated on the last 6 origins (pre-release) | 0.0431 | 0.341 | 0.1881 | 0.243 |
|
||||||
|
| model, raw (pre-release) | 0.0437 | 0.326 | 0.1860 | 0.252 |
|
||||||
|
| model + pooled Platt (pre-release) | 0.0425 | 0.337 | 0.1916 | 0.221 |
|
||||||
|
| model + pooled isotonic (pre-release) | 0.0439 | 0.308 | 0.1871 | 0.243 |
|
||||||
|
| model, NMF known (post-release) | 0.0413 | 0.346 | 0.1853 | 0.266 |
|
||||||
|
|
||||||
Skill of the event model, any channel: 0.106 (0.081 to 0.178) vs in-rotation-now, -0.008 (-0.051 to 0.068) vs last-12-weeks level.
|
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||||||
|
|
||||||
|
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| model, calibrated on the last 6 origins (pre-release) | +0.098 (+0.054 to +0.162) | +0.400 (+0.313 to +0.440) | -0.042 (-0.068 to +0.041) | +0.132 (+0.128 to +0.223) |
|
||||||
|
| model + pooled Platt (pre-release) | +0.111 (+0.054 to +0.181) | +0.408 (+0.330 to +0.440) | -0.061 (-0.091 to -0.022) | +0.116 (+0.097 to +0.160) |
|
||||||
|
| model + pooled isotonic (pre-release) | +0.083 (+0.054 to +0.187) | +0.390 (+0.333 to +0.438) | -0.037 (-0.072 to +0.031) | +0.136 (+0.121 to +0.206) |
|
||||||
|
|
||||||
|
Calibration, any channel:
|
||||||
|
|
||||||
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 0.0–0.1 | 1936 / 0.018 / 0.034 | 1897 / 0.034 / 0.033 | 1844 / 0.027 / 0.027 |
|
||||||
|
| 0.1–0.2 | 124 / 0.137 / 0.234 | 154 / 0.141 / 0.201 | 124 / 0.143 / 0.202 |
|
||||||
|
| 0.2–0.3 | 40 / 0.245 / 0.325 | 56 / 0.247 / 0.286 | 90 / 0.251 / 0.200 |
|
||||||
|
| 0.3–0.4 | 18 / 0.349 / 0.278 | 20 / 0.355 / 0.300 | 40 / 0.350 / 0.250 |
|
||||||
|
| 0.4–0.5 | 13 / 0.442 / 0.385 | 7 / 0.463 / 0.429 | 23 / 0.449 / 0.435 |
|
||||||
|
| 0.5–0.6 | 5 / 0.534 / 0.400 | 3 / 0.536 / 0.333 | 8 / 0.571 / 0.500 |
|
||||||
|
| 0.6–0.7 | 1 / 0.606 / 0.000 | 2 / 0.625 / 0.000 | 3 / 0.637 / 0.333 |
|
||||||
|
| 0.7–0.8 | 3 / 0.732 / 0.333 | — | 2 / 0.730 / 0.000 |
|
||||||
|
| 0.8–0.9 | — | 1 / 0.823 / 1.000 | 5 / 0.824 / 0.600 |
|
||||||
|
| 0.9–1.0 | — | — | 1 / 0.911 / 0.000 |
|
||||||
|
|
||||||
|
Calibration, per channel:
|
||||||
|
|
||||||
|
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 0.0–0.1 | 29258 / 0.004 / 0.007 | 29241 / 0.004 / 0.007 | 29307 / 0.004 / 0.007 |
|
||||||
|
| 0.1–0.2 | 342 / 0.140 / 0.208 | 347 / 0.143 / 0.182 | 328 / 0.143 / 0.235 |
|
||||||
|
| 0.2–0.3 | 136 / 0.246 / 0.206 | 136 / 0.252 / 0.287 | 122 / 0.248 / 0.189 |
|
||||||
|
| 0.3–0.4 | 87 / 0.347 / 0.207 | 102 / 0.335 / 0.118 | 81 / 0.344 / 0.185 |
|
||||||
|
| 0.4–0.5 | 66 / 0.447 / 0.500 | 44 / 0.450 / 0.500 | 35 / 0.449 / 0.429 |
|
||||||
|
| 0.5–0.6 | 10 / 0.527 / 0.000 | 28 / 0.538 / 0.714 | 25 / 0.551 / 0.760 |
|
||||||
|
| 0.6–0.7 | 8 / 0.675 / 0.125 | 5 / 0.655 / 0.400 | 4 / 0.652 / 0.250 |
|
||||||
|
| 0.7–0.8 | 42 / 0.745 / 0.762 | 42 / 0.757 / 0.714 | 35 / 0.751 / 0.886 |
|
||||||
|
| 0.8–0.9 | 2 / 0.845 / 1.000 | 6 / 0.817 / 0.500 | 2 / 0.845 / 1.000 |
|
||||||
|
| 0.9–1.0 | 9 / 0.912 / 1.000 | 9 / 0.912 / 1.000 | 21 / 0.954 / 0.476 |
|
||||||
|
|
||||||
| Any-channel feature (all events) | Coefficient |
|
| Any-channel feature (all events) | Coefficient |
|
||||||
|---|---|
|
|---|---|
|
||||||
| intercept | -3.665 |
|
| intercept | -3.768 |
|
||||||
| log DR plays t | +0.557 |
|
| log DR plays t | +0.560 |
|
||||||
| log DR plays 4w | +0.507 |
|
| log DR plays 4w | +0.510 |
|
||||||
| log DR plays 12w | -0.363 |
|
| log DR plays 12w | -0.409 |
|
||||||
| log max DR plays 12w | +0.249 |
|
| log max DR plays 12w | +0.318 |
|
||||||
| in rotation anywhere now | -0.476 |
|
| in rotation anywhere now | -0.474 |
|
||||||
| station groups in rotation now (/5) | -0.252 |
|
| station groups in rotation now (/5) | -0.249 |
|
||||||
| current run anywhere (/12) | -1.197 |
|
| current run anywhere (/12) | -1.173 |
|
||||||
| weeks since in rotation anywhere (/12) | -0.514 |
|
| weeks since in rotation anywhere (/12) | -0.489 |
|
||||||
| Track Top-40 score | -0.282 |
|
| Track Top-40 score | -0.265 |
|
||||||
| Airplay Top-20 score | -0.313 |
|
| Airplay Top-20 score | -0.317 |
|
||||||
| chart entry recency 6w | +0.713 |
|
| chart entry recency 6w | +0.685 |
|
||||||
| chart entry recency 26w | +0.477 |
|
| chart entry recency 26w | +0.501 |
|
||||||
| log Track Top-40 hits | -0.303 |
|
| log Track Top-40 hits | -0.291 |
|
||||||
| log Airplay Top-20 hits | -0.118 |
|
| log Airplay Top-20 hits | -0.120 |
|
||||||
| Danish | +0.580 |
|
| Danish | +0.548 |
|
||||||
| group | +0.043 |
|
| group | +0.013 |
|
||||||
| female | -0.064 |
|
| female | -0.075 |
|
||||||
| career years (/50) | +0.433 |
|
| career years (/50) | +0.422 |
|
||||||
| debut unknown | +0.530 |
|
| debut unknown | +0.435 |
|
||||||
| Christmas in window | +0.000 |
|
| Christmas in window | +0.000 |
|
||||||
| discography known | +0.000 |
|
| discography known | -0.000 |
|
||||||
| release recency 4w | +0.243 |
|
| release recency 4w | +0.216 |
|
||||||
| release recency 12w | -0.814 |
|
| release recency 12w | -0.783 |
|
||||||
| log releases last 52w | -0.224 |
|
| log releases last 52w | -0.189 |
|
||||||
| releases in the window | +0.000 |
|
| releases in the window | -0.000 |
|
||||||
| releases in window x not in rotation now | +0.476 |
|
| releases in window x not in rotation now | +0.474 |
|
||||||
| DR: weeks since last add 13-25 | -0.054 |
|
| DR: weeks since last add 13-25 | -0.050 |
|
||||||
| DR: last add 26+ weeks ago | +0.497 |
|
| DR: last add 26+ weeks ago | +0.502 |
|
||||||
| DR: no add in the panel yet | -0.684 |
|
| DR: no add in the panel yet | -0.677 |
|
||||||
| Airplay Top-20: last new song 13-25 weeks ago | +0.463 |
|
| Airplay Top-20: last new song 13-25 weeks ago | +0.465 |
|
||||||
| Airplay Top-20: 0.5-1.25x own usual gap | -0.144 |
|
| Airplay Top-20: 0.5-1.25x own usual gap | -0.144 |
|
||||||
| Airplay Top-20: last new song 2+ years ago | -0.107 |
|
| Airplay Top-20: last new song 2+ years ago | -0.113 |
|
||||||
| log archive plays per 100 sampled hours (2012-21) | +0.532 |
|
| log archive plays per 100 sampled hours (2012-21) | +0.515 |
|
||||||
| log DR lists with the artist before t (2023-) | +0.108 |
|
| log DR lists with the artist before t (2023-) | +0.105 |
|
||||||
| DR list recency 26w | -0.955 |
|
| DR list recency 26w | -0.984 |
|
||||||
| DR lists known before t at all | +0.572 |
|
| DR lists known before t at all | +0.593 |
|
||||||
| NMF Denmark in the window | +0.866 |
|
| NMF Denmark in the window | +0.836 |
|
||||||
| NMF Denmark top 10 in the window | +0.788 |
|
| NMF Denmark top 10 in the window | +0.682 |
|
||||||
| log NMF Denmark placements last 52w | -0.047 |
|
| log NMF Denmark placements last 52w | -0.067 |
|
||||||
| share of releases on NMF Denmark (2 years) | -0.354 |
|
| share of releases on NMF Denmark (2 years) | -0.273 |
|
||||||
| season index of the window (log, IFPI 2002-24) | +0.372 |
|
| season index of the window (log, IFPI 2002-24) | +0.334 |
|
||||||
|
|||||||
@@ -97,3 +97,15 @@ airplay model rank [--h 4] [--max-h 12] [--top 50] the ranking for h = 1..12 -
|
|||||||
- **Recalibration.** The add model's probabilities get a Platt map fitted honestly: an inner model trained on windows closed before the last 6 origins predicts those origins.
|
- **Recalibration.** The add model's probabilities get a Platt map fitted honestly: an inner model trained on windows closed before the last 6 origins predicts those origins.
|
||||||
- **NMF.** SuperFridayChart artist pages give every NMF placement per artist since 2022 (`nmf_artist_tracks`), including songs DR never played. Track pages give the countries, fetched for placements inside the modelling window first. Before a track page is read, NMF Denmark is recognised by its follower count. `airplay model nmf` compares DR adds after releases on NMF Denmark, on NMF elsewhere and not on NMF, raw and with controls.
|
- **NMF.** SuperFridayChart artist pages give every NMF placement per artist since 2022 (`nmf_artist_tracks`), including songs DR never played. Track pages give the countries, fetched for placements inside the modelling window first. Before a track page is read, NMF Denmark is recognised by its follower count. `airplay model nmf` compares DR adds after releases on NMF Denmark, on NMF elsewhere and not on NMF, raw and with controls.
|
||||||
- **Bauer.** The site shows each artist's plays per Bauer station over the last 7 days, with songs. Bauer has no lists or history, so Bauer adds join the model once the daily logs cover a few weeks.
|
- **Bauer.** The site shows each artist's plays per Bauer station over the last 7 days, with songs. Bauer has no lists or history, so Bauer adds join the model once the daily logs cover a few weeks.
|
||||||
|
|
||||||
|
## Update 16 Nov 2026: honest backtests
|
||||||
|
|
||||||
|
Following the external evaluation (`Airplay-Predictor: implementation evaluation`):
|
||||||
|
|
||||||
|
- **Leak test** (`airplay model leakcheck`): rebuilds every feature from a copy of the data cut at the origin week (`Panel::truncated`) and reports any feature that changes. It found the wave leak (the surge rule confirms an add with the following week). Fixed: "weeks since last add" counts only adds up to t − 1, and training windows close a week before the origin. The test now passes for all features.
|
||||||
|
- **Knowable pool:** an artist enters training, calibration, baselines and test rows only from their first knowable week: first release, DR play, chart entry, DR list, archived play or NMF placement (`Panel::first_known`). 7,219 artists are knowable at the start of the panel, 7,451 by the end, and 485 never.
|
||||||
|
- **Pre-/post-release:** both backtests report the model with the window's NMF placement blended from the artist's history (pre-release, as on the site) and with it known (post-release).
|
||||||
|
- **Release-timing baseline:** add rate by release in the window × weeks since the last release; for release events, by weeks since the previous release. It is weak: the models beat it by +0.4 per channel.
|
||||||
|
- **Recalibration:** pooled out-of-sample Platt and isotonic (`calib.rs`) are compared with the 6-origin Platt. For the release-event model (the site), pooled Platt is best per channel and isotonic for any channel, so the site uses those (`AddModel::fit_events_pooled`), fitted on rolling out-of-sample forecasts over the whole panel. For the window model, the 6-origin Platt is marginally best. All variants still over-predict above about 0.5.
|
||||||
|
- **Intervals:** block bootstrap over 8-week blocks.
|
||||||
|
- **Surge rule vs the lists** (`airplay model addcheck --tune`, docs/ADDCHECK.md): on list pairs at most 8 weeks apart, surge has recall 68% and precision ≥ 25% (F1 0.36), against onset-only 55% / 22%. A grid over the constants moves F1 only within noise, so the defaults stay.
|
||||||
|
|||||||
+73
-25
@@ -1,7 +1,12 @@
|
|||||||
//! How well the artist-level add definitions find real song adds: a song new on a DR list
|
//! How well the artist-level add definitions find real song adds, against DR's own lists.
|
||||||
//! (vs the previous list) should come with an add for the artist in between.
|
//!
|
||||||
|
//! Truth: an artist with a song that is new on a list compared with the previous list of the
|
||||||
|
//! same channel (pairs at most 8 weeks apart inside the DR panel). Detected: an add for that
|
||||||
|
//! artist on that channel between the two lists. Recall = truth found; precision = detected adds
|
||||||
|
//! that match a song newly on the list (a song added and dropped between lists, or added off-list,
|
||||||
|
//! counts against precision, so it is a lower bound). `--tune` searches the surge constants.
|
||||||
|
|
||||||
use super::adds::{Rot, ADD_RULE};
|
use super::adds::{Rot, ADD_RULE, SURGE_DEFAULT};
|
||||||
use super::common::thresholds;
|
use super::common::thresholds;
|
||||||
use super::panel::{key, Panel, CHANNELS};
|
use super::panel::{key, Panel, CHANNELS};
|
||||||
use crate::db::Db;
|
use crate::db::Db;
|
||||||
@@ -10,7 +15,14 @@ use chrono::{Duration, NaiveDate};
|
|||||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||||
use std::fmt::Write as _;
|
use std::fmt::Write as _;
|
||||||
|
|
||||||
pub fn run(db: &Db, p: &Panel) -> Result<String> {
|
struct Truth {
|
||||||
|
/// (channel indices, first week, last week, artists with a new song)
|
||||||
|
pairs: Vec<(Vec<usize>, usize, usize, HashSet<usize>)>,
|
||||||
|
/// all list pairs regardless of gap, for recall only: (channels, w0, w1, artists)
|
||||||
|
all: Vec<(Vec<usize>, usize, usize, HashSet<usize>)>,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn truth(db: &Db, p: &Panel) -> Result<Truth> {
|
||||||
let mut lists: BTreeMap<NaiveDate, HashMap<String, HashSet<(String, String)>>> = BTreeMap::new();
|
let mut lists: BTreeMap<NaiveDate, HashMap<String, HashSet<(String, String)>>> = BTreeMap::new();
|
||||||
for r in db.conn.prepare("SELECT list_date, channel, artist, title FROM rotation_lists WHERE category IN ('A', 'B', 'C', 'LIST')")?
|
for r in db.conn.prepare("SELECT list_date, channel, artist, title FROM rotation_lists WHERE category IN ('A', 'B', 'C', 'LIST')")?
|
||||||
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?, r.get::<_, String>(2)?, r.get::<_, String>(3)?)))? {
|
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?, r.get::<_, String>(2)?, r.get::<_, String>(3)?)))? {
|
||||||
@@ -22,34 +34,70 @@ pub fn run(db: &Db, p: &Panel) -> Result<String> {
|
|||||||
}
|
}
|
||||||
let by_name: HashMap<String, usize> = p.artists.iter().enumerate().map(|(i, a)| (key(&a.name), i)).collect();
|
let by_name: HashMap<String, usize> = p.artists.iter().enumerate().map(|(i, a)| (key(&a.name), i)).collect();
|
||||||
let week_of: HashMap<NaiveDate, usize> = p.weeks.iter().enumerate().map(|(i, d)| (*d, i)).collect();
|
let week_of: HashMap<NaiveDate, usize> = p.weeks.iter().enumerate().map(|(i, d)| (*d, i)).collect();
|
||||||
let tau = thresholds(p, p.w());
|
|
||||||
let mut md = String::new();
|
|
||||||
writeln!(md, "# Add definitions vs DR's own lists\n\nA song new on a list (vs the previous list for the channel) should come with an add for its artist between the two lists (± 1 week).\n")?;
|
|
||||||
writeln!(md, "| Definition | New songs on lists (artist in the pool) | Artist add found | Recall | Artist adds per artist-week |\n|---|---|---|---|---|")?;
|
|
||||||
for rule in ["onset", "surge"] {
|
|
||||||
let _ = ADD_RULE.set(rule.to_string()); // OnceLock: only the first set counts; run each rule in its own process
|
|
||||||
if ADD_RULE.get().map(String::as_str) != Some(rule) { continue; }
|
|
||||||
let r = Rot::new(p, &tau);
|
|
||||||
let (mut n, mut hit) = (0usize, 0usize);
|
|
||||||
let dates: Vec<NaiveDate> = lists.keys().copied().collect();
|
let dates: Vec<NaiveDate> = lists.keys().copied().collect();
|
||||||
|
let (mut pairs, mut all) = (Vec::new(), Vec::new());
|
||||||
for ch in ["p3", "p4", "p5", "p6beat", "p8jazz"] {
|
for ch in ["p3", "p4", "p5", "p6beat", "p8jazz"] {
|
||||||
let ds: Vec<&NaiveDate> = dates.iter().filter(|d| lists[d].contains_key(ch)).collect();
|
let ds: Vec<&NaiveDate> = dates.iter().filter(|d| lists[d].contains_key(ch)).collect();
|
||||||
for pair in ds.windows(2) {
|
|
||||||
let (Some(&w0), Some(&w1)) = (week_of.get(pair[0]), week_of.get(pair[1])) else { continue };
|
|
||||||
let (old, new) = (&lists[pair[0]][ch], &lists[pair[1]][ch]);
|
|
||||||
let chans: Vec<usize> = (0..CHANNELS.len()).filter(|&c| if ch == "p4" { CHANNELS[c].starts_with("p4") } else { CHANNELS[c] == ch }).collect();
|
let chans: Vec<usize> = (0..CHANNELS.len()).filter(|&c| if ch == "p4" { CHANNELS[c].starts_with("p4") } else { CHANNELS[c] == ch }).collect();
|
||||||
for (artist, song) in new.difference(old) {
|
for w in ds.windows(2) {
|
||||||
let _ = song;
|
let (Some(&w0), Some(&w1)) = (week_of.get(w[0]), week_of.get(w[1])) else { continue };
|
||||||
let Some(&a) = by_name.get(artist) else { continue };
|
let (old, new) = (&lists[w[0]][ch], &lists[w[1]][ch]);
|
||||||
n += 1;
|
let artists: HashSet<usize> = new.difference(old).filter_map(|(a, _)| by_name.get(a).copied()).collect();
|
||||||
let lo = w0.saturating_sub(1);
|
all.push((chans.clone(), w0, w1, artists.clone()));
|
||||||
let hi = (w1 + 1).min(p.w() - 1);
|
if w1 - w0 <= 8 { pairs.push((chans.clone(), w0, w1, artists)); }
|
||||||
if (lo..=hi).any(|w| chans.iter().any(|&c| r.add(a, c, w))) { hit += 1; }
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
Ok(Truth { pairs, all })
|
||||||
}
|
}
|
||||||
let total: usize = (0..p.a()).map(|a| (0..p.w()).filter(|&w| r.add_any(a, w)).count()).sum();
|
|
||||||
writeln!(md, "| {rule} | {n} | {hit} | {:.0}% | {:.4} |", 100.0 * hit as f64 / n.max(1) as f64, total as f64 / (p.a() * p.w()) as f64)?;
|
/// (recall on all pairs, recall on close pairs, precision on close pairs, detected, truth)
|
||||||
|
fn score(p: &Panel, r: &Rot, t: &Truth) -> (f64, f64, f64, usize, usize) {
|
||||||
|
let found = |chans: &[usize], w0: usize, w1: usize, a: usize| (w0.saturating_sub(1)..=(w1 + 1).min(p.w() - 1)).any(|w| chans.iter().any(|&c| r.add(a, c, w)));
|
||||||
|
let (mut n_all, mut hit_all) = (0usize, 0usize);
|
||||||
|
for (chans, w0, w1, arts) in &t.all { for &a in arts { n_all += 1; if found(chans, *w0, *w1, a) { hit_all += 1; } } }
|
||||||
|
let (mut n, mut hit, mut det, mut tp) = (0usize, 0usize, 0usize, 0usize);
|
||||||
|
for (chans, w0, w1, arts) in &t.pairs {
|
||||||
|
for &a in arts { n += 1; if found(chans, *w0, *w1, a) { hit += 1; } }
|
||||||
|
// detections strictly between the lists, for artists knowable then
|
||||||
|
for a in 0..p.a() {
|
||||||
|
if !p.known(a, *w0) { continue; }
|
||||||
|
if (*w0 + 1..=*w1).any(|w| chans.iter().any(|&c| r.add(a, c, w))) { det += 1; if arts.contains(&a) { tp += 1; } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(hit_all as f64 / n_all.max(1) as f64, hit as f64 / n.max(1) as f64, tp as f64 / det.max(1) as f64, det, n)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn run(db: &Db, p: &Panel, tune: bool) -> Result<String> {
|
||||||
|
let t = truth(db, p)?;
|
||||||
|
let tau = thresholds(p, p.w());
|
||||||
|
let mut md = String::new();
|
||||||
|
writeln!(md, "# Add definitions vs DR's own lists\n\nTruth: artists with a song new on a list vs the previous list of the channel. Recall over all list pairs ({}), and recall and precision over pairs at most 8 weeks apart ({}). Precision is a lower bound: songs added and dropped between lists, or added off-list, count against it.\n",
|
||||||
|
t.all.len(), t.pairs.len())?;
|
||||||
|
writeln!(md, "| Definition | Recall, all pairs | Recall, close pairs | Precision, close pairs | F1, close pairs | Adds detected | New songs |\n|---|---|---|---|---|---|---|")?;
|
||||||
|
let rule = ADD_RULE.get().cloned().unwrap_or_else(|| "surge".into());
|
||||||
|
let line = |md: &mut String, name: &str, r: &Rot| -> Result<f64> {
|
||||||
|
let (ra, rc, pc, det, n) = score(p, r, &t);
|
||||||
|
let f1 = if rc + pc > 0.0 { 2.0 * rc * pc / (rc + pc) } else { 0.0 };
|
||||||
|
writeln!(md, "| {name} | {:.0}% | {:.0}% | {:.0}% | {:.2} | {det} | {n} |", 100.0 * ra, 100.0 * rc, 100.0 * pc, f1)?;
|
||||||
|
Ok(f1)
|
||||||
|
};
|
||||||
|
if rule == "onset" {
|
||||||
|
line(&mut md, "onset", &Rot::new(p, &tau))?;
|
||||||
|
} else {
|
||||||
|
line(&mut md, &format!("surge {:?}", SURGE_DEFAULT), &Rot::new_with(p, &tau, SURGE_DEFAULT))?;
|
||||||
|
}
|
||||||
|
if tune {
|
||||||
|
let mut best: Vec<(f64, [f64; 4], f64, f64)> = Vec::new();
|
||||||
|
for f in [0.4, 0.6, 0.8] { for jt in [0.3, 0.5, 0.75] { for jb in [0.25, 0.5, 1.0] { for s in [0.0, 0.25, 0.5] {
|
||||||
|
let k = [f, jt, jb, s];
|
||||||
|
let r = Rot::new_with(p, &tau, k);
|
||||||
|
let (_, rc, pc, _, _) = score(p, &r, &t);
|
||||||
|
let f1 = if rc + pc > 0.0 { 2.0 * rc * pc / (rc + pc) } else { 0.0 };
|
||||||
|
best.push((f1, k, rc, pc));
|
||||||
|
} } } }
|
||||||
|
best.sort_by(|a, b| b.0.total_cmp(&a.0));
|
||||||
|
writeln!(md, "\nSurge constants tried (floor × τ, jump × τ, jump × baseline, held next week), best F1 on close pairs:\n\n| Constants | Recall | Precision | F1 |\n|---|---|---|---|")?;
|
||||||
|
for (f1, k, rc, pc) in best.iter().take(8) { writeln!(md, "| {k:?} | {:.0}% | {:.0}% | {f1:.3} |", 100.0 * rc, 100.0 * pc)?; }
|
||||||
}
|
}
|
||||||
Ok(md)
|
Ok(md)
|
||||||
}
|
}
|
||||||
|
|||||||
+248
-107
@@ -19,6 +19,7 @@
|
|||||||
use super::common::{mix, Rng};
|
use super::common::{mix, Rng};
|
||||||
use super::features::{self, LAGS, P};
|
use super::features::{self, LAGS, P};
|
||||||
use super::glm::{self, Design};
|
use super::glm::{self, Design};
|
||||||
|
use super::calib::Calib;
|
||||||
use super::panel::{group, Panel, C, GROUPS};
|
use super::panel::{group, Panel, C, GROUPS};
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use std::fmt::Write as _;
|
use std::fmt::Write as _;
|
||||||
@@ -96,6 +97,22 @@ pub fn released_in_week(p: &Panel, a: usize, w: usize) -> bool {
|
|||||||
dates.iter().any(|d| *d >= start && *d < start + chrono::Duration::days(7))
|
dates.iter().any(|d| *d >= start && *d < start + chrono::Duration::days(7))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Weeks since the artist's latest release as of week t, in bins: <6, 6–11, 12–25, 26–51, 52+, none.
|
||||||
|
pub fn since_bin(p: &Panel, a: usize, t: usize) -> usize {
|
||||||
|
let Some(dates) = p.releases[a].as_ref() else { return 5 };
|
||||||
|
let end_t = p.weeks[t] + chrono::Duration::days(6);
|
||||||
|
match dates.iter().rev().find(|d| **d <= end_t).map(|d| (end_t - *d).num_days() / 7) {
|
||||||
|
None => 5, Some(w) if w < 6 => 0, Some(w) if w < 12 => 1, Some(w) if w < 26 => 2, Some(w) if w < 52 => 3, Some(_) => 4,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Does the artist release something in the window weeks t+h+1 ..= t+h+win?
|
||||||
|
pub fn releases_in(p: &Panel, a: usize, t: usize, h: usize, win: usize) -> bool {
|
||||||
|
let Some(dates) = p.releases[a].as_ref() else { return false };
|
||||||
|
let (w0, w1) = window(p, t, h, win);
|
||||||
|
dates.iter().any(|d| *d >= w0 && *d < w1)
|
||||||
|
}
|
||||||
|
|
||||||
/// A clean release event in week w: a release that week and none in the 6 weeks before it, so an
|
/// A clean release event in week w: a release that week and none in the 6 weeks before it, so an
|
||||||
/// add that follows can't be the previous song's (DR's counts are per artist, not per song).
|
/// add that follows can't be the previous song's (DR's counts are per artist, not per song).
|
||||||
pub const CLEAN_WEEKS: usize = 6;
|
pub const CLEAN_WEEKS: usize = 6;
|
||||||
@@ -242,8 +259,15 @@ pub struct Rot {
|
|||||||
pub static ADD_RULE: std::sync::OnceLock<String> = std::sync::OnceLock::new();
|
pub static ADD_RULE: std::sync::OnceLock<String> = std::sync::OnceLock::new();
|
||||||
fn surge_rule() -> bool { ADD_RULE.get().is_none_or(|r| r != "onset") }
|
fn surge_rule() -> bool { ADD_RULE.get().is_none_or(|r| r != "onset") }
|
||||||
|
|
||||||
|
/// Surge-rule constants: [floor (× τ), jump over the baseline (× τ), jump (× baseline), held the
|
||||||
|
/// next week (× τ and × baseline)]. Tuned against DR's lists (`airplay model addcheck --tune`).
|
||||||
|
pub const SURGE_DEFAULT: [f64; 4] = [0.6, 0.5, 0.5, 0.25];
|
||||||
|
pub static SURGE: std::sync::OnceLock<[f64; 4]> = std::sync::OnceLock::new();
|
||||||
|
|
||||||
impl Rot {
|
impl Rot {
|
||||||
pub fn new(p: &Panel, tau: &[u32; C]) -> Rot {
|
pub fn new(p: &Panel, tau: &[u32; C]) -> Rot { Self::new_with(p, tau, *SURGE.get().unwrap_or(&SURGE_DEFAULT)) }
|
||||||
|
|
||||||
|
pub fn new_with(p: &Panel, tau: &[u32; C], k: [f64; 4]) -> Rot {
|
||||||
let nw = p.w();
|
let nw = p.w();
|
||||||
let mut rot = vec![false; p.a() * C * nw];
|
let mut rot = vec![false; p.a() * C * nw];
|
||||||
let mut any = vec![false; p.a() * nw];
|
let mut any = vec![false; p.a() * nw];
|
||||||
@@ -266,7 +290,7 @@ impl Rot {
|
|||||||
b.sort_by(f64::total_cmp);
|
b.sort_by(f64::total_cmp);
|
||||||
let base = b[4];
|
let base = b[4];
|
||||||
let (y0, y1) = (p.y(a, c, w), p.y(a, c, w + 1));
|
let (y0, y1) = (p.y(a, c, w), p.y(a, c, w + 1));
|
||||||
y0 >= (0.6 * t).max(base + (0.5 * t).max(0.5 * base)) && y1 >= (0.6 * t).max(base + (0.25 * t).max(0.25 * base))
|
y0 >= (k[0] * t).max(base + (k[1] * t).max(k[2] * base)) && y1 >= (k[0] * t).max(base + (k[3] * t).max(k[3] * base))
|
||||||
};
|
};
|
||||||
if (onset || jump) && (1..=GAP).all(|k| !adds[(a * C + c) * nw + w - k]) { adds[(a * C + c) * nw + w] = true; }
|
if (onset || jump) && (1..=GAP).all(|k| !adds[(a * C + c) * nw + w - k]) { adds[(a * C + c) * nw + w] = true; }
|
||||||
}
|
}
|
||||||
@@ -366,6 +390,9 @@ pub struct AddModel {
|
|||||||
/// enough history to fit it honestly.
|
/// enough history to fit it honestly.
|
||||||
pub platt_channel: (f64, f64),
|
pub platt_channel: (f64, f64),
|
||||||
pub platt_any: (f64, f64),
|
pub platt_any: (f64, f64),
|
||||||
|
/// Recalibration fitted on pooled out-of-sample forecasts (replaces the Platt pair when set).
|
||||||
|
pub calib_channel: Option<Calib>,
|
||||||
|
pub calib_any: Option<Calib>,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Recent origins used to fit the recalibration.
|
/// Recent origins used to fit the recalibration.
|
||||||
@@ -393,8 +420,39 @@ impl AddModel {
|
|||||||
/// the artist's state the week before (earlier releases and NMF placements count as history;
|
/// the artist's state the week before (earlier releases and NMF placements count as history;
|
||||||
/// the new song's own NMF placement is the scenario variable).
|
/// the new song's own NMF placement is the scenario variable).
|
||||||
pub fn fit_events(p: &Panel, r: &Rot, last_week: usize) -> AddModel { Self::fit_with(p, r, 0, last_week, true, W) }
|
pub fn fit_events(p: &Panel, r: &Rot, last_week: usize) -> AddModel { Self::fit_with(p, r, 0, last_week, true, W) }
|
||||||
|
/// The site's release-event model (8-week window, pooled recalibration).
|
||||||
|
pub fn fit_events_site(p: &Panel, r: &Rot, last_week: usize) -> AddModel { Self::fit_events_pooled(p, r, last_week, W) }
|
||||||
/// The release-event model for a window of `win` weeks.
|
/// The release-event model for a window of `win` weeks.
|
||||||
pub fn fit_events_w(p: &Panel, r: &Rot, last_week: usize, win: usize) -> AddModel { Self::fit_with(p, r, 0, last_week, true, win) }
|
pub fn fit_events_w(p: &Panel, r: &Rot, last_week: usize, win: usize) -> AddModel { Self::fit_events_pooled(p, r, last_week, win) }
|
||||||
|
|
||||||
|
/// The release-event model recalibrated on pooled out-of-sample forecasts: a rolling refit (every
|
||||||
|
/// 4 weeks) predicts each later week's clean releases from releases whose outcome was already
|
||||||
|
/// known; Platt per station and isotonic for any station are fitted on all those forecasts
|
||||||
|
/// (the combination that scored best in the backtest), then attached to the model fitted on
|
||||||
|
/// everything known by `last_week`.
|
||||||
|
pub fn fit_events_pooled(p: &Panel, r: &Rot, last_week: usize, win: usize) -> AddModel {
|
||||||
|
let (mut pc_pairs, mut pa_pairs) = (Vec::new(), Vec::new());
|
||||||
|
let first = LAGS - 1 + win + 1 + 8;
|
||||||
|
let last = last_week.saturating_sub(win + 1);
|
||||||
|
let mut model: Option<AddModel> = None;
|
||||||
|
for t0 in first..=last {
|
||||||
|
if (t0 - first) % 4 == 0 || model.is_none() { model = Some(Self::fit_raw(p, r, 0, t0, true, win)); }
|
||||||
|
let m = model.as_ref().unwrap();
|
||||||
|
for a in 0..p.a() {
|
||||||
|
if !p.known(a, t0) || !clean_release(p, a, t0 + 1) { continue; }
|
||||||
|
let q = p_nmf(p, a, t0);
|
||||||
|
let sc = |b: bool| Scenario::new(true, b).window(win);
|
||||||
|
let (rcn, ran) = m.predict_raw(p, r, a, t0, 0, sc(true));
|
||||||
|
let (rcr, rar) = m.predict_raw(p, r, a, t0, 0, sc(false));
|
||||||
|
pa_pairs.push((q * ran + (1.0 - q) * rar, r.add_any_in(a, t0, 0, win)));
|
||||||
|
for c in 0..C { pc_pairs.push((q * rcn[c] + (1.0 - q) * rcr[c], r.add_in(a, c, t0, 0, win))); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut m = Self::fit_raw(p, r, 0, last_week, true, win);
|
||||||
|
m.calib_channel = Some(Calib::platt(&pc_pairs));
|
||||||
|
m.calib_any = Some(Calib::isotonic(&pa_pairs));
|
||||||
|
m
|
||||||
|
}
|
||||||
|
|
||||||
fn fit_with(p: &Panel, r: &Rot, h: usize, last_week: usize, events: bool, win: usize) -> AddModel {
|
fn fit_with(p: &Panel, r: &Rot, h: usize, last_week: usize, events: bool, win: usize) -> AddModel {
|
||||||
let mut m = Self::fit_raw(p, r, h, last_week, events, win);
|
let mut m = Self::fit_raw(p, r, h, last_week, events, win);
|
||||||
@@ -407,6 +465,7 @@ impl AddModel {
|
|||||||
let (mut dc, mut da) = (Design::new(2), Design::new(2));
|
let (mut dc, mut da) = (Design::new(2), Design::new(2));
|
||||||
for t in c0..=hi {
|
for t in c0..=hi {
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
|
if !p.known(a, t) { continue; }
|
||||||
if events && !clean_release(p, a, t + 1) { continue; }
|
if events && !clean_release(p, a, t + 1) { continue; }
|
||||||
let (pc, pa) = inner.predict(p, r, a, t, Scenario::actual(win), None);
|
let (pc, pa) = inner.predict(p, r, a, t, Scenario::actual(win), None);
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
@@ -431,6 +490,8 @@ impl AddModel {
|
|||||||
let sc = Scenario::actual(win);
|
let sc = Scenario::actual(win);
|
||||||
for t in origins_w(h, last_week, win) {
|
for t in origins_w(h, last_week, win) {
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
|
// Only artists knowable at the origin (no survivorship from later discovery).
|
||||||
|
if !p.known(a, t) { continue; }
|
||||||
if events && !clean_release(p, a, t + 1) { continue; }
|
if events && !clean_release(p, a, t + 1) { continue; }
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
// Channels where the artist hasn't been heard for 12 weeks: keep a quarter, weighted 4.
|
// Channels where the artist hasn't been heard for 12 weeks: keep a quarter, weighted 4.
|
||||||
@@ -443,7 +504,7 @@ impl AddModel {
|
|||||||
da.push(&xa, r.add_any_in(a, t, h, win) as i32 as f64, 1.0);
|
da.push(&xa, r.add_any_in(a, t, h, win) as i32 as f64, 1.0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
AddModel { h, win, channel: glm::logit(&dc, 1.0), any: glm::logit(&da, 1.0), platt_channel: (0.0, 1.0), platt_any: (0.0, 1.0) }
|
AddModel { h, win, channel: glm::logit(&dc, 1.0), any: glm::logit(&da, 1.0), platt_channel: (0.0, 1.0), platt_any: (0.0, 1.0), calib_channel: None, calib_any: None }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// (P(add) per channel, P(add on any channel)) from origin t.
|
/// (P(add) per channel, P(add on any channel)) from origin t.
|
||||||
@@ -455,13 +516,24 @@ impl AddModel {
|
|||||||
/// As `predict`, for a window starting h weeks after t (the release-event model: a release in
|
/// As `predict`, for a window starting h weeks after t (the release-event model: a release in
|
||||||
/// week t+h+1, with the time-since features moved forward by `sc.shift`).
|
/// week t+h+1, with the time-since features moved forward by `sc.shift`).
|
||||||
pub fn predict_at(&self, p: &Panel, r: &Rot, a: usize, t: usize, h: usize, sc: Scenario, chart: Option<&dyn Fn(&mut [f64], [usize; 4])>) -> ([f64; C], f64) {
|
pub fn predict_at(&self, p: &Panel, r: &Rot, a: usize, t: usize, h: usize, sc: Scenario, chart: Option<&dyn Fn(&mut [f64], [usize; 4])>) -> ([f64; C], f64) {
|
||||||
|
self.predict_full(p, r, a, t, h, sc, chart, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The logits' own probabilities, without recalibration or the any-channel cap.
|
||||||
|
pub fn predict_raw(&self, p: &Panel, r: &Rot, a: usize, t: usize, h: usize, sc: Scenario) -> ([f64; C], f64) {
|
||||||
|
self.predict_full(p, r, a, t, h, sc, None, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
fn predict_full(&self, p: &Panel, r: &Rot, a: usize, t: usize, h: usize, sc: Scenario, chart: Option<&dyn Fn(&mut [f64], [usize; 4])>, raw: bool) -> ([f64; C], f64) {
|
||||||
let sc = sc.window(self.win);
|
let sc = sc.window(self.win);
|
||||||
let mut x = vec![0.0; PC];
|
let mut x = vec![0.0; PC];
|
||||||
let mut pc = [0.0; C];
|
let mut pc = [0.0; C];
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
row_channel(p, r, a, c, t, h, sc, &mut x);
|
row_channel(p, r, a, c, t, h, sc, &mut x);
|
||||||
if let Some(f) = chart { f(&mut x, CHART_IDX_CHANNEL); }
|
if let Some(f) = chart { f(&mut x, CHART_IDX_CHANNEL); }
|
||||||
pc[c] = platt(self.platt_channel, glm::sigmoid(glm::dot(&self.channel, &x)));
|
let p0 = glm::sigmoid(glm::dot(&self.channel, &x)).clamp(1e-6, 1.0 - 1e-6);
|
||||||
|
pc[c] = if raw { p0 } else { self.calib_channel.as_ref().map(|k| k.apply(p0)).unwrap_or_else(|| platt(self.platt_channel, p0)) };
|
||||||
}
|
}
|
||||||
let mut xa = vec![0.0; PA];
|
let mut xa = vec![0.0; PA];
|
||||||
row_any(p, r, a, t, h, sc, &mut xa);
|
row_any(p, r, a, t, h, sc, &mut xa);
|
||||||
@@ -471,7 +543,10 @@ impl AddModel {
|
|||||||
let mut best = [0.0f64; GROUPS];
|
let mut best = [0.0f64; GROUPS];
|
||||||
for c in 0..C { best[group(c)] = best[group(c)].max(pc[c]); }
|
for c in 0..C { best[group(c)] = best[group(c)].max(pc[c]); }
|
||||||
let cap = 1.0 - best.iter().map(|p| 1.0 - p).product::<f64>();
|
let cap = 1.0 - best.iter().map(|p| 1.0 - p).product::<f64>();
|
||||||
(pc, platt(self.platt_any, glm::sigmoid(glm::dot(&self.any, &xa))).min(cap.max(1e-6)))
|
let p0 = glm::sigmoid(glm::dot(&self.any, &xa)).clamp(1e-6, 1.0 - 1e-6);
|
||||||
|
if raw { return (pc, p0); }
|
||||||
|
let pa = self.calib_any.as_ref().map(|k| k.apply(p0)).unwrap_or_else(|| platt(self.platt_any, p0));
|
||||||
|
(pc, pa.min(cap.max(1e-6)))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -499,7 +574,8 @@ fn average_precision(mut v: Vec<(f64, bool)>) -> f64 {
|
|||||||
fn skill(model: &[f64], base: &[f64]) -> (f64, f64, f64) {
|
fn skill(model: &[f64], base: &[f64]) -> (f64, f64, f64) {
|
||||||
let point = 1.0 - model.iter().sum::<f64>() / base.iter().sum::<f64>();
|
let point = 1.0 - model.iter().sum::<f64>() / base.iter().sum::<f64>();
|
||||||
let n = model.len();
|
let n = model.len();
|
||||||
let block = 3.min(n);
|
// Blocks of 8 weeks: neighbouring origins share most of an 8-week target window.
|
||||||
|
let block = 8.min(n);
|
||||||
let mut rng = Rng(7);
|
let mut rng = Rng(7);
|
||||||
let mut d: Vec<f64> = (0..2000).map(|_| {
|
let mut d: Vec<f64> = (0..2000).map(|_| {
|
||||||
let (mut lm, mut lb, mut k) = (0.0, 0.0, 0);
|
let (mut lm, mut lb, mut k) = (0.0, 0.0, 0);
|
||||||
@@ -510,33 +586,122 @@ fn skill(model: &[f64], base: &[f64]) -> (f64, f64, f64) {
|
|||||||
(point, d[100], d[1899])
|
(point, d[100], d[1899])
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Scores of several models over walk-forward origins.
|
||||||
|
struct Scores {
|
||||||
|
names: Vec<&'static str>,
|
||||||
|
wk: Vec<Vec<f64>>, // per origin, per model: summed log-loss
|
||||||
|
pairs: Vec<Vec<(f64, bool)>>, // per model
|
||||||
|
calib: Vec<[[f64; 3]; 10]>, // per model: [sum p, hits, n] per decile
|
||||||
|
n: f64,
|
||||||
|
hits: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Scores {
|
||||||
|
fn new(names: Vec<&'static str>) -> Scores {
|
||||||
|
let k = names.len();
|
||||||
|
Scores { names, wk: Vec::new(), pairs: vec![Vec::new(); k], calib: vec![[[0.0; 3]; 10]; k], n: 0.0, hits: 0.0 }
|
||||||
|
}
|
||||||
|
fn start_origin(&mut self) { self.wk.push(vec![0.0; self.names.len()]); }
|
||||||
|
fn add(&mut self, preds: &[f64], y: bool) {
|
||||||
|
let w = self.wk.last_mut().unwrap();
|
||||||
|
for (k, p) in preds.iter().enumerate() {
|
||||||
|
let p = p.clamp(1e-6, 1.0 - 1e-6);
|
||||||
|
w[k] += ll(p, y);
|
||||||
|
self.pairs[k].push((p, y));
|
||||||
|
let d = ((p * 10.0) as usize).min(9);
|
||||||
|
self.calib[k][d][0] += p; self.calib[k][d][1] += y as i32 as f64; self.calib[k][d][2] += 1.0;
|
||||||
|
}
|
||||||
|
self.n += 1.0; self.hits += y as i32 as f64;
|
||||||
|
}
|
||||||
|
fn col(&self, k: usize) -> Vec<f64> { self.wk.iter().map(|w| w[k]).collect() }
|
||||||
|
fn logloss(&self, k: usize) -> f64 { self.col(k).iter().sum::<f64>() / self.n.max(1.0) }
|
||||||
|
fn prauc(&self, k: usize) -> f64 { average_precision(self.pairs[k].clone()) }
|
||||||
|
fn calib_table(&self, md: &mut String, ks: &[usize]) -> Result<()> {
|
||||||
|
write!(md, "| Predicted |")?;
|
||||||
|
for &k in ks { write!(md, " {}: rows / mean predicted / observed |", self.names[k])?; }
|
||||||
|
writeln!(md, "\n|---|{}", ks.iter().map(|_| "---|").collect::<String>())?;
|
||||||
|
for d in 0..10 {
|
||||||
|
if ks.iter().all(|&k| self.calib[k][d][2] == 0.0) { continue; }
|
||||||
|
write!(md, "| {:.1}–{:.1} |", d as f64 / 10.0, (d + 1) as f64 / 10.0)?;
|
||||||
|
for &k in ks {
|
||||||
|
let c = self.calib[k][d];
|
||||||
|
if c[2] > 0.0 { write!(md, " {} / {:.3} / {:.3} |", c[2], c[0] / c[2], c[1] / c[2])?; } else { write!(md, " — |")?; }
|
||||||
|
}
|
||||||
|
writeln!(md)?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pooled out-of-sample forecasts for recalibration: (origin, raw prediction, outcome).
|
||||||
|
struct Pool(Vec<(usize, f64, bool)>);
|
||||||
|
impl Pool {
|
||||||
|
/// Platt and isotonic fitted on forecasts whose outcome was known by `t0` (origin + gap <= t0).
|
||||||
|
fn fit(&self, t0: usize, gap: usize) -> (Calib, Calib) {
|
||||||
|
let pairs: Vec<(f64, bool)> = self.0.iter().filter(|(o, _, _)| o + gap <= t0).map(|(_, p, y)| (*p, *y)).collect();
|
||||||
|
(Calib::platt(&pairs), Calib::isotonic(&pairs))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The models scored in both backtests (index: meaning).
|
||||||
|
const SCORED: [&str; 8] = ["rate by in-rotation-now", "rate by last-12-weeks level", "release timing only",
|
||||||
|
"model, calibrated on the last 6 origins (pre-release)", "model, raw (pre-release)",
|
||||||
|
"model + pooled Platt (pre-release)", "model + pooled isotonic (pre-release)", "model, NMF known (post-release)"];
|
||||||
|
|
||||||
|
fn table(md: &mut String, s_c: &Scores, s_a: &Scores, rows: &mut Vec<(String, usize, String, f64)>, h: usize, prefix: &str) -> Result<()> {
|
||||||
|
writeln!(md, "| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |\n|---|---|---|---|---|")?;
|
||||||
|
for k in 0..SCORED.len() {
|
||||||
|
let (lc, pc, la, pa) = (s_c.logloss(k), s_c.prauc(k), s_a.logloss(k), s_a.prauc(k));
|
||||||
|
writeln!(md, "| {} | {lc:.4} | {pc:.3} | {la:.4} | {pa:.3} |", SCORED[k])?;
|
||||||
|
for (m, v) in [("logloss_ch", lc), ("prauc_ch", pc), ("logloss_any", la), ("prauc_any", pa)] {
|
||||||
|
rows.push((SCORED[k].to_string(), h, format!("{prefix}{m}"), v));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writeln!(md, "\nSkill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):\n")?;
|
||||||
|
writeln!(md, "| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |\n|---|---|---|---|---|")?;
|
||||||
|
for k in [3usize, 5, 6] {
|
||||||
|
let f = |s: &Scores, b: usize| { let x = skill(&s.col(k), &s.col(b)); format!("{:+.3} ({:+.3} to {:+.3})", x.0, x.1, x.2) };
|
||||||
|
writeln!(md, "| {} | {} | {} | {} | {} |", SCORED[k], f(s_c, 1), f(s_c, 2), f(s_a, 1), f(s_a, 2))?;
|
||||||
|
rows.push((SCORED[k].to_string(), h, format!("{prefix}skill_any_vs_level"), skill(&s_a.col(k), &s_a.col(1)).0));
|
||||||
|
rows.push((SCORED[k].to_string(), h, format!("{prefix}skill_ch_vs_level"), skill(&s_c.col(k), &s_c.col(1)).0));
|
||||||
|
}
|
||||||
|
writeln!(md, "\nCalibration, any channel:\n")?;
|
||||||
|
s_a.calib_table(md, &[3, 5, 6])?;
|
||||||
|
writeln!(md, "\nCalibration, per channel:\n")?;
|
||||||
|
s_c.calib_table(md, &[3, 5, 6])?;
|
||||||
|
writeln!(md)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Release-timing-only rates: per station (and any) by whether the artist releases in the window
|
||||||
|
/// and weeks since their last release. [key][hits, n]
|
||||||
|
struct Timing { ch: Vec<[f64; 2]>, any: Vec<[f64; 2]> }
|
||||||
|
impl Timing {
|
||||||
|
fn new() -> Timing { Timing { ch: vec![[1.0, 50.0]; C * 12], any: vec![[1.0, 20.0]; 12] } }
|
||||||
|
fn key(p: &Panel, a: usize, t: usize, h: usize, win: usize) -> usize { releases_in(p, a, t, h, win) as usize * 6 + since_bin(p, a, t) }
|
||||||
|
}
|
||||||
|
|
||||||
/// Walk-forward: at each origin t0 the models see only windows that had closed by t0.
|
/// Walk-forward: at each origin t0 the models see only windows that had closed by t0.
|
||||||
pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
||||||
let r = Rot::new(p, tau);
|
let r = Rot::new(p, tau);
|
||||||
let mut rows = Vec::new();
|
let mut rows = Vec::new();
|
||||||
writeln!(md, "# Backtest: new adds\n")?;
|
writeln!(md, "# Backtest: new adds\n")?;
|
||||||
writeln!(md, "An add = the artist enters rotation (plays ≥ τ) after at least {GAP} weeks out of rotation: per channel, and on any DR channel after {GAP} weeks out of rotation everywhere. Target: an add within the {W} weeks that start h weeks after the forecast origin. Origins from week {}; each fit uses only windows closed by its origin.\n", start + 1)?;
|
writeln!(md, "An add = the artist enters rotation (plays ≥ τ) after at least {GAP} weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the {W} weeks that start h weeks after the forecast origin. Origins from week {}; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.\n", start + 1)?;
|
||||||
for &h in horizons {
|
for &h in horizons {
|
||||||
let last_origin = p.w() - 1 - h - W;
|
if p.w() < start + h + W + 2 { continue; }
|
||||||
if last_origin < start { continue; }
|
let last_origin = p.w() - 1 - h - W - 1;
|
||||||
// [model][week] log-loss sums; models: base(rot now), base(popularity bins), logit
|
let (mut s_c, mut s_a) = (Scores::new(SCORED.to_vec()), Scores::new(SCORED.to_vec()));
|
||||||
let names = ["rate by in-rotation-now", "rate by last-12-weeks level", "logit (pre-release: NMF from history)", "logit (post-release: NMF known)"];
|
let (mut pool_c, mut pool_a) = (Pool(Vec::new()), Pool(Vec::new()));
|
||||||
let mut wk_c: Vec<[f64; 4]> = Vec::new();
|
let lvl = |m12: f64| ((m12.ln_1p() / 0.5) as usize).min(7);
|
||||||
let mut wk_a: Vec<[f64; 4]> = Vec::new();
|
|
||||||
let mut pairs_c: [Vec<(f64, bool)>; 4] = Default::default();
|
|
||||||
let mut pairs_a: [Vec<(f64, bool)>; 4] = Default::default();
|
|
||||||
let mut calib = [[0.0f64; 3]; 10];
|
|
||||||
let (mut nc, mut na, mut hits_c, mut hits_a) = (0.0, 0.0, 0.0, 0.0);
|
|
||||||
for t0 in start..=last_origin {
|
for t0 in start..=last_origin {
|
||||||
let m = AddModel::fit(p, &r, h, t0);
|
let m = AddModel::fit(p, &r, h, t0);
|
||||||
// Baselines from the same closed windows.
|
// Baselines from the same closed windows.
|
||||||
let mut by_rot = [[1.0f64, 2.0]; 2];
|
let (mut by_rot, mut by_lvl, mut any_rot, mut any_lvl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8], [[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
||||||
let mut by_lvl = [[1.0f64, 2.0]; 8];
|
let mut tm = Timing::new();
|
||||||
let mut any_rot = [[1.0f64, 2.0]; 2];
|
|
||||||
let mut any_lvl = [[1.0f64, 2.0]; 8];
|
|
||||||
let lvl = |m12: f64| ((m12.ln_1p() / 0.5) as usize).min(7);
|
|
||||||
for t in origins(h, t0) {
|
for t in origins(h, t0) {
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
|
if !p.known(a, t) { continue; }
|
||||||
|
let key = Timing::key(p, a, t, h, W);
|
||||||
let mut tot = 0.0;
|
let mut tot = 0.0;
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
let y = r.add_in_window(a, c, t, h) as i32 as f64;
|
let y = r.add_in_window(a, c, t, h) as i32 as f64;
|
||||||
@@ -544,76 +709,59 @@ pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md:
|
|||||||
by_rot[on][0] += y; by_rot[on][1] += 1.0;
|
by_rot[on][0] += y; by_rot[on][1] += 1.0;
|
||||||
let m12 = (0..LAGS).map(|i| p.y(a, c, t - i)).sum::<f64>() / LAGS as f64;
|
let m12 = (0..LAGS).map(|i| p.y(a, c, t - i)).sum::<f64>() / LAGS as f64;
|
||||||
let b = lvl(m12); by_lvl[b][0] += y; by_lvl[b][1] += 1.0;
|
let b = lvl(m12); by_lvl[b][0] += y; by_lvl[b][1] += 1.0;
|
||||||
|
tm.ch[c * 12 + key][0] += y; tm.ch[c * 12 + key][1] += 1.0;
|
||||||
tot += m12;
|
tot += m12;
|
||||||
}
|
}
|
||||||
let y = r.add_any_in_window(a, t, h) as i32 as f64;
|
let y = r.add_any_in_window(a, t, h) as i32 as f64;
|
||||||
let on = r.any(a, t) as usize;
|
let on = r.any(a, t) as usize;
|
||||||
any_rot[on][0] += y; any_rot[on][1] += 1.0;
|
any_rot[on][0] += y; any_rot[on][1] += 1.0;
|
||||||
let b = lvl(tot); any_lvl[b][0] += y; any_lvl[b][1] += 1.0;
|
let b = lvl(tot); any_lvl[b][0] += y; any_lvl[b][1] += 1.0;
|
||||||
|
tm.any[key][0] += y; tm.any[key][1] += 1.0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let (mut lc, mut la) = ([0.0; 4], [0.0; 4]);
|
let gap = h + W + 1;
|
||||||
|
let ((pl_c, iso_c), (pl_a, iso_a)) = (pool_c.fit(t0, gap), pool_a.fit(t0, gap));
|
||||||
|
s_c.start_origin(); s_a.start_origin();
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
// Pre-release: the window's NMF placement isn't known yet; blend by the artist's NMF rate.
|
if !p.known(a, t0) { continue; }
|
||||||
let q = p_nmf(p, a, t0);
|
let q = p_nmf(p, a, t0);
|
||||||
let (pcn, pan) = m.predict(p, &r, a, t0, Scenario { nmf: Some(true), ..Scenario::ACTUAL }, None);
|
let nmf = |b: bool| Scenario { nmf: Some(b), ..Scenario::ACTUAL };
|
||||||
let (pcr, par) = m.predict(p, &r, a, t0, Scenario { nmf: Some(false), ..Scenario::ACTUAL }, None);
|
let (pcn, pan) = m.predict(p, &r, a, t0, nmf(true), None);
|
||||||
let pc: [f64; C] = std::array::from_fn(|c| q * pcn[c] + (1.0 - q) * pcr[c]);
|
let (pcr, par) = m.predict(p, &r, a, t0, nmf(false), None);
|
||||||
let pa = q * pan + (1.0 - q) * par;
|
let (rcn, ran) = m.predict_raw(p, &r, a, t0, h, nmf(true));
|
||||||
|
let (rcr, rar) = m.predict_raw(p, &r, a, t0, h, nmf(false));
|
||||||
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::ACTUAL, None);
|
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::ACTUAL, None);
|
||||||
|
let key = Timing::key(p, a, t0, h, W);
|
||||||
let mut tot = 0.0;
|
let mut tot = 0.0;
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
let y = r.add_in_window(a, c, t0, h);
|
let y = r.add_in_window(a, c, t0, h);
|
||||||
let on = r.on(a, c, t0) as usize;
|
let on = r.on(a, c, t0) as usize;
|
||||||
let m12 = (0..LAGS).map(|i| p.y(a, c, t0 - i)).sum::<f64>() / LAGS as f64;
|
let m12 = (0..LAGS).map(|i| p.y(a, c, t0 - i)).sum::<f64>() / LAGS as f64;
|
||||||
tot += m12;
|
tot += m12;
|
||||||
let preds = [by_rot[on][0] / by_rot[on][1], by_lvl[lvl(m12)][0] / by_lvl[lvl(m12)][1], pc[c], pcx[c]];
|
let raw = q * rcn[c] + (1.0 - q) * rcr[c];
|
||||||
for k in 0..4 { lc[k] += ll(preds[k], y); pairs_c[k].push((preds[k], y)); }
|
let tmc = tm.ch[c * 12 + key];
|
||||||
let d = ((pc[c] * 10.0) as usize).min(9);
|
s_c.add(&[by_rot[on][0] / by_rot[on][1], by_lvl[lvl(m12)][0] / by_lvl[lvl(m12)][1], tmc[0] / tmc[1],
|
||||||
calib[d][0] += pc[c]; calib[d][1] += y as i32 as f64; calib[d][2] += 1.0;
|
q * pcn[c] + (1.0 - q) * pcr[c], raw, pl_c.apply(raw), iso_c.apply(raw), pcx[c]], y);
|
||||||
nc += 1.0; hits_c += y as i32 as f64;
|
pool_c.0.push((t0, raw, y));
|
||||||
}
|
}
|
||||||
let y = r.add_any_in_window(a, t0, h);
|
let y = r.add_any_in_window(a, t0, h);
|
||||||
let on = r.any(a, t0) as usize;
|
let on = r.any(a, t0) as usize;
|
||||||
let preds = [any_rot[on][0] / any_rot[on][1], any_lvl[lvl(tot)][0] / any_lvl[lvl(tot)][1], pa, pax];
|
let raw = q * ran + (1.0 - q) * rar;
|
||||||
for k in 0..4 { la[k] += ll(preds[k], y); pairs_a[k].push((preds[k], y)); }
|
let tma = tm.any[key];
|
||||||
na += 1.0; hits_a += y as i32 as f64;
|
s_a.add(&[any_rot[on][0] / any_rot[on][1], any_lvl[lvl(tot)][0] / any_lvl[lvl(tot)][1], tma[0] / tma[1],
|
||||||
|
q * pan + (1.0 - q) * par, raw, pl_a.apply(raw), iso_a.apply(raw), pax], y);
|
||||||
|
pool_a.0.push((t0, raw, y));
|
||||||
}
|
}
|
||||||
wk_c.push(lc); wk_a.push(la);
|
|
||||||
eprintln!("adds backtest h={h}: origin {} done", p.weeks[t0]);
|
eprintln!("adds backtest h={h}: origin {} done", p.weeks[t0]);
|
||||||
}
|
}
|
||||||
let nwk = wk_c.len();
|
writeln!(md, "## Window starting h = {h} weeks ahead ({} forecast origins)\n", s_c.wk.len())?;
|
||||||
writeln!(md, "## Window starting h = {h} weeks ahead ({nwk} forecast origins)\n")?;
|
writeln!(md, "Base rates: per channel {:.2}% of artist-channels get an add in the window; on any channel {:.1}% of artists.\n", 100.0 * s_c.hits / s_c.n.max(1.0), 100.0 * s_a.hits / s_a.n.max(1.0))?;
|
||||||
writeln!(md, "Base rates: per channel {:.2}% of artist-channels get an add in the window; on any channel {:.1}% of artists.\n", 100.0 * hits_c / nc, 100.0 * hits_a / na)?;
|
table(md, &s_c, &s_a, &mut rows, h, "add_")?;
|
||||||
writeln!(md, "| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |\n|---|---|---|---|---|")?;
|
if s_c.wk.len() < 8 { writeln!(md, "Fewer than 8 forecast origins: no verdict at this horizon (doc rule).\n")?; }
|
||||||
for k in 0..4 {
|
|
||||||
let (sc, sa) = (wk_c.iter().map(|w| w[k]).sum::<f64>() / nc, wk_a.iter().map(|w| w[k]).sum::<f64>() / na);
|
|
||||||
let (ac, aa) = (average_precision(pairs_c[k].clone()), average_precision(pairs_a[k].clone()));
|
|
||||||
for (metric, v) in [("add_logloss", sc), ("add_prauc", ac), ("addany_logloss", sa), ("addany_prauc", aa)] {
|
|
||||||
rows.push((names[k].to_string(), h, metric.to_string(), v));
|
|
||||||
}
|
|
||||||
writeln!(md, "| {} | {sc:.4} | {ac:.3} | {sa:.4} | {aa:.3} |", names[k])?;
|
|
||||||
}
|
|
||||||
let col = |v: &Vec<[f64; 4]>, k: usize| v.iter().map(|w| w[k]).collect::<Vec<f64>>();
|
|
||||||
let (s1, s2) = (skill(&col(&wk_c, 2), &col(&wk_c, 0)), skill(&col(&wk_c, 2), &col(&wk_c, 1)));
|
|
||||||
let (s3, s4) = (skill(&col(&wk_a, 2), &col(&wk_a, 0)), skill(&col(&wk_a, 2), &col(&wk_a, 1)));
|
|
||||||
rows.push(("logit".into(), h, "add_skill_vs_level".into(), s2.0));
|
|
||||||
rows.push(("logit".into(), h, "addany_skill_vs_level".into(), s4.0));
|
|
||||||
writeln!(md, "\nSkill of the logit (90% block-bootstrap interval over origins):\n")?;
|
|
||||||
writeln!(md, "| | vs in-rotation-now | vs last-12-weeks level |\n|---|---|---|")?;
|
|
||||||
writeln!(md, "| per channel | {:.3} ({:.3} to {:.3}) | {:.3} ({:.3} to {:.3}) |", s1.0, s1.1, s1.2, s2.0, s2.1, s2.2)?;
|
|
||||||
writeln!(md, "| any channel | {:.3} ({:.3} to {:.3}) | {:.3} ({:.3} to {:.3}) |\n", s3.0, s3.1, s3.2, s4.0, s4.1, s4.2)?;
|
|
||||||
writeln!(md, "Calibration, per channel:\n\n| Predicted | Rows | Mean predicted | Observed |\n|---|---|---|---|")?;
|
|
||||||
for (d, c) in calib.iter().enumerate() {
|
|
||||||
if c[2] > 0.0 { writeln!(md, "| {:.1}–{:.1} | {} | {:.3} | {:.3} |", d as f64 / 10.0, (d + 1) as f64 / 10.0, c[2], c[0] / c[2], c[1] / c[2])?; }
|
|
||||||
}
|
|
||||||
writeln!(md)?;
|
|
||||||
if nwk < 8 { writeln!(md, "Fewer than 8 forecast origins: no verdict at this horizon (doc rule).\n")?; }
|
|
||||||
}
|
}
|
||||||
let m = AddModel::fit(p, &r, 0, p.w() - 1);
|
let m = AddModel::fit(p, &r, 0, p.w() - 1);
|
||||||
writeln!(md, "## Coefficients, window starting now (h = 0), all closed windows\n\n| Any-channel feature | Coefficient |\n|---|---|")?;
|
writeln!(md, "## Coefficients, window starting now (h = 0), all closed windows\n\n| Any-channel feature | Coefficient |\n|---|---|")?;
|
||||||
for (n, b) in ANY_NAMES.iter().zip(&m.any) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
for (n, b) in ANY_NAMES.iter().chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).zip(&m.any) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
||||||
for (n, b) in REL_NAMES.iter().chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).zip(&m.any[21..]) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
|
||||||
writeln!(md, "\n| Per-channel feature | Coefficient |\n|---|---|")?;
|
writeln!(md, "\n| Per-channel feature | Coefficient |\n|---|---|")?;
|
||||||
for (n, b) in features::NAMES.iter().chain(EXTRA).chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).chain(AFF_NAMES).zip(&m.channel) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
for (n, b) in features::NAMES.iter().chain(EXTRA).chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).chain(AFF_NAMES).zip(&m.channel) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
||||||
writeln!(md)?;
|
writeln!(md)?;
|
||||||
@@ -621,83 +769,76 @@ pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md:
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Walk-forward test of the release-event model: at each origin t0 the model sees only events
|
/// Walk-forward test of the release-event model: at each origin t0 the model sees only events
|
||||||
/// whose 8 weeks had closed, and scores the releases of week t0+1. The new song's NMF placement
|
/// whose window had closed (a week before t0), and scores the releases of week t0+1. Pre-release
|
||||||
/// isn't known in advance, so it's blended from the artist's history (as on the site).
|
/// rows blend the new song's NMF placement from the artist's history (as on the site).
|
||||||
pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
||||||
let r = Rot::new(p, tau);
|
let r = Rot::new(p, tau);
|
||||||
let mut rows = Vec::new();
|
let mut rows = Vec::new();
|
||||||
let names = ["rate by in-rotation-now", "rate by last-12-weeks level", "event model (pre-release: NMF from history)", "event model (post-release: NMF known)"];
|
let (mut s_c, mut s_a) = (Scores::new(SCORED.to_vec()), Scores::new(SCORED.to_vec()));
|
||||||
let mut wk_any: Vec<[f64; 4]> = Vec::new();
|
let (mut pool_c, mut pool_a) = (Pool(Vec::new()), Pool(Vec::new()));
|
||||||
let mut wk_ch: Vec<[f64; 4]> = Vec::new();
|
|
||||||
let mut pairs_any: [Vec<(f64, bool)>; 4] = Default::default();
|
|
||||||
let mut pairs_ch: [Vec<(f64, bool)>; 4] = Default::default();
|
|
||||||
let (mut n_ev, mut n_hit) = (0usize, 0usize);
|
|
||||||
let lvl = |m: f64| ((m.ln_1p() / 0.5) as usize).min(7);
|
let lvl = |m: f64| ((m.ln_1p() / 0.5) as usize).min(7);
|
||||||
let level = |a: usize, t: usize| (0..LAGS).map(|i| p.group_total(a, t - i, None)).sum::<f64>() / LAGS as f64;
|
let level = |a: usize, t: usize| (0..LAGS).map(|i| p.group_total(a, t - i, None)).sum::<f64>() / LAGS as f64;
|
||||||
for t0 in start..p.w().saturating_sub(W) {
|
for t0 in start..p.w().saturating_sub(W + 1) {
|
||||||
if t0 + 1 >= p.w() { break; }
|
|
||||||
let m = AddModel::fit_events(p, &r, t0);
|
let m = AddModel::fit_events(p, &r, t0);
|
||||||
// baselines from the training events
|
// Baselines from the training events. Release timing only: by weeks since the previous release.
|
||||||
let (mut by_rot, mut by_lvl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
let (mut by_rot, mut by_lvl, mut cr, mut cl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8], [[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
||||||
let (mut cr, mut cl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
let mut tm = Timing::new();
|
||||||
for t in origins(0, t0) {
|
for t in origins(0, t0) {
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
if !clean_release(p, a, t + 1) { continue; }
|
if !p.known(a, t) || !clean_release(p, a, t + 1) { continue; }
|
||||||
|
let key = since_bin(p, a, t);
|
||||||
let y = r.add_any_in_window(a, t, 0) as i32 as f64;
|
let y = r.add_any_in_window(a, t, 0) as i32 as f64;
|
||||||
let on = r.any(a, t) as usize;
|
let on = r.any(a, t) as usize;
|
||||||
by_rot[on][0] += y; by_rot[on][1] += 1.0;
|
by_rot[on][0] += y; by_rot[on][1] += 1.0;
|
||||||
let b = lvl(level(a, t)); by_lvl[b][0] += y; by_lvl[b][1] += 1.0;
|
let b = lvl(level(a, t)); by_lvl[b][0] += y; by_lvl[b][1] += 1.0;
|
||||||
|
tm.any[key][0] += y; tm.any[key][1] += 1.0;
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
let yc = r.add_in_window(a, c, t, 0) as i32 as f64;
|
let yc = r.add_in_window(a, c, t, 0) as i32 as f64;
|
||||||
let onc = r.on(a, c, t) as usize;
|
let onc = r.on(a, c, t) as usize;
|
||||||
cr[onc][0] += yc; cr[onc][1] += 1.0;
|
cr[onc][0] += yc; cr[onc][1] += 1.0;
|
||||||
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t - i)).sum::<f64>() / LAGS as f64);
|
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t - i)).sum::<f64>() / LAGS as f64);
|
||||||
cl[bc][0] += yc; cl[bc][1] += 1.0;
|
cl[bc][0] += yc; cl[bc][1] += 1.0;
|
||||||
|
tm.ch[c * 12 + key][0] += yc; tm.ch[c * 12 + key][1] += 1.0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
let (mut la, mut lc) = ([0.0; 4], [0.0; 4]);
|
let gap = W + 1;
|
||||||
let mut any_events = false;
|
let ((pl_c, iso_c), (pl_a, iso_a)) = (pool_c.fit(t0, gap), pool_a.fit(t0, gap));
|
||||||
|
let mut started = false;
|
||||||
for a in 0..p.a() {
|
for a in 0..p.a() {
|
||||||
if !clean_release(p, a, t0 + 1) { continue; }
|
if !p.known(a, t0) || !clean_release(p, a, t0 + 1) { continue; }
|
||||||
any_events = true;
|
if !started { s_c.start_origin(); s_a.start_origin(); started = true; }
|
||||||
let q = p_nmf(p, a, t0);
|
let q = p_nmf(p, a, t0);
|
||||||
let (pcn, pan) = m.predict(p, &r, a, t0, Scenario::new(true, true), None);
|
let (pcn, pan) = m.predict(p, &r, a, t0, Scenario::new(true, true), None);
|
||||||
let (pcr, par) = m.predict(p, &r, a, t0, Scenario::new(true, false), None);
|
let (pcr, par) = m.predict(p, &r, a, t0, Scenario::new(true, false), None);
|
||||||
let pa = q * pan + (1.0 - q) * par;
|
let (rcn, ran) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, true));
|
||||||
|
let (rcr, rar) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, false));
|
||||||
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::ACTUAL, None);
|
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::ACTUAL, None);
|
||||||
|
let key = since_bin(p, a, t0);
|
||||||
let y = r.add_any_in_window(a, t0, 0);
|
let y = r.add_any_in_window(a, t0, 0);
|
||||||
let on = r.any(a, t0) as usize;
|
let on = r.any(a, t0) as usize;
|
||||||
let preds = [by_rot[on][0] / by_rot[on][1], by_lvl[lvl(level(a, t0))][0] / by_lvl[lvl(level(a, t0))][1], pa, pax];
|
let raw = q * ran + (1.0 - q) * rar;
|
||||||
for k in 0..4 { la[k] += ll(preds[k], y); pairs_any[k].push((preds[k], y)); }
|
let tma = tm.any[key];
|
||||||
n_ev += 1; n_hit += y as usize;
|
s_a.add(&[by_rot[on][0] / by_rot[on][1], by_lvl[lvl(level(a, t0))][0] / by_lvl[lvl(level(a, t0))][1], tma[0] / tma[1],
|
||||||
|
q * pan + (1.0 - q) * par, raw, pl_a.apply(raw), iso_a.apply(raw), pax], y);
|
||||||
|
pool_a.0.push((t0, raw, y));
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
let pc = q * pcn[c] + (1.0 - q) * pcr[c];
|
|
||||||
let yc = r.add_in_window(a, c, t0, 0);
|
let yc = r.add_in_window(a, c, t0, 0);
|
||||||
let onc = r.on(a, c, t0) as usize;
|
let onc = r.on(a, c, t0) as usize;
|
||||||
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t0 - i)).sum::<f64>() / LAGS as f64);
|
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t0 - i)).sum::<f64>() / LAGS as f64);
|
||||||
let pr = [cr[onc][0] / cr[onc][1], cl[bc][0] / cl[bc][1], pc, pcx[c]];
|
let rawc = q * rcn[c] + (1.0 - q) * rcr[c];
|
||||||
for k in 0..4 { lc[k] += ll(pr[k], yc); pairs_ch[k].push((pr[k], yc)); }
|
let tmc = tm.ch[c * 12 + key];
|
||||||
|
s_c.add(&[cr[onc][0] / cr[onc][1], cl[bc][0] / cl[bc][1], tmc[0] / tmc[1],
|
||||||
|
q * pcn[c] + (1.0 - q) * pcr[c], rawc, pl_c.apply(rawc), iso_c.apply(rawc), pcx[c]], yc);
|
||||||
|
pool_c.0.push((t0, rawc, yc));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if any_events { wk_any.push(la); wk_ch.push(lc); }
|
|
||||||
eprintln!("event backtest: origin {} done", p.weeks[t0]);
|
eprintln!("event backtest: origin {} done", p.weeks[t0]);
|
||||||
}
|
}
|
||||||
writeln!(md, "# Backtest: if the artist releases a new song, will DR add it?\n")?;
|
writeln!(md, "# Backtest: if the artist releases a new song, will DR add it?\n")?;
|
||||||
writeln!(md, "Events: an artist in the panel releases something (Deezer) in week t0+1, with no other release in the {CLEAN_WEEKS} weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after {GAP}+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the {W} weeks from the release week. The model knows the artist's state the week before; the new song's NMF Denmark placement is blended from the artist's history. {n_ev} release events scored, {n_hit} ({:.1}%) got an add on some DR channel.\n",
|
writeln!(md, "Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the {CLEAN_WEEKS} weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after {GAP}+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the {W} weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. \"Release timing only\" = the add rate by weeks since the artist's previous release. {} release events scored, {} ({:.1}%) got an add on some DR channel.\n",
|
||||||
100.0 * n_hit as f64 / n_ev.max(1) as f64)?;
|
s_a.n, s_a.hits, 100.0 * s_a.hits / s_a.n.max(1.0))?;
|
||||||
writeln!(md, "| Model | Log-loss, any channel | PR-AUC, any channel | Log-loss, per channel | PR-AUC, per channel |\n|---|---|---|---|---|")?;
|
table(md, &s_c, &s_a, &mut rows, 0, "event_")?;
|
||||||
let nch = (n_ev * C).max(1) as f64;
|
|
||||||
for k in 0..4 {
|
|
||||||
let (a, c) = (wk_any.iter().map(|w| w[k]).sum::<f64>() / n_ev.max(1) as f64, wk_ch.iter().map(|w| w[k]).sum::<f64>() / nch);
|
|
||||||
let (pa, pc) = (average_precision(pairs_any[k].clone()), average_precision(pairs_ch[k].clone()));
|
|
||||||
writeln!(md, "| {} | {a:.4} | {pa:.3} | {c:.4} | {pc:.3} |", names[k])?;
|
|
||||||
for (metric, v) in [("event_logloss_any", a), ("event_prauc_any", pa), ("event_logloss_ch", c), ("event_prauc_ch", pc)] { rows.push((names[k].to_string(), 0, metric.to_string(), v)); }
|
|
||||||
}
|
|
||||||
let col = |v: &Vec<[f64; 4]>, k: usize| v.iter().map(|w| w[k]).collect::<Vec<f64>>();
|
|
||||||
let (s1, s2) = (skill(&col(&wk_any, 2), &col(&wk_any, 0)), skill(&col(&wk_any, 2), &col(&wk_any, 1)));
|
|
||||||
rows.push(("event model".into(), 0, "event_skill_any_vs_level".into(), s2.0));
|
|
||||||
writeln!(md, "\nSkill of the event model, any channel: {:.3} ({:.3} to {:.3}) vs in-rotation-now, {:.3} ({:.3} to {:.3}) vs last-12-weeks level.\n", s1.0, s1.1, s1.2, s2.0, s2.1, s2.2)?;
|
|
||||||
let m = AddModel::fit_events(p, &r, p.w() - 1);
|
let m = AddModel::fit_events(p, &r, p.w() - 1);
|
||||||
writeln!(md, "| Any-channel feature (all events) | Coefficient |\n|---|---|")?;
|
writeln!(md, "| Any-channel feature (all events) | Coefficient |\n|---|---|")?;
|
||||||
for (n, b) in ANY_NAMES.iter().chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).zip(&m.any) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
for (n, b) in ANY_NAMES.iter().chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).zip(&m.any) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
||||||
|
|||||||
@@ -76,7 +76,8 @@ fn average_precision(mut v: Vec<(f64, bool)>) -> f64 {
|
|||||||
fn skill(model: &[f64], base: &[f64]) -> (f64, f64, f64) {
|
fn skill(model: &[f64], base: &[f64]) -> (f64, f64, f64) {
|
||||||
let point = 1.0 - model.iter().sum::<f64>() / base.iter().sum::<f64>();
|
let point = 1.0 - model.iter().sum::<f64>() / base.iter().sum::<f64>();
|
||||||
let n = model.len();
|
let n = model.len();
|
||||||
let block = 4.min(n);
|
// Blocks of 8 weeks: neighbouring origins share most of an 8-week target window.
|
||||||
|
let block = 8.min(n);
|
||||||
let mut rng = Rng(42);
|
let mut rng = Rng(42);
|
||||||
let mut draws: Vec<f64> = (0..2000).map(|_| {
|
let mut draws: Vec<f64> = (0..2000).map(|_| {
|
||||||
let (mut lm, mut lb, mut k) = (0.0, 0.0, 0);
|
let (mut lm, mut lb, mut k) = (0.0, 0.0, 0);
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
//! Recalibration of predicted probabilities, fitted on pooled out-of-sample forecasts:
|
||||||
|
//! Platt (a logistic map on the logit) or isotonic regression (pool-adjacent-violators, a
|
||||||
|
//! monotone step function, interpolated linearly between block centres).
|
||||||
|
|
||||||
|
use super::glm::{self, Design};
|
||||||
|
|
||||||
|
#[derive(Clone, Debug)]
|
||||||
|
pub enum Calib {
|
||||||
|
Identity,
|
||||||
|
Platt(f64, f64),
|
||||||
|
/// Knots (predicted, observed), increasing in both.
|
||||||
|
Iso(Vec<(f64, f64)>),
|
||||||
|
}
|
||||||
|
|
||||||
|
fn logit(p: f64) -> f64 { let p = p.clamp(1e-6, 1.0 - 1e-6); (p / (1.0 - p)).ln() }
|
||||||
|
|
||||||
|
impl Calib {
|
||||||
|
pub fn apply(&self, p: f64) -> f64 {
|
||||||
|
let out = match self {
|
||||||
|
Calib::Identity => p,
|
||||||
|
Calib::Platt(a, b) => glm::sigmoid(a + b * logit(p)),
|
||||||
|
Calib::Iso(k) => {
|
||||||
|
if k.is_empty() { p }
|
||||||
|
else if p <= k[0].0 { k[0].1 }
|
||||||
|
else if p >= k[k.len() - 1].0 { k[k.len() - 1].1 }
|
||||||
|
else {
|
||||||
|
let i = k.partition_point(|x| x.0 <= p);
|
||||||
|
let ((x0, y0), (x1, y1)) = (k[i - 1], k[i]);
|
||||||
|
if x1 > x0 { y0 + (y1 - y0) * (p - x0) / (x1 - x0) } else { y1 }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
out.clamp(1e-5, 1.0 - 1e-5)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Platt on pooled (prediction, outcome) pairs; identity if there are too few positives.
|
||||||
|
pub fn platt(pairs: &[(f64, bool)]) -> Calib {
|
||||||
|
if pairs.iter().filter(|x| x.1).count() < 10 { return Calib::Identity; }
|
||||||
|
let mut d = Design::new(2);
|
||||||
|
for (p, y) in pairs { d.push(&[1.0, logit(*p)], *y as i32 as f64, 1.0); }
|
||||||
|
let b = glm::logit(&d, 0.0);
|
||||||
|
if b[1] > 0.0 { Calib::Platt(b[0], b[1]) } else { Calib::Identity }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Isotonic regression (pool-adjacent-violators) on pooled pairs. A small prior (one pseudo
|
||||||
|
/// observation at the block's mean prediction) keeps thin blocks off 0 and 1.
|
||||||
|
pub fn isotonic(pairs: &[(f64, bool)]) -> Calib {
|
||||||
|
if pairs.iter().filter(|x| x.1).count() < 10 { return Calib::Identity; }
|
||||||
|
let mut v: Vec<(f64, f64)> = pairs.iter().map(|(p, y)| (*p, *y as i32 as f64)).collect();
|
||||||
|
v.sort_by(|a, b| a.0.total_cmp(&b.0));
|
||||||
|
// blocks: (sum p, sum y, n)
|
||||||
|
let mut blocks: Vec<(f64, f64, f64)> = Vec::new();
|
||||||
|
for (p, y) in v {
|
||||||
|
blocks.push((p, y, 1.0));
|
||||||
|
while blocks.len() >= 2 {
|
||||||
|
let n = blocks.len();
|
||||||
|
let (a, b) = (blocks[n - 2], blocks[n - 1]);
|
||||||
|
if a.1 / a.2 >= b.1 / b.2 { blocks.pop(); blocks.pop(); blocks.push((a.0 + b.0, a.1 + b.1, a.2 + b.2)); } else { break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let knots: Vec<(f64, f64)> = blocks.iter().map(|(sp, sy, n)| { let mp = sp / n; (mp, (sy + mp) / (n + 1.0)) }).collect();
|
||||||
|
Calib::Iso(knots)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
#[test]
|
||||||
|
fn isotonic_is_monotone_and_fixes_overconfidence() {
|
||||||
|
// predictions twice too high
|
||||||
|
let pairs: Vec<(f64, bool)> = (0..20000).map(|i| { let p = (i % 100) as f64 / 100.0; (p, ((i * 7919) % 1000) as f64 / 1000.0 < p / 2.0) }).collect();
|
||||||
|
let c = Calib::isotonic(&pairs);
|
||||||
|
let (a, b) = (c.apply(0.2), c.apply(0.8));
|
||||||
|
assert!(a < b && (a - 0.1).abs() < 0.05 && (b - 0.4).abs() < 0.06, "{a} {b}");
|
||||||
|
let pl = Calib::platt(&pairs);
|
||||||
|
assert!(pl.apply(0.8) < 0.6);
|
||||||
|
}
|
||||||
|
}
|
||||||
+7
-1
@@ -14,6 +14,7 @@
|
|||||||
pub mod addcheck;
|
pub mod addcheck;
|
||||||
pub mod adds;
|
pub mod adds;
|
||||||
pub mod backtest;
|
pub mod backtest;
|
||||||
|
pub mod calib;
|
||||||
pub mod calibrate;
|
pub mod calibrate;
|
||||||
pub mod common;
|
pub mod common;
|
||||||
pub mod features;
|
pub mod features;
|
||||||
@@ -49,6 +50,9 @@ pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
|||||||
let charted = (0..p.a()).filter(|&a| p.chart(a, last).t40_hits > 0 || p.chart(a, last).air_hits > 0).count();
|
let charted = (0..p.a()).filter(|&a| p.chart(a, last).t40_hits > 0 || p.chart(a, last).air_hits > 0).count();
|
||||||
let danish = p.artists.iter().filter(|a| a.danish).count();
|
let danish = p.artists.iter().filter(|a| a.danish).count();
|
||||||
println!("artists matched to IFPI chart history: {charted}; Danish: {danish}");
|
println!("artists matched to IFPI chart history: {charted}; Danish: {danish}");
|
||||||
|
let known = |t: usize| (0..p.a()).filter(|&a| p.known(a, t)).count();
|
||||||
|
println!("artists knowable (first release, DR play, chart, list, archive or NMF) by week 0: {}, week 26: {}, week {}: {} (of {})",
|
||||||
|
known(0), known(26), p.w() - 1, known(p.w() - 1), p.a());
|
||||||
}
|
}
|
||||||
Some("backtest") => {
|
Some("backtest") => {
|
||||||
let horizons: Vec<usize> = arg(args, "--h").unwrap_or_else(|| "1,4,8".into()).split(',').filter_map(|s| s.trim().parse().ok()).collect();
|
let horizons: Vec<usize> = arg(args, "--h").unwrap_or_else(|| "1,4,8".into()).split(',').filter_map(|s| s.trim().parse().ok()).collect();
|
||||||
@@ -88,7 +92,9 @@ pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
|||||||
Some("leakcheck") => { print!("{}", leakcheck::run(&p)?); }
|
Some("leakcheck") => { print!("{}", leakcheck::run(&p)?); }
|
||||||
Some("addcheck") => {
|
Some("addcheck") => {
|
||||||
if let Some(r) = arg(args, "--rule") { let _ = adds::ADD_RULE.set(r); }
|
if let Some(r) = arg(args, "--rule") { let _ = adds::ADD_RULE.set(r); }
|
||||||
print!("{}", addcheck::run(db, &p)?);
|
let md = addcheck::run(db, &p, args.iter().any(|a| a == "--tune"))?;
|
||||||
|
if let Some(out) = arg(args, "--out") { std::fs::write(out, &md)?; }
|
||||||
|
print!("{md}");
|
||||||
}
|
}
|
||||||
Some("events") => {
|
Some("events") => {
|
||||||
if let Some(r) = arg(args, "--rule") { let _ = adds::ADD_RULE.set(r); }
|
if let Some(r) = arg(args, "--rule") { let _ = adds::ADD_RULE.set(r); }
|
||||||
|
|||||||
+40
-1
@@ -89,6 +89,9 @@ pub struct Panel {
|
|||||||
/// How much DR plays the artist's genre on each station group, relative to all music: ln of
|
/// How much DR plays the artist's genre on each station group, relative to all music: ln of
|
||||||
/// (genre's share of plays on the group / the group's share of all plays); 0 = genre unknown.
|
/// (genre's share of plays on the group / the group's share of all plays); 0 = genre unknown.
|
||||||
pub genre_fit: Vec<[f64; GROUPS]>,
|
pub genre_fit: Vec<[f64; GROUPS]>,
|
||||||
|
/// The first week (index) at which the artist was knowable: their first release, DR play,
|
||||||
|
/// chart entry, DR list, archived play or NMF placement (0 = before the panel; w() = never).
|
||||||
|
pub first_known: Vec<usize>,
|
||||||
/// Season index per ISO week (1..=53): new songs reaching the Airplay Top-20 that week of the
|
/// Season index per ISO week (1..=53): new songs reaching the Airplay Top-20 that week of the
|
||||||
/// year, relative to an average week, 2002–2024 (before the DR panel), smoothed over 5 weeks.
|
/// year, relative to an average week, 2002–2024 (before the DR panel), smoothed over 5 weeks.
|
||||||
pub season: [f64; 54],
|
pub season: [f64; 54],
|
||||||
@@ -102,6 +105,10 @@ impl Panel {
|
|||||||
#[inline]
|
#[inline]
|
||||||
pub fn chart(&self, a: usize, w: usize) -> Chart { self.chart[a * self.weeks.len() + w] }
|
pub fn chart(&self, a: usize, w: usize) -> Chart { self.chart[a * self.weeks.len() + w] }
|
||||||
|
|
||||||
|
/// Was the artist knowable by the end of week t (so a forecast made then could include them)?
|
||||||
|
#[inline]
|
||||||
|
pub fn known(&self, a: usize, t: usize) -> bool { self.first_known[a] <= t }
|
||||||
|
|
||||||
/// The panel as it was known at the end of week t: plays after t removed (and taken out of the
|
/// The panel as it was known at the end of week t: plays after t removed (and taken out of the
|
||||||
/// all-time counts), releases, NMF placements and lists after t dropped. For the leak test.
|
/// all-time counts), releases, NMF placements and lists after t dropped. For the leak test.
|
||||||
pub fn truncated(&self, t: usize) -> Panel {
|
pub fn truncated(&self, t: usize) -> Panel {
|
||||||
@@ -180,7 +187,9 @@ impl Panel {
|
|||||||
let nmf_dk = load_nmf_dk(db, &id_of, &artists)?;
|
let nmf_dk = load_nmf_dk(db, &id_of, &artists)?;
|
||||||
let season = load_season(db)?;
|
let season = load_season(db)?;
|
||||||
let (alltime, genre_fit) = load_affinity(db, &id_of, artists.len())?;
|
let (alltime, genre_fit) = load_affinity(db, &id_of, artists.len())?;
|
||||||
Ok(Panel { weeks, artists, y, chart, list_tau, releases, archive_rate, listed_on, nmf_dk, season, alltime, genre_fit })
|
let mut p = Panel { weeks, artists, y, chart, list_tau, releases, archive_rate, listed_on, nmf_dk, season, alltime, genre_fit, first_known: Vec::new() };
|
||||||
|
p.first_known = first_known(db, &p, &id_of)?;
|
||||||
|
Ok(p)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -427,3 +436,33 @@ fn load_affinity(db: &Db, id_of: &HashMap<String, usize>, n: usize) -> Result<(V
|
|||||||
}
|
}
|
||||||
Ok((alltime, fit))
|
Ok((alltime, fit))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// First knowable week per artist (see Panel::first_known).
|
||||||
|
fn first_known(db: &Db, p: &Panel, id_of: &HashMap<String, usize>) -> Result<Vec<usize>> {
|
||||||
|
let nw = p.w();
|
||||||
|
let week_of_date = |d: NaiveDate| -> usize {
|
||||||
|
if d < p.weeks[0] { 0 } else { (((d - p.weeks[0]).num_days() / 7) as usize).min(nw) }
|
||||||
|
};
|
||||||
|
let mut first = vec![nw; p.a()];
|
||||||
|
let mut old: HashMap<usize, NaiveDate> = HashMap::new();
|
||||||
|
let exists: i64 = db.conn.query_row("SELECT count(*) FROM sqlite_master WHERE name = 'dr_artist_history'", [], |r| r.get(0))?;
|
||||||
|
if exists > 0 {
|
||||||
|
for r in db.conn.prepare("SELECT artist_id, min(week_start) FROM dr_artist_history WHERE plays > 0 GROUP BY artist_id")?
|
||||||
|
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?)))? {
|
||||||
|
let (id, d) = r?;
|
||||||
|
if let (Some(&a), Ok(d)) = (id_of.get(&id), NaiveDate::parse_from_str(&d, "%Y-%m-%d")) { old.insert(a, d); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for a in 0..p.a() {
|
||||||
|
let mut f = nw;
|
||||||
|
if let Some(d) = p.releases[a].as_ref().and_then(|v| v.first()) { f = f.min(week_of_date(*d)); }
|
||||||
|
if let Some(w) = (0..nw).find(|&w| (0..C).any(|c| p.y(a, c, w) > 0.0)) { f = f.min(w); }
|
||||||
|
if let Some(w) = (0..nw).find(|&w| { let c = p.chart(a, w); c.t40_hits > 0 || c.air_hits > 0 }) { f = f.min(w); }
|
||||||
|
if let Some(d) = p.listed_on[a].first() { f = f.min(week_of_date(*d)); }
|
||||||
|
if let Some((d, _)) = p.nmf_dk[a].first() { f = f.min(week_of_date(*d)); }
|
||||||
|
if p.archive_rate[a] > 0.0 { f = 0; }
|
||||||
|
if let Some(d) = old.get(&a) { f = f.min(week_of_date(*d)); }
|
||||||
|
first[a] = f;
|
||||||
|
}
|
||||||
|
Ok(first)
|
||||||
|
}
|
||||||
|
|||||||
+1
-1
@@ -116,7 +116,7 @@ pub fn run(db: &Db, p: &Panel, max_h: usize, show_h: usize) -> Result<Vec<Line>>
|
|||||||
let rot = Rot::new(p, &tau);
|
let rot = Rot::new(p, &tau);
|
||||||
// The release-event model ("if they release a new song, will DR add it?"), the same for every
|
// The release-event model ("if they release a new song, will DR add it?"), the same for every
|
||||||
// window start: the artist's state now, the new song's NMF placement blended from history.
|
// window start: the artist's state now, the new song's NMF placement blended from history.
|
||||||
let ev = AddModel::fit_events(p, &rot, p.w() - 1);
|
let ev = AddModel::fit_events_site(p, &rot, p.w() - 1);
|
||||||
let t0 = p.w() - 1;
|
let t0 = p.w() - 1;
|
||||||
// A new song released in week t0+h+1: time-since features moved forward to then; none if too
|
// A new song released in week t0+h+1: time-since features moved forward to then; none if too
|
||||||
// soon after the artist's last release.
|
// soon after the artist's last release.
|
||||||
|
|||||||
+1
-1
@@ -129,7 +129,7 @@ fn fit_models(db_path: &str, made_at: String) -> Result<Models> {
|
|||||||
let direct = (1..=MAX_H).map(|h| Direct::fit(&panel, h, panel.w() - 1, &tau, false)).collect();
|
let direct = (1..=MAX_H).map(|h| Direct::fit(&panel, h, panel.w() - 1, &tau, false)).collect();
|
||||||
let rot = Rot::new(&panel, &tau);
|
let rot = Rot::new(&panel, &tau);
|
||||||
let adds = (0..=MAX_H).map(|h| AddModel::fit(&panel, &rot, h, panel.w() - 1)).collect();
|
let adds = (0..=MAX_H).map(|h| AddModel::fit(&panel, &rot, h, panel.w() - 1)).collect();
|
||||||
let events = AddModel::fit_events(&panel, &rot, panel.w() - 1);
|
let events = AddModel::fit_events_site(&panel, &rot, panel.w() - 1);
|
||||||
let events_w = WINDOWS.iter().map(|&w| (w, AddModel::fit_events_w(&panel, &rot, panel.w() - 1, w))).collect();
|
let events_w = WINDOWS.iter().map(|&w| (w, AddModel::fit_events_w(&panel, &rot, panel.w() - 1, w))).collect();
|
||||||
let index = panel.artists.iter().enumerate().map(|(i, a)| (a.id.clone(), i)).collect();
|
let index = panel.artists.iter().enumerate().map(|(i, a)| (a.id.clone(), i)).collect();
|
||||||
Ok(Models { made_at, panel, tau, direct, adds, events, events_w, rot, index })
|
Ok(Models { made_at, panel, tau, direct, adds, events, events_w, rot, index })
|
||||||
|
|||||||
Reference in New Issue
Block a user