Default window 2 weeks (1-week option added) on the site, best week and what-if; new-song backtest takes the window (--w, default 2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+68
-71
@@ -1,97 +1,94 @@
|
||||
# Backtest: if the artist releases a new song, will DR add it?
|
||||
|
||||
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the 8 weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 2140 release events scored, 120 (5.6%) got an add on some DR channel.
|
||||
Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the 6 weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after 4+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within 2 week(s) from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. "Release timing only" = the add rate by weeks since the artist's previous release. 3193 release events scored, 63 (2.0%) got an add on some DR channel.
|
||||
|
||||
| Model | Log-loss, per channel | PR-AUC, per channel | Log-loss, any channel | PR-AUC, any channel |
|
||||
|---|---|---|---|---|
|
||||
| rate by in-rotation-now | 0.0666 | 0.061 | 0.2070 | 0.113 |
|
||||
| rate by last-12-weeks level | 0.0478 | 0.216 | 0.1805 | 0.221 |
|
||||
| release timing only | 0.0719 | 0.014 | 0.2166 | 0.068 |
|
||||
| model, calibrated on the last 6 origins (pre-release) | 0.0431 | 0.341 | 0.1881 | 0.243 |
|
||||
| model, raw (pre-release) | 0.0437 | 0.326 | 0.1860 | 0.252 |
|
||||
| model + pooled Platt (pre-release) | 0.0425 | 0.337 | 0.1916 | 0.221 |
|
||||
| model + pooled isotonic (pre-release) | 0.0439 | 0.308 | 0.1871 | 0.243 |
|
||||
| model, NMF known (post-release) | 0.0413 | 0.346 | 0.1853 | 0.266 |
|
||||
| rate by in-rotation-now | 0.0245 | 0.015 | 0.0941 | 0.062 |
|
||||
| rate by last-12-weeks level | 0.0174 | 0.082 | 0.0738 | 0.140 |
|
||||
| release timing only | 0.0267 | 0.003 | 0.0982 | 0.021 |
|
||||
| model, calibrated on the last 6 origins (pre-release) | 0.0183 | 0.080 | 0.0748 | 0.170 |
|
||||
| model, raw (pre-release) | 0.0182 | 0.091 | 0.0696 | 0.233 |
|
||||
| model + pooled Platt (pre-release) | 0.0177 | 0.094 | 0.0704 | 0.224 |
|
||||
| model + pooled isotonic (pre-release) | 0.0176 | 0.091 | 0.0706 | 0.235 |
|
||||
| model, NMF known (post-release) | 0.0184 | 0.085 | 0.0750 | 0.163 |
|
||||
|
||||
Skill (1 − log-loss ÷ baseline log-loss; 90% interval from a bootstrap over blocks of 8 forecast weeks):
|
||||
|
||||
| Model | Per channel vs last-12-weeks | Per channel vs release timing | Any channel vs last-12-weeks | Any channel vs release timing |
|
||||
|---|---|---|---|---|
|
||||
| model, calibrated on the last 6 origins (pre-release) | +0.098 (+0.054 to +0.162) | +0.400 (+0.313 to +0.440) | -0.042 (-0.068 to +0.041) | +0.132 (+0.128 to +0.223) |
|
||||
| model + pooled Platt (pre-release) | +0.111 (+0.054 to +0.181) | +0.408 (+0.330 to +0.440) | -0.061 (-0.091 to -0.022) | +0.116 (+0.097 to +0.160) |
|
||||
| model + pooled isotonic (pre-release) | +0.083 (+0.054 to +0.187) | +0.390 (+0.333 to +0.438) | -0.037 (-0.072 to +0.031) | +0.136 (+0.121 to +0.206) |
|
||||
| model, calibrated on the last 6 origins (pre-release) | -0.053 (-0.131 to +0.002) | +0.313 (+0.257 to +0.378) | -0.013 (-0.017 to +0.048) | +0.239 (+0.226 to +0.288) |
|
||||
| model + pooled Platt (pre-release) | -0.015 (-0.053 to +0.020) | +0.338 (+0.286 to +0.399) | +0.046 (-0.008 to +0.107) | +0.283 (+0.234 to +0.334) |
|
||||
| model + pooled isotonic (pre-release) | -0.010 (-0.095 to +0.039) | +0.342 (+0.234 to +0.430) | +0.044 (-0.011 to +0.100) | +0.281 (+0.223 to +0.335) |
|
||||
|
||||
Calibration, any channel:
|
||||
|
||||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||
|---|---|---|---|
|
||||
| 0.0–0.1 | 1936 / 0.018 / 0.034 | 1897 / 0.034 / 0.033 | 1844 / 0.027 / 0.027 |
|
||||
| 0.1–0.2 | 124 / 0.137 / 0.234 | 154 / 0.141 / 0.201 | 124 / 0.143 / 0.202 |
|
||||
| 0.2–0.3 | 40 / 0.245 / 0.325 | 56 / 0.247 / 0.286 | 90 / 0.251 / 0.200 |
|
||||
| 0.3–0.4 | 18 / 0.349 / 0.278 | 20 / 0.355 / 0.300 | 40 / 0.350 / 0.250 |
|
||||
| 0.4–0.5 | 13 / 0.442 / 0.385 | 7 / 0.463 / 0.429 | 23 / 0.449 / 0.435 |
|
||||
| 0.5–0.6 | 5 / 0.534 / 0.400 | 3 / 0.536 / 0.333 | 8 / 0.571 / 0.500 |
|
||||
| 0.6–0.7 | 1 / 0.606 / 0.000 | 2 / 0.625 / 0.000 | 3 / 0.637 / 0.333 |
|
||||
| 0.7–0.8 | 3 / 0.732 / 0.333 | — | 2 / 0.730 / 0.000 |
|
||||
| 0.8–0.9 | — | 1 / 0.823 / 1.000 | 5 / 0.824 / 0.600 |
|
||||
| 0.9–1.0 | — | — | 1 / 0.911 / 0.000 |
|
||||
| 0.0–0.1 | 3081 / 0.015 / 0.013 | 3033 / 0.011 / 0.010 | 2979 / 0.009 / 0.008 |
|
||||
| 0.1–0.2 | 82 / 0.136 / 0.207 | 107 / 0.140 / 0.168 | 169 / 0.135 / 0.160 |
|
||||
| 0.2–0.3 | 16 / 0.245 / 0.312 | 20 / 0.239 / 0.300 | 16 / 0.238 / 0.188 |
|
||||
| 0.3–0.4 | 6 / 0.316 / 0.333 | 12 / 0.340 / 0.250 | 8 / 0.344 / 0.375 |
|
||||
| 0.4–0.5 | 4 / 0.451 / 0.000 | 13 / 0.437 / 0.231 | 8 / 0.436 / 0.250 |
|
||||
| 0.5–0.6 | 2 / 0.533 / 0.000 | 6 / 0.566 / 0.667 | 4 / 0.563 / 0.250 |
|
||||
| 0.6–0.7 | 2 / 0.636 / 0.000 | 1 / 0.605 / 0.000 | 2 / 0.620 / 0.000 |
|
||||
| 0.7–0.8 | — | — | 5 / 0.775 / 0.600 |
|
||||
| 0.8–0.9 | — | 1 / 0.818 / 0.000 | 2 / 0.827 / 0.500 |
|
||||
|
||||
Calibration, per channel:
|
||||
|
||||
| Predicted | model, calibrated on the last 6 origins (pre-release): rows / mean predicted / observed | model + pooled Platt (pre-release): rows / mean predicted / observed | model + pooled isotonic (pre-release): rows / mean predicted / observed |
|
||||
|---|---|---|---|
|
||||
| 0.0–0.1 | 29258 / 0.004 / 0.007 | 29241 / 0.004 / 0.007 | 29307 / 0.004 / 0.007 |
|
||||
| 0.1–0.2 | 342 / 0.140 / 0.208 | 347 / 0.143 / 0.182 | 328 / 0.143 / 0.235 |
|
||||
| 0.2–0.3 | 136 / 0.246 / 0.206 | 136 / 0.252 / 0.287 | 122 / 0.248 / 0.189 |
|
||||
| 0.3–0.4 | 87 / 0.347 / 0.207 | 102 / 0.335 / 0.118 | 81 / 0.344 / 0.185 |
|
||||
| 0.4–0.5 | 66 / 0.447 / 0.500 | 44 / 0.450 / 0.500 | 35 / 0.449 / 0.429 |
|
||||
| 0.5–0.6 | 10 / 0.527 / 0.000 | 28 / 0.538 / 0.714 | 25 / 0.551 / 0.760 |
|
||||
| 0.6–0.7 | 8 / 0.675 / 0.125 | 5 / 0.655 / 0.400 | 4 / 0.652 / 0.250 |
|
||||
| 0.7–0.8 | 42 / 0.745 / 0.762 | 42 / 0.757 / 0.714 | 35 / 0.751 / 0.886 |
|
||||
| 0.8–0.9 | 2 / 0.845 / 1.000 | 6 / 0.817 / 0.500 | 2 / 0.845 / 1.000 |
|
||||
| 0.9–1.0 | 9 / 0.912 / 1.000 | 9 / 0.912 / 1.000 | 21 / 0.954 / 0.476 |
|
||||
| 0.0–0.1 | 44463 / 0.004 / 0.003 | 44305 / 0.002 / 0.003 | 44334 / 0.003 / 0.003 |
|
||||
| 0.1–0.2 | 125 / 0.136 / 0.128 | 229 / 0.137 / 0.105 | 278 / 0.123 / 0.090 |
|
||||
| 0.2–0.3 | 69 / 0.259 / 0.159 | 86 / 0.242 / 0.093 | 30 / 0.245 / 0.067 |
|
||||
| 0.3–0.4 | 21 / 0.337 / 0.048 | 42 / 0.346 / 0.071 | 35 / 0.358 / 0.057 |
|
||||
| 0.4–0.5 | 8 / 0.456 / 0.000 | 28 / 0.454 / 0.143 | 13 / 0.448 / 0.308 |
|
||||
| 0.5–0.6 | 11 / 0.527 / 0.000 | 6 / 0.540 / 0.500 | 6 / 0.546 / 0.500 |
|
||||
| 0.6–0.7 | 3 / 0.658 / 0.333 | 4 / 0.634 / 0.250 | 1 / 0.697 / 0.000 |
|
||||
| 0.7–0.8 | 2 / 0.745 / 0.000 | 2 / 0.770 / 0.000 | 5 / 0.754 / 0.200 |
|
||||
|
||||
| Any-channel feature (all events) | Coefficient |
|
||||
|---|---|
|
||||
| intercept | -3.768 |
|
||||
| log DR plays t | +0.560 |
|
||||
| log DR plays 4w | +0.510 |
|
||||
| log DR plays 12w | -0.409 |
|
||||
| log max DR plays 12w | +0.318 |
|
||||
| in rotation anywhere now | -0.474 |
|
||||
| station groups in rotation now (/5) | -0.249 |
|
||||
| current run anywhere (/12) | -1.173 |
|
||||
| weeks since in rotation anywhere (/12) | -0.489 |
|
||||
| Track Top-40 score | -0.265 |
|
||||
| Airplay Top-20 score | -0.317 |
|
||||
| chart entry recency 6w | +0.685 |
|
||||
| chart entry recency 26w | +0.501 |
|
||||
| log Track Top-40 hits | -0.291 |
|
||||
| log Airplay Top-20 hits | -0.120 |
|
||||
| Danish | +0.548 |
|
||||
| group | +0.013 |
|
||||
| female | -0.075 |
|
||||
| career years (/50) | +0.422 |
|
||||
| debut unknown | +0.435 |
|
||||
| intercept | -5.682 |
|
||||
| log DR plays t | +0.672 |
|
||||
| log DR plays 4w | +0.859 |
|
||||
| log DR plays 12w | -0.429 |
|
||||
| log max DR plays 12w | +0.201 |
|
||||
| in rotation anywhere now | -0.575 |
|
||||
| station groups in rotation now (/5) | -0.131 |
|
||||
| current run anywhere (/12) | -1.254 |
|
||||
| weeks since in rotation anywhere (/12) | -0.225 |
|
||||
| Track Top-40 score | +0.602 |
|
||||
| Airplay Top-20 score | -0.736 |
|
||||
| chart entry recency 6w | +0.145 |
|
||||
| chart entry recency 26w | -0.668 |
|
||||
| log Track Top-40 hits | -0.292 |
|
||||
| log Airplay Top-20 hits | -0.136 |
|
||||
| Danish | +0.821 |
|
||||
| group | -0.081 |
|
||||
| female | -0.185 |
|
||||
| career years (/50) | +0.645 |
|
||||
| debut unknown | +0.056 |
|
||||
| Christmas in window | +0.000 |
|
||||
| discography known | -0.000 |
|
||||
| release recency 4w | +0.216 |
|
||||
| release recency 12w | -0.783 |
|
||||
| log releases last 52w | -0.189 |
|
||||
| release recency 4w | -0.154 |
|
||||
| release recency 12w | -1.148 |
|
||||
| log releases last 52w | -0.286 |
|
||||
| releases in the window | -0.000 |
|
||||
| releases in window x not in rotation now | +0.474 |
|
||||
| DR: weeks since last add 13-25 | -0.050 |
|
||||
| DR: last add 26+ weeks ago | +0.502 |
|
||||
| DR: no add in the panel yet | -0.677 |
|
||||
| Airplay Top-20: last new song 13-25 weeks ago | +0.465 |
|
||||
| Airplay Top-20: 0.5-1.25x own usual gap | -0.144 |
|
||||
| Airplay Top-20: last new song 2+ years ago | -0.113 |
|
||||
| log archive plays per 100 sampled hours (2012-21) | +0.515 |
|
||||
| log DR lists with the artist before t (2023-) | +0.105 |
|
||||
| DR list recency 26w | -0.984 |
|
||||
| DR lists known before t at all | +0.593 |
|
||||
| NMF Denmark in the window | +0.836 |
|
||||
| NMF Denmark top 10 in the window | +0.682 |
|
||||
| log NMF Denmark placements last 52w | -0.067 |
|
||||
| share of releases on NMF Denmark (2 years) | -0.273 |
|
||||
| season index of the window (log, IFPI 2002-24) | +0.334 |
|
||||
| releases in window x not in rotation now | +0.575 |
|
||||
| DR: weeks since last add 13-25 | -0.412 |
|
||||
| DR: last add 26+ weeks ago | -0.142 |
|
||||
| DR: no add in the panel yet | -0.811 |
|
||||
| Airplay Top-20: last new song 13-25 weeks ago | +0.833 |
|
||||
| Airplay Top-20: 0.5-1.25x own usual gap | -0.222 |
|
||||
| Airplay Top-20: last new song 2+ years ago | +0.215 |
|
||||
| log archive plays per 100 sampled hours (2012-21) | +0.591 |
|
||||
| log DR lists with the artist before t (2023-) | +0.160 |
|
||||
| DR list recency 26w | -0.506 |
|
||||
| DR lists known before t at all | -0.279 |
|
||||
| NMF Denmark in the window | +0.923 |
|
||||
| NMF Denmark top 10 in the window | +0.249 |
|
||||
| log NMF Denmark placements last 52w | +0.276 |
|
||||
| share of releases on NMF Denmark (2 years) | +0.002 |
|
||||
| season index of the window (log, IFPI 2002-24) | -0.071 |
|
||||
|
||||
+19
-18
@@ -419,7 +419,8 @@ impl AddModel {
|
||||
/// within 8 weeks?" Trained only on artist-weeks where the artist did release in week t+1, with
|
||||
/// the artist's state the week before (earlier releases and NMF placements count as history;
|
||||
/// the new song's own NMF placement is the scenario variable).
|
||||
pub fn fit_events(p: &Panel, r: &Rot, last_week: usize) -> AddModel { Self::fit_with(p, r, 0, last_week, true, W) }
|
||||
/// As `fit_events`, for a window of `win` weeks (6-origin recalibration; used in backtests).
|
||||
pub fn fit_events_win(p: &Panel, r: &Rot, last_week: usize, win: usize) -> AddModel { Self::fit_with(p, r, 0, last_week, true, win) }
|
||||
/// The site's release-event model (8-week window, pooled recalibration).
|
||||
pub fn fit_events_site(p: &Panel, r: &Rot, last_week: usize) -> AddModel { Self::fit_events_pooled(p, r, last_week, W) }
|
||||
/// The release-event model for a window of `win` weeks.
|
||||
@@ -771,29 +772,29 @@ pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md:
|
||||
/// Walk-forward test of the release-event model: at each origin t0 the model sees only events
|
||||
/// whose window had closed (a week before t0), and scores the releases of week t0+1. Pre-release
|
||||
/// rows blend the new song's NMF placement from the artist's history (as on the site).
|
||||
pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
||||
pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, win: usize, md: &mut String) -> Result<Vec<(String, usize, String, f64)>> {
|
||||
let r = Rot::new(p, tau);
|
||||
let mut rows = Vec::new();
|
||||
let (mut s_c, mut s_a) = (Scores::new(SCORED.to_vec()), Scores::new(SCORED.to_vec()));
|
||||
let (mut pool_c, mut pool_a) = (Pool(Vec::new()), Pool(Vec::new()));
|
||||
let lvl = |m: f64| ((m.ln_1p() / 0.5) as usize).min(7);
|
||||
let level = |a: usize, t: usize| (0..LAGS).map(|i| p.group_total(a, t - i, None)).sum::<f64>() / LAGS as f64;
|
||||
for t0 in start..p.w().saturating_sub(W + 1) {
|
||||
let m = AddModel::fit_events(p, &r, t0);
|
||||
for t0 in start..p.w().saturating_sub(win + 1) {
|
||||
let m = AddModel::fit_events_win(p, &r, t0, win);
|
||||
// Baselines from the training events. Release timing only: by weeks since the previous release.
|
||||
let (mut by_rot, mut by_lvl, mut cr, mut cl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8], [[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
||||
let mut tm = Timing::new();
|
||||
for t in origins(0, t0) {
|
||||
for t in origins_w(0, t0, win) {
|
||||
for a in 0..p.a() {
|
||||
if !p.known(a, t) || !clean_release(p, a, t + 1) { continue; }
|
||||
let key = since_bin(p, a, t);
|
||||
let y = r.add_any_in_window(a, t, 0) as i32 as f64;
|
||||
let y = r.add_any_in(a, t, 0, win) as i32 as f64;
|
||||
let on = r.any(a, t) as usize;
|
||||
by_rot[on][0] += y; by_rot[on][1] += 1.0;
|
||||
let b = lvl(level(a, t)); by_lvl[b][0] += y; by_lvl[b][1] += 1.0;
|
||||
tm.any[key][0] += y; tm.any[key][1] += 1.0;
|
||||
for c in 0..C {
|
||||
let yc = r.add_in_window(a, c, t, 0) as i32 as f64;
|
||||
let yc = r.add_in(a, c, t, 0, win) as i32 as f64;
|
||||
let onc = r.on(a, c, t) as usize;
|
||||
cr[onc][0] += yc; cr[onc][1] += 1.0;
|
||||
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t - i)).sum::<f64>() / LAGS as f64);
|
||||
@@ -802,20 +803,20 @@ pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String)
|
||||
}
|
||||
}
|
||||
}
|
||||
let gap = W + 1;
|
||||
let gap = win + 1;
|
||||
let ((pl_c, iso_c), (pl_a, iso_a)) = (pool_c.fit(t0, gap), pool_a.fit(t0, gap));
|
||||
let mut started = false;
|
||||
for a in 0..p.a() {
|
||||
if !p.known(a, t0) || !clean_release(p, a, t0 + 1) { continue; }
|
||||
if !started { s_c.start_origin(); s_a.start_origin(); started = true; }
|
||||
let q = p_nmf(p, a, t0);
|
||||
let (pcn, pan) = m.predict(p, &r, a, t0, Scenario::new(true, true), None);
|
||||
let (pcr, par) = m.predict(p, &r, a, t0, Scenario::new(true, false), None);
|
||||
let (rcn, ran) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, true));
|
||||
let (rcr, rar) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, false));
|
||||
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::ACTUAL, None);
|
||||
let (pcn, pan) = m.predict(p, &r, a, t0, Scenario::new(true, true).window(win), None);
|
||||
let (pcr, par) = m.predict(p, &r, a, t0, Scenario::new(true, false).window(win), None);
|
||||
let (rcn, ran) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, true).window(win));
|
||||
let (rcr, rar) = m.predict_raw(p, &r, a, t0, 0, Scenario::new(true, false).window(win));
|
||||
let (pcx, pax) = m.predict(p, &r, a, t0, Scenario::actual(win), None);
|
||||
let key = since_bin(p, a, t0);
|
||||
let y = r.add_any_in_window(a, t0, 0);
|
||||
let y = r.add_any_in(a, t0, 0, win);
|
||||
let on = r.any(a, t0) as usize;
|
||||
let raw = q * ran + (1.0 - q) * rar;
|
||||
let tma = tm.any[key];
|
||||
@@ -823,7 +824,7 @@ pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String)
|
||||
q * pan + (1.0 - q) * par, raw, pl_a.apply(raw), iso_a.apply(raw), pax], y);
|
||||
pool_a.0.push((t0, raw, y));
|
||||
for c in 0..C {
|
||||
let yc = r.add_in_window(a, c, t0, 0);
|
||||
let yc = r.add_in(a, c, t0, 0, win);
|
||||
let onc = r.on(a, c, t0) as usize;
|
||||
let bc = lvl((0..LAGS).map(|i| p.y(a, c, t0 - i)).sum::<f64>() / LAGS as f64);
|
||||
let rawc = q * rcn[c] + (1.0 - q) * rcr[c];
|
||||
@@ -836,10 +837,10 @@ pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, md: &mut String)
|
||||
eprintln!("event backtest: origin {} done", p.weeks[t0]);
|
||||
}
|
||||
writeln!(md, "# Backtest: if the artist releases a new song, will DR add it?\n")?;
|
||||
writeln!(md, "Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the {CLEAN_WEEKS} weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after {GAP}+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within the {W} weeks from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. \"Release timing only\" = the add rate by weeks since the artist's previous release. {} release events scored, {} ({:.1}%) got an add on some DR channel.\n",
|
||||
writeln!(md, "Events: an artist knowable at the time releases something (Deezer) in week t0+1, with no other release in the {CLEAN_WEEKS} weeks before (so an add can't be the previous song's); target: a DR add (entering rotation after {GAP}+ weeks out, or plays jumping well above the artist's own 8-week baseline and staying up) within {win} week(s) from the release week. The model knows the artist's state the week before; pre-release rows blend the new song's NMF Denmark placement from the artist's history. \"Release timing only\" = the add rate by weeks since the artist's previous release. {} release events scored, {} ({:.1}%) got an add on some DR channel.\n",
|
||||
s_a.n, s_a.hits, 100.0 * s_a.hits / s_a.n.max(1.0))?;
|
||||
table(md, &s_c, &s_a, &mut rows, 0, "event_")?;
|
||||
let m = AddModel::fit_events(p, &r, p.w() - 1);
|
||||
table(md, &s_c, &s_a, &mut rows, win, "event_")?;
|
||||
let m = AddModel::fit_events_win(p, &r, p.w() - 1, win);
|
||||
writeln!(md, "| Any-channel feature (all events) | Coefficient |\n|---|---|")?;
|
||||
for (n, b) in ANY_NAMES.iter().chain(REL_NAMES).chain(WAVE_NAMES).chain(LONG_NAMES).chain(NMF_NAMES).chain(SEASON_NAMES).zip(&m.any) { writeln!(md, "| {n} | {b:+.3} |")?; }
|
||||
Ok(rows)
|
||||
|
||||
+2
-1
@@ -100,7 +100,8 @@ pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
||||
if let Some(r) = arg(args, "--rule") { let _ = adds::ADD_RULE.set(r); }
|
||||
let tau = common::thresholds(&p, 26);
|
||||
let mut md = String::new();
|
||||
let rows = adds::backtest_events(&p, &tau, 26, &mut md)?;
|
||||
let win: usize = arg(args, "--w").and_then(|w| w.parse().ok()).unwrap_or(2);
|
||||
let rows = adds::backtest_events(&p, &tau, 26, win, &mut md)?;
|
||||
rank::create_tables(db)?;
|
||||
let at = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string();
|
||||
for (m, h, metric, v) in &rows {
|
||||
|
||||
+2
-1
@@ -116,7 +116,8 @@ pub fn run(db: &Db, p: &Panel, max_h: usize, show_h: usize) -> Result<Vec<Line>>
|
||||
let rot = Rot::new(p, &tau);
|
||||
// The release-event model ("if they release a new song, will DR add it?"), the same for every
|
||||
// window start: the artist's state now, the new song's NMF placement blended from history.
|
||||
let ev = AddModel::fit_events_site(p, &rot, p.w() - 1);
|
||||
// The site's default: added within 2 weeks of the release.
|
||||
let ev = AddModel::fit_events_pooled(p, &rot, p.w() - 1, 2);
|
||||
let t0 = p.w() - 1;
|
||||
// A new song released in week t0+h+1: time-since features moved forward to then; none if too
|
||||
// soon after the artist's last release.
|
||||
|
||||
+8
-4
@@ -26,7 +26,10 @@ use tiny_http::{Header, Method, Response, Server};
|
||||
const PAGE: &str = include_str!("web/index.html");
|
||||
const MAX_H: usize = 12;
|
||||
/// Window lengths offered (weeks after the release within which DR must add the song).
|
||||
const WINDOWS: [usize; 6] = [2, 4, 6, 8, 10, 12];
|
||||
const WINDOWS: [usize; 7] = [1, 2, 4, 6, 8, 10, 12];
|
||||
/// The default window: most adds that come quickly come within two weeks, and a short window
|
||||
/// shows the difference between release weeks most clearly.
|
||||
const DEFAULT_WINDOW: usize = 2;
|
||||
/// Release weeks shown in the best-week calendar.
|
||||
const CALENDAR: usize = 26;
|
||||
|
||||
@@ -174,7 +177,7 @@ fn handle(st: &State, method: &Method, url: &str) -> Result<Reply> {
|
||||
"/" | "/index.html" => Ok((200, "text/html; charset=utf-8", Arc::new(PAGE.to_string()))),
|
||||
"/api/meta" => meta(st),
|
||||
"/api/rank" => rank(st, q.get("h").and_then(|h| h.parse().ok()).unwrap_or(0).min(CALENDAR - 1),
|
||||
q.get("w").and_then(|w| w.parse().ok()).filter(|w| WINDOWS.contains(w)).unwrap_or(8)),
|
||||
q.get("w").and_then(|w| w.parse().ok()).filter(|w| WINDOWS.contains(w)).unwrap_or(DEFAULT_WINDOW)),
|
||||
"/api/bestweek" => bestweek(st, &q),
|
||||
"/api/artist" => artist(st, q.get("id").map(String::as_str).unwrap_or("")),
|
||||
"/api/whatif" => whatif(st, &q),
|
||||
@@ -322,7 +325,7 @@ fn bestweek(st: &State, q: &HashMap<String, String>) -> Result<Reply> {
|
||||
let Some(&a) = m.index.get(q.get("id").map(String::as_str).unwrap_or("")) else {
|
||||
return Ok((404, "application/json", Arc::new(json!({"error": "artist not in the model"}).to_string())));
|
||||
};
|
||||
let win = q.get("w").and_then(|w| w.parse().ok()).filter(|w| WINDOWS.contains(w)).unwrap_or(8);
|
||||
let win = q.get("w").and_then(|w| w.parse().ok()).filter(|w| WINDOWS.contains(w)).unwrap_or(DEFAULT_WINDOW);
|
||||
let nmf = match q.get("nmf").map(String::as_str) { Some("yes") => Some(true), Some("no") => Some(false), _ => None };
|
||||
let p = &m.panel;
|
||||
let t0 = p.w() - 1;
|
||||
@@ -449,7 +452,8 @@ fn whatif(st: &State, q: &HashMap<String, String>) -> Result<Reply> {
|
||||
let run = |overrides: bool| -> Value {
|
||||
let f: Option<&dyn Fn(&mut [f64], [usize; 4])> = if overrides { Some(&over) } else { None };
|
||||
// A new release: the release-event model; no release: the window model (current songs only).
|
||||
let (add_c, add_any) = if release { m.events.predict_at(p, &m.rot, a, t0, h, crate::model::adds::Scenario::new(true, nmf).at(h), f) }
|
||||
let ev = m.events_w.get(&DEFAULT_WINDOW).unwrap_or(&m.events);
|
||||
let (add_c, add_any) = if release { ev.predict_at(p, &m.rot, a, t0, h, crate::model::adds::Scenario::new(true, nmf).at(h), f) }
|
||||
else { am.predict(p, &m.rot, a, t0, crate::model::adds::Scenario::new(false, false), f) };
|
||||
let mut x = vec![0.0; P];
|
||||
let mut mu = [0.0; C];
|
||||
|
||||
+15
-12
@@ -125,7 +125,7 @@ svg text { fill: var(--muted); font-size: 11px; }
|
||||
<div class="controls">
|
||||
<div class="ctl">Release week <b id="hlabel">—</b><input type="range" id="h" min="0" max="25" value="0"></div>
|
||||
<div class="ctl">Added within
|
||||
<select id="w"><option value="2">2 weeks</option><option value="4">4 weeks</option><option value="6">6 weeks</option><option value="8" selected>8 weeks</option><option value="10">10 weeks</option><option value="12">12 weeks</option></select>
|
||||
<select id="w"><option value="1">1 week</option><option value="2" selected>2 weeks</option><option value="4">4 weeks</option><option value="6">6 weeks</option><option value="8">8 weeks</option><option value="10">10 weeks</option><option value="12">12 weeks</option></select>
|
||||
</div>
|
||||
<div class="ctl">Station
|
||||
<select id="ch">
|
||||
@@ -170,7 +170,7 @@ svg text { fill: var(--muted); font-size: 11px; }
|
||||
<div class="ctl">From <input type="date" id="bw-from"></div>
|
||||
<div class="ctl">To <input type="date" id="bw-to"></div>
|
||||
<div class="ctl">Added within
|
||||
<select id="bw-w"><option value="2">2 weeks</option><option value="4">4 weeks</option><option value="6">6 weeks</option><option value="8" selected>8 weeks</option><option value="10">10 weeks</option><option value="12">12 weeks</option></select>
|
||||
<select id="bw-w"><option value="1">1 week</option><option value="2" selected>2 weeks</option><option value="4">4 weeks</option><option value="6">6 weeks</option><option value="8">8 weeks</option><option value="10">10 weeks</option><option value="12">12 weeks</option></select>
|
||||
</div>
|
||||
<div class="ctl">New Music Friday Denmark
|
||||
<select id="bw-nmf"><option value="auto">As often as their releases were placed</option><option value="yes">Placed</option><option value="no">Not placed</option></select>
|
||||
@@ -420,7 +420,7 @@ function whatIfUI(a, h) {
|
||||
const d = (x, y, f) => { const v = y - x; if (Math.abs(v) < 0.005 * (f === 'p' ? 1 : 100)) return ''; return `<span class="d ${v > 0 ? 'up' : 'down'}">${v > 0 ? '+' : ''}${f === 'p' ? Math.round(v * 100) + ' pts' : Math.round(v)}</span>`; };
|
||||
const rows = [['Any DR station', w.base.add_any, w.scenario.add_any]].concat(GROUPS.map(([g, l]) => [l, chanP(w.base, g), chanP(w.scenario, g)]));
|
||||
const onNow = Object.keys(w.current.in_rotation || {});
|
||||
$('wi-out').innerHTML = `<p class="note">${w.release ? (w.too_soon ? 'Too soon after their last release for a new-song estimate this week; the numbers below are the model\'s extrapolation.' : `A new song released the week of ${fmtDate(w.window_start)}: chance DR adds it within 8 weeks.`) : `No new release: chance of a new add from current songs in the 8 weeks from ${fmtDate(w.window_start)}.`} In rotation now on ${onNow.length ? onNow.length + ' station(s)' : 'no station'}. Now: ${w.current.t40 ? 'Track Top-40 #' + w.current.t40 : 'not on the Track Top-40'}, ${w.current.air ? 'Airplay Top-20 #' + w.current.air : 'not on the Airplay Top-20'}, ${w.current.entry == null ? 'no chart entry on record' : 'latest new chart entry ' + w.current.entry + ' weeks ago'}.</p>
|
||||
$('wi-out').innerHTML = `<p class="note">${w.release ? (w.too_soon ? 'Too soon after their last release for a new-song estimate this week; the numbers below are the model\'s extrapolation.' : `A new song released the week of ${fmtDate(w.window_start)}: chance DR adds it within 2 weeks.`) : `No new release: chance of a new add from current songs in the 8 weeks from ${fmtDate(w.window_start)}.`} In rotation now on ${onNow.length ? onNow.length + ' station(s)' : 'no station'}. Now: ${w.current.t40 ? 'Track Top-40 #' + w.current.t40 : 'not on the Track Top-40'}, ${w.current.air ? 'Airplay Top-20 #' + w.current.air : 'not on the Airplay Top-20'}, ${w.current.entry == null ? 'no chart entry on record' : 'latest new chart entry ' + w.current.entry + ' weeks ago'}.</p>
|
||||
<div class="kpis">${rows.map(([l, b, s]) => `<div class="kpi"><div>${l}</div><b>${pct(s)}</b>${d(b, s, 'p')}<div>was ${pct(b)}</div></div>`).join('')}
|
||||
<div class="kpi"><div>Plays, first week</div><b>${Math.round(w.scenario.mean)}</b>${d(w.base.mean, w.scenario.mean, 'n')}<div>was ${Math.round(w.base.mean)}</div></div></div>`;
|
||||
};
|
||||
@@ -430,17 +430,20 @@ function whatIfUI(a, h) {
|
||||
|
||||
// ---------- model tab ----------
|
||||
function renderModel() {
|
||||
const sc = meta.backtest || [];
|
||||
const get = (m, k) => sc.find(x => x.model === m && x.metric === k)?.value;
|
||||
const ms = [['rate by in-rotation-now', 'Rate by "in rotation now"'], ['rate by last-12-weeks level', 'Rate by last-12-weeks airplay'],
|
||||
['event model (pre-release: NMF from history)', 'Model, before release (as on this site)'], ['event model (post-release: NMF known)', 'Model, NMF placement known']];
|
||||
const sc = (meta.backtest || []).filter(x => x.metric.startsWith('event_'));
|
||||
const win = sc.length ? Math.max(...sc.map(x => x.h)) : 2;
|
||||
const get = (m, k) => sc.find(x => x.model === m && x.metric === k && x.h === win)?.value;
|
||||
const ms = [['rate by in-rotation-now', 'Rate by "in rotation now"'], ['rate by last-12-weeks level', 'Rate by last-12-weeks airplay'], ['release timing only', 'Release timing only'],
|
||||
['model + pooled isotonic (pre-release)', 'Model before release, isotonic (any station on this site)'], ['model + pooled Platt (pre-release)', 'Model before release, Platt (per station on this site)'],
|
||||
['model, NMF known (post-release)', 'Model, NMF placement known']];
|
||||
const f = (v, d) => v == null || isNaN(v) ? '—' : v.toFixed(d);
|
||||
const sk = get('event model', 'event_skill_any_vs_level');
|
||||
$('cards').innerHTML = sc.some(x => x.metric.startsWith('event_')) ? `<div class="card"><h3>If an artist releases a new song, does DR add it within 8 weeks?</h3>
|
||||
<table class="wtable"><thead><tr><th class="l"></th><th>Any station<br>log-loss</th><th>Any station<br>PR-AUC</th><th>Per station<br>PR-AUC</th></tr></thead><tbody>
|
||||
${ms.map(([m, l]) => `<tr><td class="l">${l}</td><td>${f(get(m, 'event_logloss_any'), 3)}</td><td>${f(get(m, 'event_prauc_any'), 3)}</td><td>${f(get(m, 'event_prauc_ch'), 3)}</td></tr>`).join('')}
|
||||
const ska = get('model + pooled isotonic (pre-release)', 'event_skill_any_vs_level'), skc = get('model + pooled Platt (pre-release)', 'event_skill_ch_vs_level');
|
||||
const sign = v => `<b class="${v > 0 ? 'up' : 'down'}">${v > 0 ? '+' : ''}${f(v, 3)}</b>`;
|
||||
$('cards').innerHTML = sc.length ? `<div class="card"><h3>If an artist releases a new song, does DR add it within ${win} week${win > 1 ? 's' : ''}?</h3>
|
||||
<table class="wtable"><thead><tr><th class="l"></th><th>Any station<br>log-loss</th><th>Any station<br>PR-AUC</th><th>Per station<br>log-loss</th><th>Per station<br>PR-AUC</th></tr></thead><tbody>
|
||||
${ms.map(([m, l]) => `<tr><td class="l">${l}</td><td>${f(get(m, 'event_logloss_any'), 3)}</td><td>${f(get(m, 'event_prauc_any'), 3)}</td><td>${f(get(m, 'event_logloss_ch'), 4)}</td><td>${f(get(m, 'event_prauc_ch'), 3)}</td></tr>`).join('')}
|
||||
</tbody></table>
|
||||
<p>Walk-forward over the past year: each week's releases scored by a model fitted only on releases whose outcome was already known. Skill of the before-release model vs last-12-weeks airplay: <b class="${sk > 0 ? 'up' : 'down'}">${sk > 0 ? '+' : ''}${f(sk, 3)}</b> (any station). Lower log-loss and higher PR-AUC are better; a leak test confirms no feature uses data from after the forecast week.</p></div>`
|
||||
<p>Walk-forward over the past year: each week's releases scored by a model fitted only on releases whose outcome was already known, artists included only from when they were knowable. Skill vs last-12-weeks airplay: ${sign(ska)} any station, ${sign(skc)} per station. Lower log-loss and higher PR-AUC are better; a leak test confirms no feature uses data from after the forecast week.</p></div>`
|
||||
: '<p class="note">No backtest stored yet (`airplay model events`).</p>';
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user