Old DR artist pages from the Wayback Machine (53 weeks of plays per capture, 2019-2024) into dr_artist_history; leak fixes (weeks since last add counts only adds up to t-1; training windows close a week before the origin) and a leak test (model leakcheck) that rebuilds every feature from data cut at the origin; backtests split pre-release (NMF from history) and post-release (NMF known); site shows the new-song backtest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-16 00:22:55 +00:00
co-authored by Claude Opus 5.5
parent 7a16eb45ef
commit 94abfa0255
9 changed files with 512 additions and 223 deletions
+9 -11
View File
@@ -186,8 +186,7 @@ svg text { fill: var(--muted); font-size: 11px; }
<p>DR publishes plays per artist, not per song, so "new add" can't yet tell a new song from an old one coming back. In the one week of song-level plays we have, only 3–12% of artists in rotation on a station had two songs in rotation at once, so "drops out first, then gets added" is the usual pattern. The weekly rotation lists (song level) will sharpen this.</p>
<p id="taus" class="note"></p>
</div>
<h2>Backtest: new adds</h2>
<p class="note prose">Walk-forward: for each week from week 27, the models are fitted only on what was known at the time and forecast 1, 4 and 8 weeks ahead. "Adds" are rows where the artist was not in rotation at the time of the forecast, which is the hard and interesting part. Skill above 0 means better than the baseline.</p>
<h2>Backtest</h2>
<div class="cards" id="cards"></div>
<div class="prose">
<h2>20 years of IFPI charts</h2>
@@ -432,18 +431,17 @@ function whatIfUI(a, h) {
// ---------- model tab ----------
function renderModel() {
const sc = meta.backtest || [];
const get = (m, h, k) => sc.find(x => x.model === m && x.h === h && x.metric === k)?.value;
const hs = [...new Set(sc.filter(x => x.metric.startsWith('add')).map(x => x.h))].sort((a, b) => a - b);
const ms = [['rate by in-rotation-now', 'Rate by "in rotation now"'], ['rate by last-12-weeks level', 'Rate by last-12-weeks level'], ['logit', 'Model']];
const get = (m, k) => sc.find(x => x.model === m && x.metric === k)?.value;
const ms = [['rate by in-rotation-now', 'Rate by "in rotation now"'], ['rate by last-12-weeks level', 'Rate by last-12-weeks airplay'],
['event model (pre-release: NMF from history)', 'Model, before release (as on this site)'], ['event model (post-release: NMF known)', 'Model, NMF placement known']];
const f = (v, d) => v == null || isNaN(v) ? '—' : v.toFixed(d);
$('cards').innerHTML = hs.map(h => {
const sk = get('logit', h, 'addany_skill_vs_level'), skc = get('logit', h, 'add_skill_vs_level');
return `<div class="card"><h3>Window starting ${h ? h + ' weeks ahead' : 'now'}</h3>
const sk = get('event model', 'event_skill_any_vs_level');
$('cards').innerHTML = sc.some(x => x.metric.startsWith('event_')) ? `<div class="card"><h3>If an artist releases a new song, does DR add it within 8 weeks?</h3>
<table class="wtable"><thead><tr><th class="l"></th><th>Any station<br>log-loss</th><th>Any station<br>PR-AUC</th><th>Per station<br>PR-AUC</th></tr></thead><tbody>
${ms.map(([m, l]) => `<tr><td class="l">${l}</td><td>${f(get(m, h, 'addany_logloss'), 3)}</td><td>${f(get(m, h, 'addany_prauc'), 3)}</td><td>${f(get(m, h, 'add_prauc'), 3)}</td></tr>`).join('')}
${ms.map(([m, l]) => `<tr><td class="l">${l}</td><td>${f(get(m, 'event_logloss_any'), 3)}</td><td>${f(get(m, 'event_prauc_any'), 3)}</td><td>${f(get(m, 'event_prauc_ch'), 3)}</td></tr>`).join('')}
</tbody></table>
<p>Skill vs the last-12-weeks baseline: <b class="${sk > 0 ? 'up' : 'down'}">${sk > 0 ? '+' : ''}${f(sk, 3)}</b> any station, <b class="${skc > 0 ? 'up' : 'down'}">${skc > 0 ? '+' : ''}${f(skc, 3)}</b> per station. Lower log-loss and higher PR-AUC are better.</p></div>`;
}).join('') || '<p class="note">No add backtest stored yet (`airplay model adds`).</p>';
<p>Walk-forward over the past year: each week's releases scored by a model fitted only on releases whose outcome was already known. Skill of the before-release model vs last-12-weeks airplay: <b class="${sk > 0 ? 'up' : 'down'}">${sk > 0 ? '+' : ''}${f(sk, 3)}</b> (any station). Lower log-loss and higher PR-AUC are better; a leak test confirms no feature uses data from after the forecast week.</p></div>`
: '<p class="note">No backtest stored yet (`airplay model events`).</p>';
}
// ---------- wiring ----------