Per-channel history 2023- behind AIRPLAY_HISTORY=1: observation mask, Christmas re-entries out of the add target, archive weights; results in docs/HISTORY.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,37 @@
|
|||||||
|
# Per-channel history 2023- (archived DR artist pages)
|
||||||
|
|
||||||
|
`AIRPLAY_HISTORY=1` puts the Wayback copies of DR's current artist pages (per channel, from 2023-09-11)
|
||||||
|
before the live panel (from 2025-11-17): 165 weeks instead of 51. An artist-week in the archive part
|
||||||
|
counts only inside an archived copy's 52 weeks (17% of archive artist-weeks); a training row needs the
|
||||||
|
artist observed for the 20 weeks before its origin and the 15 after. Christmas re-entries are taken out
|
||||||
|
of the add target (119 Christmas artists; `AIRPLAY_XMAS=0` keeps them). `AIRPLAY_ARCH_W` weights
|
||||||
|
archive rows in training. Test origins are the same calendar weeks in every run (from 2026-05-18).
|
||||||
|
|
||||||
|
Release events (3,588, 64 adds): log loss / PR-AUC / ROC-AUC, raw model
|
||||||
|
|
||||||
|
| Training | Per channel | Any channel |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 year | 0.0163 / 0.095 / 0.964 | 0.0649 / 0.235 / 0.888 |
|
||||||
|
| 2023-26, full weight | 0.0150 / 0.122 / 0.968 | 0.0665 / 0.193 / 0.888 |
|
||||||
|
| 2023-26, archive weight 0.3 | 0.0146 / 0.140 / 0.971 | 0.0651 / 0.213 / 0.894 |
|
||||||
|
| 2023-26, archive weight 0.1 | 0.0146 / 0.142 / 0.969 | 0.0649 / 0.218 / 0.894 |
|
||||||
|
|
||||||
|
All artists, add in the next 8 weeks (h = 0 / h = 4): log loss and PR-AUC, raw model
|
||||||
|
|
||||||
|
| Training | Per channel, h=0 | Any, h=0 | Per channel, h=4 | Any, h=4 |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 year | 0.0134 / 0.266 | 0.0741 / 0.249 | 0.0160 / 0.170 | 0.0789 / 0.199 |
|
||||||
|
| 2023-26, full weight | 0.0134 / 0.279 | 0.0776 / 0.241 | 0.0160 / 0.191 | 0.0850 / 0.202 |
|
||||||
|
| 2023-26, archive weight 0.3 | 0.0138 / 0.275 | 0.0783 / 0.238 | 0.0171 / 0.180 | 0.0889 / 0.194 |
|
||||||
|
|
||||||
|
Findings (2026-11-19):
|
||||||
|
- More history ranks *which channel* adds better (per-channel PR-AUC up 5-50%), most for release events.
|
||||||
|
- *Whether any channel adds* gets worse in log loss: the history-trained model overpredicts today
|
||||||
|
(former favourites 4.5% predicted vs 2.1% observed). An archive-era dummy came out at +0.02, so it is
|
||||||
|
not a different base rate in the archive; more likely features whose meaning depends on how far back
|
||||||
|
the panel goes ("weeks since last add", "no add in the panel yet") shift between archive and live rows.
|
||||||
|
- NMF position score with history: no gain (per-channel PR-AUC 0.116 vs 0.122).
|
||||||
|
- Christmas 2025 release events (origins 17 Nov - 29 Dec 2025): 105 events, 9 adds - too few to judge;
|
||||||
|
the model did worse than "recent plays" there.
|
||||||
|
- Not switched on for the site. Next: cap the history-dependent features at 52 weeks so archive and live
|
||||||
|
rows mean the same thing, then re-test.
|
||||||
+30
-7
@@ -25,6 +25,8 @@ use anyhow::Result;
|
|||||||
use std::fmt::Write as _;
|
use std::fmt::Write as _;
|
||||||
|
|
||||||
pub const GAP: usize = 4;
|
pub const GAP: usize = 4;
|
||||||
|
/// Last forecast origin for the backtests (index), when set.
|
||||||
|
pub static TEST_END: std::sync::OnceLock<usize> = std::sync::OnceLock::new();
|
||||||
pub const W: usize = 8;
|
pub const W: usize = 8;
|
||||||
|
|
||||||
/// Per-channel rows: the panel features plus these.
|
/// Per-channel rows: the panel features plus these.
|
||||||
@@ -160,10 +162,11 @@ pub fn nmf_feats(p: &Panel, a: usize, t: usize, h: usize, sc: Scenario) -> [f64;
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Long-history covariates: DR's 2012–2021 archive and DR's real rotation lists before week t.
|
/// Long-history covariates: DR's 2012–2021 archive and DR's real rotation lists before week t.
|
||||||
pub const LONG: usize = 5;
|
pub const LONG: usize = 6;
|
||||||
pub const LONG_NAMES: &[&str] = &["log archive plays per 100 sampled hours (2012-21)", "log DR lists with the artist before t (2023-)",
|
pub const LONG_NAMES: &[&str] = &["log archive plays per 100 sampled hours (2012-21)", "log DR lists with the artist before t (2023-)",
|
||||||
"DR list recency 26w", "DR lists known before t at all",
|
"DR list recency 26w", "DR lists known before t at all",
|
||||||
"quiet former favourite (4+ weeks on DR 2019-24, out of rotation 26w)"];
|
"quiet former favourite (4+ weeks on DR 2019-24, out of rotation 26w)",
|
||||||
|
"archive-era row (before the live panel; absorbs the archive's popular-artist bias)"];
|
||||||
|
|
||||||
/// Not in rotation anywhere in the 26 weeks to t (as far as the panel goes back).
|
/// Not in rotation anywhere in the 26 weeks to t (as far as the panel goes back).
|
||||||
fn quiet(r: &Rot, a: usize, t: usize) -> bool { (t.saturating_sub(25)..=t).all(|w| !r.any(a, w)) }
|
fn quiet(r: &Rot, a: usize, t: usize) -> bool { (t.saturating_sub(25)..=t).all(|w| !r.any(a, w)) }
|
||||||
@@ -180,7 +183,7 @@ pub fn long_feats(p: &Panel, r: &Rot, a: usize, t: usize, _h: usize, sc: Scenari
|
|||||||
let before: Vec<&chrono::NaiveDate> = p.listed_on[a].iter().filter(|d| **d <= end_t).collect();
|
let before: Vec<&chrono::NaiveDate> = p.listed_on[a].iter().filter(|d| **d <= end_t).collect();
|
||||||
let since = before.last().map(|d| (end_t - **d).num_days() as f64 / 7.0 + shift as f64);
|
let since = before.last().map(|d| (end_t - **d).num_days() as f64 / 7.0 + shift as f64);
|
||||||
[p.archive_rate[a].ln_1p(), (before.len() as f64).ln_1p(), since.map(|s| (-s / 26.0).exp()).unwrap_or(0.0), (!before.is_empty()) as i32 as f64,
|
[p.archive_rate[a].ln_1p(), (before.len() as f64).ln_1p(), since.map(|s| (-s / 26.0).exp()).unwrap_or(0.0), (!before.is_empty()) as i32 as f64,
|
||||||
fav as i32 as f64]
|
fav as i32 as f64, (t < p.live0) as i32 as f64]
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Season: the mean log season index over the window's weeks (new Airplay Top-20 songs by week of
|
/// Season: the mean log season index over the window's weeks (new Airplay Top-20 songs by week of
|
||||||
@@ -323,6 +326,20 @@ impl Rot {
|
|||||||
if some && (1..=GAP).all(|k| !adds_any[a * nw + w - k]) { adds_any[a * nw + w] = true; }
|
if some && (1..=GAP).all(|k| !adds_any[a * nw + w - k]) { adds_any[a * nw + w] = true; }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Christmas re-entries aren't new adds: a Christmas artist's add in a week starting 27 Nov -
|
||||||
|
// 31 Dec with no release in the 12 weeks before it is taken out (AIRPLAY_XMAS=0 keeps them).
|
||||||
|
if std::env::var("AIRPLAY_XMAS").map_or(true, |v| v != "0") {
|
||||||
|
for a in (0..p.a()).filter(|&a| p.xmas[a]) {
|
||||||
|
for w in 0..nw {
|
||||||
|
let d = p.weeks[w];
|
||||||
|
if !(chrono::Datelike::month(&d) == 12 || (chrono::Datelike::month(&d) == 11 && chrono::Datelike::day(&d) >= 27)) { continue; }
|
||||||
|
let fresh = p.releases[a].as_ref().is_some_and(|r| r.iter().any(|x| *x > d - chrono::Duration::weeks(12) && *x <= d + chrono::Duration::days(6)));
|
||||||
|
if fresh { continue; }
|
||||||
|
adds_any[a * nw + w] = false;
|
||||||
|
for c in 0..C { adds[(a * C + c) * nw + w] = false; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Rot { nw, tau: *tau, rot, any, adds, adds_any }
|
Rot { nw, tau: *tau, rot, any, adds, adds_any }
|
||||||
}
|
}
|
||||||
pub fn on(&self, a: usize, c: usize, w: usize) -> bool { self.rot[(a * C + c) * self.nw + w] }
|
pub fn on(&self, a: usize, c: usize, w: usize) -> bool { self.rot[(a * C + c) * self.nw + w] }
|
||||||
@@ -417,6 +434,12 @@ pub struct AddModel {
|
|||||||
pub calib_any: Option<Calib>,
|
pub calib_any: Option<Calib>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Training weight of an archive-era row (AIRPLAY_ARCH_W, default 1; live rows always 1).
|
||||||
|
fn arch_w(p: &Panel, t: usize) -> f64 {
|
||||||
|
static W: std::sync::OnceLock<f64> = std::sync::OnceLock::new();
|
||||||
|
if t >= p.live0 { 1.0 } else { *W.get_or_init(|| std::env::var("AIRPLAY_ARCH_W").ok().and_then(|v| v.parse().ok()).unwrap_or(1.0)) }
|
||||||
|
}
|
||||||
|
|
||||||
/// Recent origins used to fit the recalibration.
|
/// Recent origins used to fit the recalibration.
|
||||||
const CAL_ORIGINS: usize = 6;
|
const CAL_ORIGINS: usize = 6;
|
||||||
|
|
||||||
@@ -521,10 +544,10 @@ impl AddModel {
|
|||||||
let quiet = (0..LAGS).all(|i| p.y(a, c, t - i) == 0.0);
|
let quiet = (0..LAGS).all(|i| p.y(a, c, t - i) == 0.0);
|
||||||
let wgt = if quiet && !events { if mix((a as u64) << 32 ^ (c as u64) << 20 ^ t as u64 ^ 0xadd) % 4 != 0 { continue; } 4.0 } else { 1.0 };
|
let wgt = if quiet && !events { if mix((a as u64) << 32 ^ (c as u64) << 20 ^ t as u64 ^ 0xadd) % 4 != 0 { continue; } 4.0 } else { 1.0 };
|
||||||
row_channel(p, r, a, c, t, h, sc, &mut x);
|
row_channel(p, r, a, c, t, h, sc, &mut x);
|
||||||
dc.push(&x, r.add_in(a, c, t, h, win) as i32 as f64, wgt);
|
dc.push(&x, r.add_in(a, c, t, h, win) as i32 as f64, wgt * arch_w(p, t));
|
||||||
}
|
}
|
||||||
row_any(p, r, a, t, h, sc, &mut xa);
|
row_any(p, r, a, t, h, sc, &mut xa);
|
||||||
da.push(&xa, r.add_any_in(a, t, h, win) as i32 as f64, 1.0);
|
da.push(&xa, r.add_any_in(a, t, h, win) as i32 as f64, arch_w(p, t));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
AddModel { h, win, channel: glm::logit(&dc, 1.0), any: glm::logit(&da, 1.0), platt_channel: (0.0, 1.0), platt_any: (0.0, 1.0), calib_channel: None, calib_any: None }
|
AddModel { h, win, channel: glm::logit(&dc, 1.0), any: glm::logit(&da, 1.0), platt_channel: (0.0, 1.0), platt_any: (0.0, 1.0), calib_channel: None, calib_any: None }
|
||||||
@@ -730,7 +753,7 @@ pub fn backtest(p: &Panel, tau: &[u32; C], horizons: &[usize], start: usize, md:
|
|||||||
writeln!(md, "An add = the artist enters rotation (plays ≥ τ) after at least {GAP} weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the {W} weeks that start h weeks after the forecast origin. Origins from week {}; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.\n", start + 1)?;
|
writeln!(md, "An add = the artist enters rotation (plays ≥ τ) after at least {GAP} weeks out of rotation, or their plays jump well above their own 8-week baseline and stay up; per channel, and on any DR channel. Target: an add within the {W} weeks that start h weeks after the forecast origin. Origins from week {}; each fit uses only windows closed (a week before) its origin; only artists knowable at the origin. Pre-release rows blend the window's NMF placement from the artist's history; release timing in the window is a scenario input.\n", start + 1)?;
|
||||||
for &h in horizons {
|
for &h in horizons {
|
||||||
if p.w() < start + h + W + 2 { continue; }
|
if p.w() < start + h + W + 2 { continue; }
|
||||||
let last_origin = p.w() - 1 - h - W - 1;
|
let last_origin = (p.w() - 1 - h - W - 1).min(*TEST_END.get().unwrap_or(&usize::MAX));
|
||||||
let (mut s_c, mut s_a) = (Scores::new(SCORED.to_vec()), Scores::new(SCORED.to_vec()));
|
let (mut s_c, mut s_a) = (Scores::new(SCORED.to_vec()), Scores::new(SCORED.to_vec()));
|
||||||
let (mut pool_c, mut pool_a) = (Pool(Vec::new()), Pool(Vec::new()));
|
let (mut pool_c, mut pool_a) = (Pool(Vec::new()), Pool(Vec::new()));
|
||||||
let lvl = |m12: f64| ((m12.ln_1p() / 0.5) as usize).min(7);
|
let lvl = |m12: f64| ((m12.ln_1p() / 0.5) as usize).min(7);
|
||||||
@@ -822,7 +845,7 @@ pub fn backtest_events(p: &Panel, tau: &[u32; C], start: usize, win: usize, md:
|
|||||||
// Release events by quiet artists (out of rotation 26 weeks): former favourites (4+ weeks on DR 2019-24) or not.
|
// Release events by quiet artists (out of rotation 26 weeks): former favourites (4+ weeks on DR 2019-24) or not.
|
||||||
let mut favs: Vec<(f64, f64, bool)> = Vec::new();
|
let mut favs: Vec<(f64, f64, bool)> = Vec::new();
|
||||||
let mut quiet_other: Vec<(f64, f64, bool)> = Vec::new();
|
let mut quiet_other: Vec<(f64, f64, bool)> = Vec::new();
|
||||||
for t0 in start..p.w().saturating_sub(win + 1) {
|
for t0 in start..p.w().saturating_sub(win + 1).min(TEST_END.get().map_or(usize::MAX, |e| e + 1)) {
|
||||||
let m = AddModel::fit_events_win(p, &r, t0, win);
|
let m = AddModel::fit_events_win(p, &r, t0, win);
|
||||||
// Baselines from the training events. Release timing only: by weeks since the previous release.
|
// Baselines from the training events. Release timing only: by weeks since the previous release.
|
||||||
let (mut by_rot, mut by_lvl, mut cr, mut cl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8], [[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
let (mut by_rot, mut by_lvl, mut cr, mut cl) = ([[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8], [[1.0f64, 2.0]; 2], [[1.0f64, 2.0]; 8]);
|
||||||
|
|||||||
+15
-3
@@ -38,11 +38,21 @@ fn arg(args: &[String], name: &str) -> Option<String> {
|
|||||||
args.iter().position(|a| a == name).and_then(|i| args.get(i + 1)).cloned()
|
args.iter().position(|a| a == name).and_then(|i| args.get(i + 1)).cloned()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/// Backtest origins: from --from (date) or 26 weeks into the live panel; up to --to (date) if given.
|
||||||
|
fn test_start(p: &Panel, args: &[String]) -> usize {
|
||||||
|
let idx = |s: String| chrono::NaiveDate::parse_from_str(&s, "%Y-%m-%d").ok().map(|d| p.weeks.iter().position(|w| *w >= d).unwrap_or(p.w()));
|
||||||
|
if let Some(e) = arg(args, "--to").and_then(idx) { let _ = adds::TEST_END.set(e); }
|
||||||
|
arg(args, "--from").and_then(idx).unwrap_or(p.live0 + 26)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
||||||
let p = Panel::load(db)?;
|
let p = Panel::load(db)?;
|
||||||
match args.get(2).map(String::as_str) {
|
match args.get(2).map(String::as_str) {
|
||||||
Some("panel") => {
|
Some("panel") => {
|
||||||
println!("{} artists × {} channels × {} complete weeks ({} to {})", p.a(), C, p.w(), p.weeks[0], p.weeks[p.w() - 1]);
|
println!("{} artists × {} channels × {} complete weeks ({} to {}; live from {})", p.a(), C, p.w(), p.weeks[0], p.weeks[p.w() - 1], p.weeks[p.live0]);
|
||||||
|
println!("observed artist-weeks in the archive part: {} of {}; Christmas artists: {}",
|
||||||
|
(0..p.a()).map(|a| (0..p.live0).filter(|&w| p.observed(a, w)).count()).sum::<usize>(), p.a() * p.live0, p.xmas.iter().filter(|x| **x).count());
|
||||||
let tau = common::thresholds(&p, p.w());
|
let tau = common::thresholds(&p, p.w());
|
||||||
for c in 0..C {
|
for c in 0..C {
|
||||||
let plays: f64 = (0..p.a()).flat_map(|a| (0..p.w()).map(move |w| (a, w))).map(|(a, w)| p.y(a, c, w)).sum();
|
let plays: f64 = (0..p.a()).flat_map(|a| (0..p.w()).map(move |w| (a, w))).map(|(a, w)| p.y(a, c, w)).sum();
|
||||||
@@ -119,7 +129,8 @@ pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
|||||||
let tau = common::thresholds(&p, 26);
|
let tau = common::thresholds(&p, 26);
|
||||||
let mut md = String::new();
|
let mut md = String::new();
|
||||||
let win: usize = arg(args, "--w").and_then(|w| w.parse().ok()).unwrap_or(2);
|
let win: usize = arg(args, "--w").and_then(|w| w.parse().ok()).unwrap_or(2);
|
||||||
let rows = adds::backtest_events(&p, &tau, 26, win, &mut md)?;
|
let start = test_start(&p, args);
|
||||||
|
let rows = adds::backtest_events(&p, &tau, start, win, &mut md)?;
|
||||||
rank::create_tables(db)?;
|
rank::create_tables(db)?;
|
||||||
let at = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string();
|
let at = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string();
|
||||||
for (m, h, metric, v) in &rows {
|
for (m, h, metric, v) in &rows {
|
||||||
@@ -135,7 +146,8 @@ pub fn run(db: &Db, args: &[String]) -> Result<()> {
|
|||||||
let horizons: Vec<usize> = arg(args, "--h").unwrap_or_else(|| "0,4".into()).split(',').filter_map(|s| s.trim().parse().ok()).collect();
|
let horizons: Vec<usize> = arg(args, "--h").unwrap_or_else(|| "0,4".into()).split(',').filter_map(|s| s.trim().parse().ok()).collect();
|
||||||
let tau = common::thresholds(&p, 26);
|
let tau = common::thresholds(&p, 26);
|
||||||
let mut md = String::new();
|
let mut md = String::new();
|
||||||
let rows = adds::backtest(&p, &tau, &horizons, 26, &mut md)?;
|
let start = test_start(&p, args);
|
||||||
|
let rows = adds::backtest(&p, &tau, &horizons, start, &mut md)?;
|
||||||
rank::create_tables(db)?;
|
rank::create_tables(db)?;
|
||||||
let at = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string();
|
let at = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string();
|
||||||
for (m, h, metric, v) in &rows {
|
for (m, h, metric, v) in &rows {
|
||||||
|
|||||||
+114
-4
@@ -6,9 +6,9 @@
|
|||||||
|
|
||||||
use crate::db::Db;
|
use crate::db::Db;
|
||||||
use anyhow::{bail, Result};
|
use anyhow::{bail, Result};
|
||||||
use chrono::{Duration, NaiveDate, Weekday};
|
use chrono::{Datelike, Duration, NaiveDate, Weekday};
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
use std::collections::{BTreeSet, HashMap};
|
use std::collections::{BTreeSet, HashMap, HashSet};
|
||||||
|
|
||||||
/// The DR music channels with one play count per artist and week (the P4 total is left out:
|
/// The DR music channels with one play count per artist and week (the P4 total is left out:
|
||||||
/// it double counts the ten regional P4s; P1 and P2 are talk and classical).
|
/// it double counts the ten regional P4s; P1 and P2 are talk and classical).
|
||||||
@@ -92,6 +92,15 @@ pub struct Panel {
|
|||||||
/// The first week (index) at which the artist was knowable: their first release, DR play,
|
/// The first week (index) at which the artist was knowable: their first release, DR play,
|
||||||
/// chart entry, DR list, archived play or NMF placement (0 = before the panel; w() = never).
|
/// chart entry, DR list, archived play or NMF placement (0 = before the panel; w() = never).
|
||||||
pub first_known: Vec<usize>,
|
pub first_known: Vec<usize>,
|
||||||
|
/// The first live week: weeks before it come from archived artist pages (Wayback, 2023-), where
|
||||||
|
/// an artist-week is only observed inside an archived copy's 52 weeks. Only with AIRPLAY_HISTORY=1 (off by default until it beats the one-year panel).
|
||||||
|
pub live0: usize,
|
||||||
|
/// Per artist and week: consecutive observed weeks ending at / starting from that week.
|
||||||
|
back: Vec<u16>,
|
||||||
|
fwd: Vec<u16>,
|
||||||
|
/// Christmas artists: plays at least 3x higher in December than in the surrounding half-year,
|
||||||
|
/// in at least two different years (2019-). Their December re-entries aren't new adds.
|
||||||
|
pub xmas: Vec<bool>,
|
||||||
/// Weeks with 20+ DR plays (all channels) on the archived artist pages, 2019-24 (before the panel).
|
/// Weeks with 20+ DR plays (all channels) on the archived artist pages, 2019-24 (before the panel).
|
||||||
pub fav_weeks: Vec<f64>,
|
pub fav_weeks: Vec<f64>,
|
||||||
/// Season index per ISO week (1..=53): new songs reaching the Airplay Top-20 that week of the
|
/// Season index per ISO week (1..=53): new songs reaching the Airplay Top-20 that week of the
|
||||||
@@ -109,7 +118,18 @@ impl Panel {
|
|||||||
|
|
||||||
/// Was the artist knowable by the end of week t (so a forecast made then could include them)?
|
/// Was the artist knowable by the end of week t (so a forecast made then could include them)?
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn known(&self, a: usize, t: usize) -> bool { self.first_known[a] <= t }
|
pub fn known(&self, a: usize, t: usize) -> bool { self.first_known[a] <= t && self.observed_around(a, t) }
|
||||||
|
|
||||||
|
/// Is the artist observed in the 20 weeks to t (or since the panel began) and the 15 weeks
|
||||||
|
/// after it (or to the panel's end)? Always true in the live weeks; in the archive weeks it keeps
|
||||||
|
/// unobserved weeks from counting as silence, in features and in targets.
|
||||||
|
#[inline]
|
||||||
|
pub fn observed_around(&self, a: usize, t: usize) -> bool {
|
||||||
|
let i = a * self.weeks.len() + t;
|
||||||
|
self.back[i] as usize >= 20.min(t + 1) && self.fwd[i] as usize >= 15.min(self.weeks.len() - t)
|
||||||
|
}
|
||||||
|
#[inline]
|
||||||
|
pub fn observed(&self, a: usize, w: usize) -> bool { self.back[a * self.weeks.len() + w] > 0 }
|
||||||
|
|
||||||
/// The panel as it was known at the end of week t: plays after t removed (and taken out of the
|
/// The panel as it was known at the end of week t: plays after t removed (and taken out of the
|
||||||
/// all-time counts), releases, NMF placements and lists after t dropped. For the leak test.
|
/// all-time counts), releases, NMF placements and lists after t dropped. For the leak test.
|
||||||
@@ -142,6 +162,19 @@ impl Panel {
|
|||||||
.collect();
|
.collect();
|
||||||
weeks.dedup();
|
weeks.dedup();
|
||||||
if weeks.len() < 30 { bail!("only {} complete weeks of DR artist history; run `airplay collect --only dr --backfill`", weeks.len()); }
|
if weeks.len() < 30 { bail!("only {} complete weeks of DR artist history; run `airplay collect --only dr --backfill`", weeks.len()); }
|
||||||
|
// Archive weeks (Wayback copies of the artist pages) before the live ones.
|
||||||
|
let has = |t: &str| -> Result<bool> { Ok(db.conn.query_row("SELECT count(*) FROM sqlite_master WHERE name = ?1", [t], |r| r.get::<_, i64>(0))? > 0) };
|
||||||
|
let history = std::env::var("AIRPLAY_HISTORY").is_ok_and(|v| v == "1") && has("dr_artist_weeks_arch")?;
|
||||||
|
let mut arch: Vec<NaiveDate> = Vec::new();
|
||||||
|
if history {
|
||||||
|
let first: Option<String> = db.conn.query_row("SELECT min(week_start) FROM dr_artist_weeks_arch", [], |r| r.get(0))?;
|
||||||
|
if let Some(mut d) = first.and_then(|s| NaiveDate::parse_from_str(&s, "%Y-%m-%d").ok()) {
|
||||||
|
while d < weeks[0] { arch.push(d); d += Duration::weeks(1); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let live0 = arch.len();
|
||||||
|
arch.extend(weeks);
|
||||||
|
let weeks = arch;
|
||||||
let week_of: HashMap<NaiveDate, usize> = weeks.iter().enumerate().map(|(i, d)| (*d, i)).collect();
|
let week_of: HashMap<NaiveDate, usize> = weeks.iter().enumerate().map(|(i, d)| (*d, i)).collect();
|
||||||
|
|
||||||
let mut artists = Vec::new();
|
let mut artists = Vec::new();
|
||||||
@@ -181,6 +214,33 @@ impl Panel {
|
|||||||
let Some(&w) = week_of.get(&d) else { continue };
|
let Some(&w) = week_of.get(&d) else { continue };
|
||||||
y[(a * C + c) * nw + w] = plays as f32;
|
y[(a * C + c) * nw + w] = plays as f32;
|
||||||
}
|
}
|
||||||
|
// Observed: every live week; in the archive, the 52 complete weeks before each archived copy.
|
||||||
|
let mut obs = vec![false; artists.len() * nw];
|
||||||
|
for a in 0..artists.len() { for w in live0..nw { obs[a * nw + w] = true; } }
|
||||||
|
if live0 > 0 {
|
||||||
|
for r in db.conn.prepare("SELECT artist_id, snapshot FROM arch_artist_captures WHERE fetched_at IS NOT NULL AND weeks > 0")?
|
||||||
|
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?)))? {
|
||||||
|
let (id, snap) = r?;
|
||||||
|
let (Some(&a), Ok(day)) = (id_of.get(&id), NaiveDate::parse_from_str(&snap[..8], "%Y%m%d")) else { continue };
|
||||||
|
let monday = day - Duration::days(day.weekday().num_days_from_monday() as i64);
|
||||||
|
for k in 1..=52 { if let Some(&w) = week_of.get(&(monday - Duration::weeks(k))) { if w < live0 { obs[a * nw + w] = true; } } }
|
||||||
|
}
|
||||||
|
let mut st = db.conn.prepare("SELECT artist_id, week_start, channel, plays FROM dr_artist_weeks_arch")?;
|
||||||
|
for r in st.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?, r.get::<_, String>(2)?, r.get::<_, i64>(3)?)))? {
|
||||||
|
let (id, ws, ch, plays) = r?;
|
||||||
|
let (Some(&a), Some(&c)) = (id_of.get(&id), ch_of.get(ch.as_str())) else { continue };
|
||||||
|
let Ok(d) = NaiveDate::parse_from_str(&ws, "%Y-%m-%d") else { continue };
|
||||||
|
let Some(&w) = week_of.get(&d) else { continue };
|
||||||
|
if w < live0 && obs[a * nw + w] { y[(a * C + c) * nw + w] = plays as f32; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let (mut back, mut fwd) = (vec![0u16; artists.len() * nw], vec![0u16; artists.len() * nw]);
|
||||||
|
for a in 0..artists.len() {
|
||||||
|
let mut run = 0u16;
|
||||||
|
for w in 0..nw { run = if obs[a * nw + w] { run.saturating_add(1) } else { 0 }; back[a * nw + w] = run; }
|
||||||
|
run = 0;
|
||||||
|
for w in (0..nw).rev() { run = if obs[a * nw + w] { run.saturating_add(1) } else { 0 }; fwd[a * nw + w] = run; }
|
||||||
|
}
|
||||||
|
|
||||||
let chart = chart_states(db, &weeks, &names, artists.len())?;
|
let chart = chart_states(db, &weeks, &names, artists.len())?;
|
||||||
let list_tau = super::calibrate::stored(db)?;
|
let list_tau = super::calibrate::stored(db)?;
|
||||||
@@ -189,13 +249,63 @@ impl Panel {
|
|||||||
let nmf_dk = load_nmf_dk(db, &id_of, &artists)?;
|
let nmf_dk = load_nmf_dk(db, &id_of, &artists)?;
|
||||||
let season = load_season(db)?;
|
let season = load_season(db)?;
|
||||||
let (alltime, genre_fit) = load_affinity(db, &id_of, artists.len())?;
|
let (alltime, genre_fit) = load_affinity(db, &id_of, artists.len())?;
|
||||||
let mut p = Panel { weeks, artists, y, chart, list_tau, releases, archive_rate, listed_on, nmf_dk, season, alltime, genre_fit, first_known: Vec::new(), fav_weeks: Vec::new() };
|
let mut p = Panel { weeks, artists, y, chart, list_tau, releases, archive_rate, listed_on, nmf_dk, season, alltime, genre_fit, first_known: Vec::new(), fav_weeks: Vec::new(),
|
||||||
|
live0, back, fwd, xmas: Vec::new() };
|
||||||
p.fav_weeks = load_fav_weeks(db, &id_of, p.a(), p.weeks[0])?;
|
p.fav_weeks = load_fav_weeks(db, &id_of, p.a(), p.weeks[0])?;
|
||||||
|
p.xmas = load_xmas(db, &id_of, &p)?;
|
||||||
p.first_known = first_known(db, &p, &id_of)?;
|
p.first_known = first_known(db, &p, &id_of)?;
|
||||||
Ok(p)
|
Ok(p)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Christmas artists, from weekly DR totals: the old artist pages (2019-24, within each copy's
|
||||||
|
/// 52 weeks) and the panel's observed weeks. Per December (weeks starting 27 Nov - 31 Dec, at least
|
||||||
|
/// 3 observed) against the observed weeks within 26 weeks of 10 December (at least 12):
|
||||||
|
/// (Dec mean + 0.5) / (other mean + 0.5) >= 3 in at least two different years.
|
||||||
|
fn load_xmas(db: &Db, id_of: &HashMap<String, usize>, p: &Panel) -> Result<Vec<bool>> {
|
||||||
|
let n = p.a();
|
||||||
|
let mut tot: Vec<HashMap<NaiveDate, f64>> = vec![HashMap::new(); n];
|
||||||
|
let mut seen: Vec<HashSet<NaiveDate>> = vec![HashSet::new(); n];
|
||||||
|
let has = |t: &str| -> Result<bool> { Ok(db.conn.query_row("SELECT count(*) FROM sqlite_master WHERE name = ?1", [t], |r| r.get::<_, i64>(0))? > 0) };
|
||||||
|
if has("old_artist_captures")? && has("dr_artist_history")? {
|
||||||
|
for r in db.conn.prepare("SELECT artist_id, snapshot FROM old_artist_captures WHERE fetched_at IS NOT NULL AND weeks > 0")?
|
||||||
|
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?)))? {
|
||||||
|
let (id, snap) = r?;
|
||||||
|
let (Some(&a), Ok(day)) = (id_of.get(&id), NaiveDate::parse_from_str(&snap[..8], "%Y%m%d")) else { continue };
|
||||||
|
let monday = day - Duration::days(day.weekday().num_days_from_monday() as i64);
|
||||||
|
for k in 1..=52 { seen[a].insert(monday - Duration::weeks(k)); }
|
||||||
|
}
|
||||||
|
for r in db.conn.prepare("SELECT artist_id, week_start, plays FROM dr_artist_history")?
|
||||||
|
.query_map([], |r| Ok((r.get::<_, String>(0)?, r.get::<_, String>(1)?, r.get::<_, f64>(2)?)))? {
|
||||||
|
let (id, ws, v) = r?;
|
||||||
|
if let (Some(&a), Ok(d)) = (id_of.get(&id), NaiveDate::parse_from_str(&ws, "%Y-%m-%d")) { if seen[a].contains(&d) { tot[a].insert(d, v); } }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for a in 0..n {
|
||||||
|
for w in 0..p.w() {
|
||||||
|
if !p.observed(a, w) { continue; }
|
||||||
|
let d = p.weeks[w];
|
||||||
|
if seen[a].insert(d) { tot[a].insert(d, (0..C).map(|c| p.y(a, c, w)).sum()); }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let xmas_week = |d: NaiveDate| d.month() == 12 || (d.month() == 11 && d.day() >= 27);
|
||||||
|
let last = p.weeks.last().map(|d| d.year()).unwrap_or(2026);
|
||||||
|
Ok((0..n).map(|a| {
|
||||||
|
let mut strong = 0;
|
||||||
|
for y in 2019..=last {
|
||||||
|
let mid = NaiveDate::from_ymd_opt(y, 12, 10).unwrap();
|
||||||
|
let (mut xs, mut xn, mut os, mut on) = (0.0, 0, 0.0, 0);
|
||||||
|
for d in &seen[a] {
|
||||||
|
if (*d - mid).num_days().abs() > 26 * 7 { continue; }
|
||||||
|
let v = tot[a].get(d).copied().unwrap_or(0.0);
|
||||||
|
if xmas_week(*d) && d.year() == y { xs += v; xn += 1; } else if !xmas_week(*d) { os += v; on += 1; }
|
||||||
|
}
|
||||||
|
if xn >= 3 && on >= 12 && (xs / xn as f64 + 0.5) / (os / on as f64 + 0.5) >= 3.0 { strong += 1; }
|
||||||
|
}
|
||||||
|
strong >= 2
|
||||||
|
}).collect())
|
||||||
|
}
|
||||||
|
|
||||||
fn load_fav_weeks(db: &Db, id_of: &HashMap<String, usize>, n: usize, start: NaiveDate) -> Result<Vec<f64>> {
|
fn load_fav_weeks(db: &Db, id_of: &HashMap<String, usize>, n: usize, start: NaiveDate) -> Result<Vec<f64>> {
|
||||||
let mut out = vec![0.0; n];
|
let mut out = vec![0.0; n];
|
||||||
let has: i64 = db.conn.query_row("SELECT count(*) FROM sqlite_master WHERE name = 'dr_artist_history'", [], |r| r.get(0))?;
|
let has: i64 = db.conn.query_row("SELECT count(*) FROM sqlite_master WHERE name = 'dr_artist_history'", [], |r| r.get(0))?;
|
||||||
|
|||||||
Reference in New Issue
Block a user