Collector for DR and Bauer airplay; data sources and modelling plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-15 00:42:05 +00:00
co-authored by Claude Opus 5.5
commit 6e4d2f1585
13 changed files with 2329 additions and 0 deletions
+47
View File
@@ -0,0 +1,47 @@
# Data sources
Found on 2026-11-15. All of these sources are public and need no login or API key. The collector
identifies itself and waits between requests.
**The big constraint:** every source of exact plays keeps only **about 7 days**. History only
exists from the day collection starts, so the collector runs every 6 hours.
## DR (P3, P4 regions, P5, P6 Beat, P8 Jazz)
DR's pages are Next.js. Each page's data is the JSON in `<script id="__NEXT_DATA__">` under
`props.pageProps`. (`api.dr.dk` needs an API key that isn't in the public pages, so it isn't used.)
| What | URL | Gives |
|---|---|---|
| A day's programmes | `/playlister/{channel}/{YYYY-MM-DD}` | `schedule.items[]`: `slug`, `title`, `startTime`, `endTime` (UTC) |
| A programme's plays | `/playlister/{channel}/{date}/{slug}` | `playlistIndexPoints[]`: `playedTime` (with offset), `title`, `durationMilliseconds`, `trackUrn`, `roles[]` (`artistUrn`, `name`, `role`; the main artist is `Hovedkunstner`) |
| An artist | `/lyd/musik/kunstner/{id}` (the slug is optional) | `weekPlaySums[]`: **52 weeks** of plays per channel; `mostPlayed[]`: all-time plays per track per channel; `latestPlayed[]`: the last 20 plays |
| Rotation lists | linked weekly from `dr.dk/rotationer` (`asset.dr.dk/.../DD-MM-YYYY-dr-rotationer.pdf`) | per channel, the songs in each rotation category A / B / C for the coming week |
- **Channels:** `p3`, `p4kbh`, `p4sjaelland`, `p4fyn`, `p4syd`, `p4trekanten`, `p4vest`, `p4aarhus`, `p4nord`, `p4bornholm`, `p4esbjerg`, `p5`, `p6beat`, `p8jazz`. The national `p4` page exists too, but its plays come from the regions.
- **Window:** dates older than about 7 days give 404.
- **Artist pages:** they give a year of weekly history the moment an artist is discovered. This is the only long history available.
- **Rotation PDF:**
- Text is extractable: categories as headings, then artist and title on the same line.
- Pages 0–2 and 11 hold the lists; the others are graphics.
- The channel names aren't in the text (probably logos), so mapping pages to channels is still to do.
- Only the current PDF is linked, and its URL changes weekly, so old weeks can't be fetched later.
## Bauer Media Denmark (NOVA, Radio Soft, The Voice, Pop FM, Radio 100, + web stations)
Radioplay became Rayo (`rayo.dk`). It uses Bauer's international API:
| What | URL | Gives |
|---|---|---|
| Stations | `https://listenapi.planetradio.co.uk/api9.2/stations/DK` | `stationCode`, `stationName` (about 80, many web-only) |
| Plays | `.../api9.2/events/{code}/{YYYY-MM-DD HH:MM:SS}/{n}` | the `n` (≤ 100) plays before that time, newest first: `nowPlayingTime` (Danish local time), `nowPlayingArtist`, `nowPlayingTrack`, `nowPlayingTrackId`, `nowPlayingDuration` (seconds) |
| Now playing | `.../api9.2/nowplaying/{code}` | the current event |
- **Codes:** NOVA `nov`, Radio Soft `dso`, The Voice `the`, Pop FM `pop`, Radio 100 `rhu`.
- **Errors:** older than about 7 days gives HTTP 200 with `["invalid date"]`; `n` > 100 gives `["invalid limit"]`.
- **Artists:** one string, featured artists included (`"Eminem ft. Rihanna"`), with no artist ids. Normalising names is part of the modelling work.
- **Why Bauer matters:** it records every play, including songs outside any published rotation list, and the same API serves Bauer's stations in other countries.
## Station ranking (Kantar)
The four biggest Danish stations for music are DR P4, DR P3, NOVA and Radio Soft; DR P4 is split into 10 regional channels. The collector covers all four, plus the other DR music channels and Bauer's other Danish stations.
+49
View File
@@ -0,0 +1,49 @@
# Models: predicting when an artist gets played again
## How airplay actually happens
The models should follow how stations really decide what to play:
1. **The playlist decides how often.** Each station has a playlist, split into rotation categories. DR publishes them weekly as A / B / C; roughly, A = several plays a day and C = a few a week. Songs are added and dropped weekly.
- DR adds mostly new releases.
- NOVA often adds a song only once it reaches the Spotify Top 50.
2. **The scheduler decides when.** Music-scheduling software places songs into clock slots under separation rules: a minimum gap before the same song or artist repeats, plus dayparting (which hours a song may play).
3. **The rest is outside the lists:** specialist shows, guest DJs, recurrent/gold songs, live sessions. Bauer's exact play logs capture these too.
So the question splits into three:
| Question | Time scale | Data |
|---|---|---|
| Will the song be added to the playlist, and at which station first? | weeks | rotation PDFs, plays across stations |
| While on the list, how often, and when in the day? | hours–days | exact plays |
| When does it drop out (and does it come back as a recurrent)? | weeks–months | plays, DR artist weekly history |
## Methods borrowed from finance and insurance
| Method | Comes from | Used here for |
|---|---|---|
| **Hazard / survival models** (Cox proportional hazards, Weibull, competing risks) | company bankruptcy, credit default, life insurance | how long a song stays in rotation; when it gets dropped (the "default"); being added (the competing event is "never added"). Covariates: station, category, artist history, release age, Spotify chart position. |
| **Hawkes processes** (self-exciting point processes) | trade arrivals, default contagion, earthquake aftershocks | play times: each play raises the short-term rate, and the decay shows the rotation. A multivariate Hawkes model across stations measures spillover: does a P3 play make a P4 or NOVA play more likely? |
| **Inhomogeneous Poisson / renewal processes** with a minimum gap | insurance claims arrivals | inter-play times within a category: a dead time (the separation rule) plus a time-of-day / day-of-week baseline. |
| **Regime switching** (hidden Markov models) | currency regimes, bull/bear markets | a song's lifecycle: not on list → new → A rotation → B/C → recurrent → gone. The weekly rotation lists label the states (for DR) and train the model, which can then infer them for Bauer, where nothing is published. |
| **Claims-frequency GLMs** (Poisson / negative binomial) | car insurance pricing | weekly play counts per song and station, with the overdispersion real data has. |
| **Credibility theory** (Bühlmann / empirical Bayes) | insurance premiums for small portfolios | new or small artists have little history; shrink their estimates towards the station or genre average. |
| **Bass diffusion / lead-lag** (Granger) | product adoption, cross-market lead-lag | how a song spreads from one station to the others, and which stations lead. |
| **State-space / Kalman filter** | volatility, nowcasting | a smooth "airplay momentum" per artist from noisy daily counts. |
## Evaluating predictions
The same way trading models are tested:
- **Walk-forward backtests:** fit on the past, predict the next day or week, roll forward. Never fit on the future.
- **Scores:** log-likelihood of the observed next-play time (probabilistic forecasts, not single numbers), and calibration ("events predicted at 30% happen 30% of the time").
- **Baselines to beat:** "same as last week", and a per-song average rate. A model that can't beat these isn't worth having.
## Order of work
1. **Collect** (done: every 6 hours) and **clean**: artist name normalisation across DR and Bauer, featured artists, track ids across stations.
2. **Describe:**
- inter-play distributions per category (they should show the separation gap);
- hour-of-day profiles;
- list lifetimes from DR's 52-week artist histories.
3. **Baselines**, then **survival models** for list lifetime, and **Hawkes / renewal models** for play times.
4. **Cross-station** models once there are a few months of data.