History: boundary fix (spurious adds, dropped rows), archive rows subsampled, both models use archive at 0.3; optional any>=channel floor; ROC; docs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,3 +35,30 @@ Findings (2026-11-19):
|
||||
the model did worse than "recent plays" there.
|
||||
- Not switched on for the site. Next: cap the history-dependent features at 52 weeks so archive and live
|
||||
rows mean the same thing, then re-test.
|
||||
|
||||
## Correction (2026-11-20): the earlier "history hurts" verdict was a bug
|
||||
|
||||
At the boundary between archive and live weeks, artists without an archived copy looked silent in the
|
||||
archive weeks: an artist already in rotation when the live data begins showed a spurious add, and the
|
||||
"20 observed weeks before a row" rule dropped uncovered artists from the first 20 live weeks, so the
|
||||
general model trained mostly on popular (archived) artists. Fixed: an add needs GAP (surge: 8) observed
|
||||
weeks before it, and a row needs the 12 observed lag weeks the one-year panel needs; with archive weight 0
|
||||
the history panel then reproduces the one-year model (up to the larger Christmas-artist list).
|
||||
|
||||
After the fix, recalibrated (Platt), same test weeks (May-Nov 2026):
|
||||
|
||||
| | 1 year | 2023-26, archive weight 0.3 |
|
||||
|---|---|---|
|
||||
| General model, any channel, log loss h=0 / h=4 | 0.0742 / 0.0789 | 0.0736 / 0.0782 |
|
||||
| General model, per channel, log loss h=0 / h=4 | 0.0135 / 0.0160 | 0.0130 / 0.0152 |
|
||||
| General model, per channel, PR-AUC h=0 / h=4 | 0.266 / 0.170 | 0.287 / 0.198 |
|
||||
| Release events, any channel, log loss / ROC-AUC / PR-AUC | 0.0655 / 0.882 / 0.224 | 0.0644 / 0.895 / 0.211 |
|
||||
| Release events, per channel, log loss | 0.0158 | 0.0151 |
|
||||
|
||||
History now wins on the coming period nearly everywhere; by the project rule (largest dataset, then the
|
||||
best accuracy on the coming period) it should be on. Archive rows are subsampled (share = weight) to keep
|
||||
memory down. Switching it on for the site was reverted after the service hit its 1.5 GB memory limit.
|
||||
|
||||
Also found: per-channel probabilities above ~0.2 are overconfident (40-50% predicted, ~8% observed),
|
||||
mostly P4 regions for artists already in rotation (Mumle: 44% on P4 vs 6% any channel). A floor making
|
||||
"any" >= the best station group made "any" worse (0.0664 -> 0.0690) until those are recalibrated.
|
||||
|
||||
Reference in New Issue
Block a user