james/crator
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Track enrichment: Deezer popularity + ReccoBeats audio features #1

Merged
james merged 11 commits from enrichment into main 2026-08-07 18:50:10 +00:00
Conversation 0 Commits 11 Files changed 31 +2144 -8
james commented 2026-08-07 18:48:29 +00:00
Owner
Copy link

Populates danceability, valence and deezer_rank so the set arc runs on all four composite-intensity inputs instead of energy alone.

Before this, IntensityCtx::intensity renormalized over present fields and three of the four were None in every row — the arc worked, but it was a loudness curve wearing the composite's clothes.

Architecture

Two independent queues behind one runner. The providers look alike and behave nothing alike:

Deezer ReccoBeats
Payload artist + title text 4.9 MB audio excerpt
Per track ~200 ms decode + resample + upload
Rate limit 50 req / 5 s, published unpublished
Determinism drifts, 90-day TTL deterministic, cached until the file changes

Chaining them behind one queue would delay whole-library popularity (minutes) behind the upload grind (hours).

Measured, not assumed

enrichment/coverage_bench.rs runs the actual Rust matcher against a real 110-track library rather than inheriting figures from a throwaway prototype: 106/110 = 96%, better than the prototype's 95%. All four trap cases verified live:

Source Matched
Cedric Gervais — Molly (Quintino Remix) Cedric Gervais — MOLLY (213,287) — not Tyga at 369,021
Tiesto, Charli Xcx — Hot In It (Tiësto's…) Tiësto — Hot In It (feat. Charli XCX)
John Summit — palm of my hands (Odd Mob Extended Remix) John Summit — … (Odd Mob Remix)
The Schmidt — (Original Mix) The Schmidt — (Radio Edit)

The four misses are three genuine white labels plus Psytech — Fishbait, which the prototype had wrongly matched to Fishbone.

Decisions worth knowing

  • Match verification is a hard gate with no loose fallback tier. Deezer's artist:"..." is a soft ranking hint, not a filter. A high-rank false positive silently distorts the arc, which is worse than no data — renormalization handles absence gracefully but cannot detect a lie.
  • Resampling to 44.1 kHz is mandatory, not a preference. 28 s stereo at 48 kHz is 5,376,258 bytes and gets a hard 413. Measured.
  • Negative caching needed no new state. deezer_fetched_at set with deezer_rank NULL already means "looked, found nothing".
  • Exhausted retries leave a track unfetched, not negative-cached — a transient outage must not permanently blank a track.
  • ureq not reqwest — Cargo.toml says tokio is "kept narrow"; this job is rate-limited and sequential, so async buys nothing.

A failing test found a real design gap

strip_mix_suffix only removes generic markers, so Hot In It (Tiësto's Hotter Extended Mix) could never match its catalog entry — an artist's own named mix isn't generic. Fixed with strip_trailing_group as a last-resort candidate, tried after the exact title. Correct for this metric specifically: popularity answers "will the crowd recognize this tune".

Verification

  • 213 Rust tests pass, cargo check --all-targets clean, npm run build clean
  • Live Deezer test and the coverage bench both run green

Not covered

ReccoBeats has never run at scale — only ~40 hand-made requests, and its rate limit is unpublished. Paced conservatively at 1 req/s; the first real run needs watching for 429s.

Deezer's terms are non-commercial only — a documented release blocker before any open-source publication.

🤖 Generated with Claude Code

Populates `danceability`, `valence` and `deezer_rank` so the set arc runs on all four composite-intensity inputs instead of `energy` alone. Before this, `IntensityCtx::intensity` renormalized over present fields and three of the four were `None` in every row — the arc worked, but it was a loudness curve wearing the composite's clothes. ## Architecture Two independent queues behind one runner. The providers look alike and behave nothing alike: | | Deezer | ReccoBeats | |---|---|---| | Payload | artist + title text | 4.9 MB audio excerpt | | Per track | ~200 ms | decode + resample + upload | | Rate limit | 50 req / 5 s, published | **unpublished** | | Determinism | drifts, 90-day TTL | deterministic, cached until the file changes | Chaining them behind one queue would delay whole-library popularity (minutes) behind the upload grind (hours). ## Measured, not assumed `enrichment/coverage_bench.rs` runs the **actual Rust matcher** against a real 110-track library rather than inheriting figures from a throwaway prototype: **106/110 = 96%**, better than the prototype's 95%. All four trap cases verified live: | Source | Matched | |---|---| | Cedric Gervais — Molly (Quintino Remix) | **Cedric Gervais — MOLLY** (213,287) — *not* Tyga at 369,021 | | Tiesto, Charli Xcx — Hot In It (Tiësto's…) | Tiësto — Hot In It (feat. Charli XCX) | | John Summit — palm of my hands (Odd Mob Extended Remix) | John Summit — … (Odd Mob Remix) | | The Schmidt — (Original Mix) | The Schmidt — (Radio Edit) | The four misses are three genuine white labels plus `Psytech — Fishbait`, which the prototype had **wrongly** matched to Fishbone. ## Decisions worth knowing - **Match verification is a hard gate with no loose fallback tier.** Deezer's `artist:"..."` is a soft ranking hint, not a filter. A high-rank false positive silently distorts the arc, which is worse than no data — renormalization handles absence gracefully but cannot detect a lie. - **Resampling to 44.1 kHz is mandatory, not a preference.** 28 s stereo at 48 kHz is 5,376,258 bytes and gets a hard `413`. Measured. - **Negative caching needed no new state.** `deezer_fetched_at` set with `deezer_rank` NULL already means "looked, found nothing". - **Exhausted retries leave a track unfetched, not negative-cached** — a transient outage must not permanently blank a track. - **`ureq` not `reqwest`** — `Cargo.toml` says tokio is "kept narrow"; this job is rate-limited and sequential, so async buys nothing. ## A failing test found a real design gap `strip_mix_suffix` only removes *generic* markers, so `Hot In It (Tiësto's Hotter Extended Mix)` could never match its catalog entry — an artist's own named mix isn't generic. Fixed with `strip_trailing_group` as a **last-resort** candidate, tried after the exact title. Correct for this metric specifically: popularity answers "will the crowd recognize this tune". ## Verification - 213 Rust tests pass, `cargo check --all-targets` clean, `npm run build` clean - Live Deezer test and the coverage bench both run green ## Not covered **ReccoBeats has never run at scale** — only ~40 hand-made requests, and its rate limit is unpublished. Paced conservatively at 1 req/s; the first real run needs watching for 429s. Deezer's terms are **non-commercial only** — a documented release blocker before any open-source publication. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Pure matching primitives. Verification is a hard gate because Deezer's
artist:"..." is a soft ranking hint - artist:"Cedric Gervais"
track:"Molly" really returns Tyga at rank 369,021.

Tests pin both confirmed false positives and the diacritic case that must
still pass. Remix credits are deliberately NOT stripped: a remix is a
different recording.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resampling to 44.1kHz is mandatory, not a quality preference: 28s stereo at
48kHz is 5,376,258 bytes and gets a hard 413 from ReccoBeats. Downsampling
is equally forbidden - at 22.05kHz the same audio scored danceability 0.534
vs 0.787.

Mono sources are duplicated to stereo because mono uploads are rejected
with a 400. Every track gets an identical spec so the returned features stay
comparable across the library, which percentile ranking depends on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ureq rather than reqwest: this job is rate-limited and sequential, so async
buys nothing and reqwest would reverse the deliberate 'keep tokio narrow'
decision already recorded in Cargo.toml.

is_retryable draws the line that drives the whole backoff policy - transport
errors, 5xx and 429 retry; every other 4xx negative-caches, including 413
where resending identical bytes cannot help.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
reccobeats_fetched_at completes the pair-based negative cache: a timestamp
with no value means 'we looked and found nothing', which is what stops the
three unmatchable white labels being re-queried on every launch.

Deezer queues on a 90-day TTL because popularity is consumed as a percentile
within the crate - uniform catalogue drift does not reorder anything.
ReccoBeats has no TTL at all since its analysis is deterministic; staleness
is handled by clearing the cache when the file signature changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
select_match verifies artist AND title client-side and takes the first hit
that passes, rather than trusting data[0]. No loose fallback tier: dropping
it costs ~2% coverage and removes both confirmed false positives.

A failing test surfaced a real gap: "Hot In It (Tiesto's Hotter Extended
Mix)" could not match its catalog entry, because strip_mix_suffix only
removes GENERIC markers and an artist's own named mix is not one. Added
strip_trailing_group as a LAST-RESORT title candidate - the exact title is
still tried first. That is correct for this metric specifically: popularity
answers "will the crowd recognize this tune", and for both a third-party
remix and an artist's own extended mix the underlying track's recognition
is the right signal.

Verified against the live API, not just fixtures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both danceability and valence are required - a response carrying only one
parses as no result, so a half-populated row can never look like a success
to the composite-intensity renormalization.

Decode failures return Ok(None), a permanent negative-cacheable outcome,
rather than Err, which the runner would retry as a transient fault.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deezer drains at 40 req/5s on its own thread while ReccoBeats grinds at
1 req/s - chaining them would delay whole-library popularity behind hours of
audio upload.

Exhausting retries leaves a track UNFETCHED rather than negative-cached, so
a transient outage does not permanently blank a track; only a definitive
answer from the provider writes a timestamp.

ReccoBeats pacing is deliberately conservative: its rate limit is
unpublished, and ~40 requests without a 429 is weak evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Starts on launch, drains whatever is unenriched, then idles. The queue is a
DB query rather than in-memory state, so a restart resumes where it left off
with no extra bookkeeping. The existing ExitRequested handler already calls
jobs.cancel_all(), so enrichment winds down on exit for free.

Re-analysis now clears the ReccoBeats cache: analysis only re-runs when the
file signature changed, so the stored features describe audio that is gone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The status line appears only while the job is working. The per-track marker
is the signal that a track contributes less to the set arc than its
neighbours, which is otherwise completely invisible.

Import completion re-triggers enrichment so new tracks do not sit
unenriched until the next launch; start_enrichment is idempotent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 97%/95% figures in the design came from a Python prototype of the
matching rules, not from this code. This runs the actual Rust matcher so
the claim can be checked rather than inherited. Reads the DB path from
CRATOR_BENCH_DB and skips when unset, following accuracy_bench.

Measured on the 110-track library: 106/110 (96%), better than the
prototype. All four trap cases resolve correctly - Cedric Gervais - Molly
matches Cedric Gervais at rank 213,287 rather than Tyga at 369,021, and
Psytech - Fishbait is now correctly a MISS rather than a wrong match to
Fishbone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
james merged commit eb8ea449d4 into main 2026-08-07 18:50:10 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
james/crator!1
No description provided.