We deleted the stage that was supposed to fix our lyric timing
In short
We ran a spectral onset-snapping stage on top of forced alignment for six months because the literature and our own architecture doc both described it as the main accuracy lever. Measured against the JamendoLyrics human-annotated benchmark, it was making timing worse: mean absolute error 113ms and 35% of words within 50ms of truth, against 98ms and 51% with the stage simply turned off. Snapping moved words onto vibrato peaks, breath and mid-consonant transients as often as onto real word starts. We replaced it with a bounded refinement that can only sharpen an edge within 60ms of where the aligner already put it, for a final 97ms mean error, 47ms median and 52% within 50ms.
For about six months, Revori's lyric alignment ended with a stage called onset snapping. Every piece of research we read described it as the load-bearing part: the thing that turns approximate word timings into tight ones.
We measured it against human-labelled ground truth. It was making the timing worse. We deleted it, and the median word landed 31% closer to where it belongs.
This is what we did, what the numbers were, and why we couldn't see it for six months.
What the stage did
Forced alignment gives you a timestamp for every word. Ours comes from wav2vec2 running on a demucs-separated vocal stem. The timestamps are good but slightly late, because an aligner tends to mark where a phoneme is clearly present rather than where it began.
The standard fix is to snap. You run a spectral onset detector (superflux, in our case) over the vocal stem, get a list of acoustic events, and move each word onset to the nearest one within some window. Ours used plus or minus 120ms. The logic is clean: the aligner knows which word, the onset detector knows when something started, so combine them.
Everyone does this. It is in the papers. It was in our own architecture doc, described in those words: the load-bearing lever.
Why we couldn't tell it was broken
Because lyric timing is judged by ear, and the ear is a terrible A/B instrument.
When you watch a lyric video, you are not measuring milliseconds. You are forming an overall impression, and that impression is polluted by the font, the animation curve, the footage cut, the loudness of the kick, and whether you already know what the next word is. Change the snapping and re-watch, and you will have an opinion. It will not be reliable. We had opinions in both directions for months.
Worse, the failure was not uniform. Snapping helped some words and hurt others. Averaged across a song, it felt like noise. Averaged across a catalogue with a metric, it was not noise at all.
So the actual fix was not an algorithm. It was building something that could tell us we were wrong.
The benchmark
We used JamendoLyrics: 20 English songs, 1520 words, with manually labelled word onsets, released under MIT. Human ground truth, not another model's output.
Two design decisions made it useful rather than merely existent.
Score a window, not a whole song. Real uploads are roughly 30 second clips, not four minute masters. Scoring full songs adds a long tail of instrumental traversal that no user ever sees, and runs about eight times slower. We take the densest singing window of about 35 seconds per song, which is both faster and closer to what production actually does.
Split prep from eval. The expensive part (demucs, then forced alignment) does not change when you vary the post-processing. So we run it once per song and cache the raw pre-snap alignment. Each variant then re-applies only its own post-processing step against that identical cached alignment.
That second decision is what makes the comparison trustworthy. Every variant scores against a byte-identical alignment, so the only thing varying is the step under test. It is a perfectly controlled A/B rather than two separate pipeline runs that happen to differ in one setting.
We report mean absolute onset error (AAE), median error, and PCO@50: the percentage of words landing within 50ms of truth, which is roughly a frame and a half at 30fps.
The result
| mode | what it does | AAE | median | PCO@50 |
|---|---|---|---|---|
snap (legacy) | snap onset to nearest superflux onset, ±120ms | 113ms | 68ms | 35% |
off | trust the debiased aligner onset | 98ms | 48ms | 51% |
refine | nudge onset to the foot of its own energy rise, ±60ms | 97ms | 47ms | 52% |
Doing nothing beat the load-bearing lever. Turning snapping off dropped median error from 68ms to 48ms and took words landing within 50ms from 35% to 51%.
That is the whole finding. The most sophisticated stage in the chain was worse than deleting it.
Why snapping lost
Superflux onsets are not word onsets. They fire on any spectral flux increase, which in a sung vocal means vibrato peaks, consonant mid-points, breath, and the attack of a note that started three syllables ago.
So when you snap an already-decent onset to the nearest detected event, you move it away from truth roughly as often as toward it. The wins and losses cancel, and the losses are uglier, because a word yanked 90ms onto a vibrato peak reads as broken in a way that a word 40ms late does not.
We also tested the obvious response, which is to loosen detection so there are more onsets to choose from. That made it worse. More candidates means a closer wrong answer is always available.
What replaced it
The version we ship is called refine, and it is deliberately timid.
It anchors at the aligner's own onset and walks backward to the foot of that word's energy rise, bounded to plus or minus 60ms. It cannot jump to a different transient, because it never leaves the neighbourhood of the onset it started from. It only sharpens a leading edge that is already in the right place.
It also halves the systematic lateness. Mean bias went from +63ms to +24ms, which matters because a consistent lateness is exactly what reads as "not quite on the beat" even when nothing is obviously wrong.
The gain over simply switching snapping off is small: 48ms to 47ms median, 51% to 52% within tolerance. We kept it because the bias improvement is real and it costs almost nothing. But the honest summary is that most of the win came from deleting a stage, not from adding a better one.
It ships behind LYRIC_ONSET_REFINE, which takes refine, off, or snap, so the old behaviour is one environment variable away if we are wrong again.
What this benchmark does not prove
It uses perfect lyrics. Ground truth text is handed to the aligner, so the score isolates alignment quality from transcription quality.
Real uploads do not work that way. They depend on our ASR ensemble and the fusion step that reconciles it, and errors there produce timing failures that this benchmark cannot see. So the benchmark proves the aligner. It does not prove the product. A click track and a person listening remain the final gate before anything ships.
There is also a tail we have not fixed. Most of the remaining error sits in a handful of outlier songs rather than spread evenly. One track in the set sits at 466ms mean error with a p90 of 1309ms while the median song is fine. That tail is a different problem from the one this post is about, and it is the next thing worth paying for.
If you are building something similar
Three things we would tell ourselves six months earlier.
A stage everyone agrees is important is exactly the one to measure first. Consensus is why nobody checks. Ours had been described as load-bearing in our own documentation, which made it the last thing we suspected.
Build the controlled comparison before the improvement. The prep/eval cache took an afternoon. It is the only reason we could run seven variants and believe the ordering. Every hour spent on it bought back weeks of arguing about whether a change felt better.
Prefer bounded corrections to unbounded ones. Snapping could move a word anywhere within 120ms. Refine can only sharpen an edge within 60ms of where it already was. When your correction step cannot verify its own target, the bound is the thing keeping it honest.
The benchmark harness lives in scripts/bench/ and the ground truth data is a fetch script away, so any future alignment change has to beat 97 / 47 / 52% before it ships.
Revori is at revori.app if you want to hear what this sounds like on a real track.
Questions people ask about this
- Does snapping word timings to detected onsets improve lyric alignment?
- In our measurements, no. Snapping forced-alignment word onsets to the nearest spectral onset within 120ms increased mean absolute error from 98ms to 113ms and cut the share of words landing within 50ms of ground truth from 51% to 35%. Spectral onset detectors fire on vibrato peaks, breath and consonant mid-points as well as word starts, so the snap target is wrong often enough to cancel out the cases where it is right.
- What is a good benchmark for lyric alignment accuracy?
- JamendoLyrics, from Stoller et al., is the practical choice: 20 English songs with manually labelled word onsets, released under MIT. Score mean absolute onset error, median error and percentage of correct onsets within a 50ms tolerance, which is roughly a frame and a half at 30fps. Cache the expensive vocal separation and alignment steps once per song so that post-processing variants are compared against a byte-identical alignment.
- How accurate is automatic lyric timing?
- On a human-annotated benchmark with correct lyrics supplied, our aligner reaches 97ms mean absolute error, 47ms median, and 52% of words within 50ms of the labelled onset. That figure isolates alignment quality: it assumes perfect lyric text, so it measures the aligner rather than the end-to-end product, where transcription errors introduce timing failures the benchmark cannot see.