3 ms·
> performs within 1% WER From the paper, for short-form audio: > the distil-large-v2 model achieves the lowest overall average WER of 10.1%. It is one percent
by mkl 3y ago
> performs within 1% WER
From the paper, for short-form audio:
> the distil-large-v2 model achieves the lowest overall average WER of 10.1%. It is one percentage point higher than the large-v2 baseline, with 5.8 times faster inference speed and fewer than half the parameters.
Long-form is similar, except Distil-Whisper does slightly better than Whisper (fewer hallucinations, apparently).
10% WER seems awfully high, and doesn't match my experience with Whisper. Maybe my audio is nice and clean relative to their test set?
- GaggiX 3y agoReading the paper the table is showing performance on out-of-ditribution test sets.
- refulgentis 3y agoI see: what does that mean, exactly? :) if it means "data they dont usually test on", 10% does still sound pretty high
- rjwilmsi 3y agoYes, it's not that clear to me either what test sets get a 10% error rate. Because in my use (native English dictation or native English podcast transcription) the small or medium original whisper models have what I'll call a "discrepancy" rate of say 1-2% which is mostly punctuation and "umms/errs" inclusion or not. The actual "error" rate is below 1% in my experience, and excluding surnames, brands and place names that I don't know how to spell either the remaining errors tend to be minor (missed plural etc.). So I infer that these data sets are some deliberately difficult audio: call centre recordings with lots of background noise, phoneline quality audio etc. Maybe non-native speakers. If I only heard that sort of audio once I also might have an error rate of 10%.
- rjwilmsi 3y agoI agree. When using the small or medium en models either for real-time speech recognition of a native English speaker or for transcribing podcasts of native English speakers the error rate is nowhere near 10%. I might say it's something like 1% of which the majority of errors are possibly subjective decisions about punctuation. But I have found the error rates are much higher on the tiny model and higher on the base model. I assume therefore that the 10% word error rate is on very difficult audio such as pilots speaking to Air Traffic Control (distorted or clipped microphones with significant background noise), which I personally find can be difficult to 100% understand even though I'm a native English speaker and when both pilots and air traffic control are native English speakers.
- smallerfish 3y agoWER is a pretty strict metric. IIRC it can penalize missing repeats, disfluencies, and the like that an ASR model may reasonably decide to drop. Additionally it will penalize for incorrect pluralization, unique proper nouns that aren't in the model, etc. 10% is still very readable. I built a tool in the mid-201Xs on an ASR engine with 20%+ WER, and even that was good enough for what we were trying to do.