3 ms·
They list the error rate on the git repo directly, it was never good even when it was the best. I saw mediocre results from the biggest model even when I gave
by ben_w 2y ago
They list the error rate on the git repo directly, it was never good even when it was the best.
I saw mediocre results from the biggest model even when I gave it a video of Tom Scott speaking at the Royal Institution where I could be extremely confident about the quality of the recording.
- mmcwilliams 2y agoWER is a decent metric to compare models but there's a difference between mistranscribing "effect" for "affect" and the kind of hallucinations Whisper has. I've run thousands of hours of audio through it for comparisons to other models and the kinds of thing you see Whisper inventing out of whole cloth is phrases like "please like and subscribe" in periods of silence. To me it suggested that it's trained off a lot of YouTube.
- ben_w 2y agoInteresting; that's certainly a bigger baseline than the one hour or so that I tried, which wasn't big enough to reveal that.