3 ms·
> * https://github.com/openai/whisper https://github.com/openai/whisper I would be very concerned about any LLM model being used for "transcription", since the
by Terr_ 2y ago
> * https://github.com/openai/whisper https://github.com/openai/whisper
I would be very concerned about any LLM model being used for "transcription", since they may injecting things that nobody said, as in this recent item:
https://news.ycombinator.com/item?id=41968191 https://news.ycombinator.com/item?id=41968191
- ben_w 2y agoThey list the error rate on the git repo directly, it was never good even when it was the best. I saw mediocre results from the biggest model even when I gave it a video of Tom Scott speaking at the Royal Institution where I could be extremely confident about the quality of the recording.
- mmcwilliams 2y agoWER is a decent metric to compare models but there's a difference between mistranscribing "effect" for "affect" and the kind of hallucinations Whisper has. I've run thousands of hours of audio through it for comparisons to other models and the kinds of thing you see Whisper inventing out of whole cloth is phrases like "please like and subscribe" in periods of silence. To me it suggested that it's trained off a lot of YouTube.
- ben_w 2y agoInteresting; that's certainly a bigger baseline than the one hour or so that I tried, which wasn't big enough to reveal that.