5 ms·

Whisper is great, but I also wouldn't overlook other next-generation models (RNN-T/Zipformer/etc) trained on 50k+ hour datasets. These also perform very well.

Whisper is great, but I also wouldn't overlook other next-generation models (RNN-T/Zipformer/etc) trained on 50k+ hour datasets. These also perform very well. That being said, Whisper is clearly a far cry from "intelligence". This should be clear when you feed it 5 seconds of silence and get hallucinated garbage in return. It's much more akin to compressing those huge datasets into something that can feasibly be run on recent hardware. That's not to downplay how impressive that is, just to draw a clear line between "compression" and "intelligence".

wyldfire 4y ago
Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines, I know I've gotten transcribed voicemails that really just are silence but the transcript shows lots and lots of hallucinated words. Regardless of "intelligence" it's got real utility.
- nmfisher 4y ago
  I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
  - pixl97 4y ago
    I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
- qwertox 4y ago
  I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
- uncanneyvalley 4y ago
  It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.
panarky 4y ago
It's a tool that's free as in freedom, and it is incredibly useful. How does discovering that it doesn't handle some weird use case diminish its utility as a tool? Hammers are awesome at driving nails into wood. But if you strike the wood directly without a nail, the hammer puts a dent in the wood. Is that a defect in the hammer? Does it somehow make the hammer any less useful as a tool?
- nmfisher 4y ago
  Not sure what makes you think I'm diminishing its utility as a tool; like I said, it's an incredible tool and I lean on it very heavily for various speech-processing pipelines. I'm just pointing out that Whisper definitely hasn't "solved" speech recognition, and there's still a lot more fertile ground to cover from a research perspective.
  - pixl97 4y ago
    In my personal believe I don't think speech recognition, at least based on any human model is solvable, at best we can get 'as good as an average set of humans that speak the language' People mumble crap all the time to other humans and need to repeat what they say with proper enunciation. People have hearing problems, which would correlate to microphone quality/placement issues when dealing with computer systems. Then there are issues where people say one thing incorrectly, but the person following/listening to the directions knows the procedure and does the correct thing. If you asked the speaker what they said again, they'd say they said the 'correct' thing in the first place. And this is something that I've done before as an example. Me: "Click the X button to start the process" Person writing the notes: "Click the Y button to start the process" Person writing the notes: You meant click the Y button right. Me: Yea, that's what I said. Oops, its when we get in things like this that we run into the unsolvable speech recognition issues because we don't generally understand our own error bars on what we say. The speech quality between a public speaker and the average joe I'm sure has a very wide range.
    - panarky 4y ago
      > at best we can get as good as an average set of humans Whisper is already superhuman, more accurate than experienced human transcribers.
jasode 4y ago
>, just to draw a clear line between "compression" and "intelligence" Not disagreeing with you but your sentence reminded of research looking at the link between "compression" and "intelligence" : https://www.google.com/search?q=compression+is+a+form+of+intelligence https://www.google.com/search?q=compression+is+a+form+of+int...

wyldfire 4y ago

Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines, I know I've gotten transcribed voicemails that really just are silence but the transcript shows lots and lots of hallucinated words. Regardless of "intelligence" it's got real utility.

nmfisher 4y ago
I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
- pixl97 4y ago
  I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
qwertox 4y ago
I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
uncanneyvalley 4y ago
It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.

panarky 4y ago

It's a tool that's free as in freedom, and it is incredibly useful. How does discovering that it doesn't handle some weird use case diminish its utility as a tool? Hammers are awesome at driving nails into wood. But if you strike the wood directly without a nail, the hammer puts a dent in the wood. Is that a defect in the hammer? Does it somehow make the hammer any less useful as a tool?

nmfisher 4y ago
Not sure what makes you think I'm diminishing its utility as a tool; like I said, it's an incredible tool and I lean on it very heavily for various speech-processing pipelines. I'm just pointing out that Whisper definitely hasn't "solved" speech recognition, and there's still a lot more fertile ground to cover from a research perspective.
- pixl97 4y ago
  In my personal believe I don't think speech recognition, at least based on any human model is solvable, at best we can get 'as good as an average set of humans that speak the language' People mumble crap all the time to other humans and need to repeat what they say with proper enunciation. People have hearing problems, which would correlate to microphone quality/placement issues when dealing with computer systems. Then there are issues where people say one thing incorrectly, but the person following/listening to the directions knows the procedure and does the correct thing. If you asked the speaker what they said again, they'd say they said the 'correct' thing in the first place. And this is something that I've done before as an example. Me: "Click the X button to start the process" Person writing the notes: "Click the Y button to start the process" Person writing the notes: You meant click the Y button right. Me: Yea, that's what I said. Oops, its when we get in things like this that we run into the unsolvable speech recognition issues because we don't generally understand our own error bars on what we say. The speech quality between a public speaker and the average joe I'm sure has a very wide range.
  - panarky 4y ago
    > at best we can get as good as an average set of humans Whisper is already superhuman, more accurate than experienced human transcribers.

jasode 4y ago

>, just to draw a clear line between "compression" and "intelligence" Not disagreeing with you but your sentence reminded of research looking at the link between "compression" and "intelligence" : https://www.google.com/search?q=compression+is+a+form+of+intelligence https://www.google.com/search?q=compression+is+a+form+of+int...