4 ms·

Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines

Don't state of the art commercial systems do something similar? I assume there must be some automatic gain boosting the noise at the frontend of most pipelines, I know I've gotten transcribed voicemails that really just are silence but the transcript shows lots and lots of hallucinated words. Regardless of "intelligence" it's got real utility.

nmfisher 4y ago
I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.
- pixl97 4y ago
  I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.
qwertox 4y ago
I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.
- IanCal 4y ago
  Instead of setting up a machine for inference, try modal labs (no affiliation): https://modal.com/docs/guide/whisper-transcriber https://modal.com/docs/guide/whisper-transcriber Pay per second GPU processing, with an example of running whisper over 10 GPUs in parallel.
  - qwertox 4y ago
    Interesting. I thought that with these offerings I had to rent a VM with GPU and pay the hourly rate for as long as a VM is running. So this is really 0 USD when not in use? I'm also intending to use this for transcribing my phone answering machine recordings, so the transcription requests come in at random times which means that the transcription service should be constantly available.
    - IanCal 4y ago
      Most are, modal is a very different offering where it's $0 when not in use. They have some other very interesting ideas like charging you for CPU time rather than wall time. It's a newer business so I guess you should factor that risk in though.
uncanneyvalley 4y ago
It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.

nmfisher 4y ago

I occasionally get a single hallucinated word (more like a mis-transcription) where the audio contains a clunk/bang/cough/etc, but I've never had full hallucinated phrases from clean silence. There are a couple of GitHub discussions on the Whisper repository with various fixes/hacks to deal with it: https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/679 https://github.com/openai/whisper/discussions/813 https://github.com/openai/whisper/discussions/813 If you get a chance, I encourage you to try out the other newer models I mentioned, I think you'd be very impressed.

pixl97 4y ago
I don't see this much different than what commonly happens with humans when we hear our named called when it was some environmental noise. As for the silence, I wonder why the the model even receives it. I would think a lot of that would be compressed out of existence to save bandwidth.

qwertox 4y ago

I have an app on my phone which creates a 1 minute audio file when I press a button. I have a lavalier microphone connected to the phone and use it to record notes while riding my bike. It's always 1 minute because that is usually enough, and if I see that I need more time, I record an overlapping second file. Last week I set up a Whisper instance on my server and have been feeding it with these files. The result is pretty good. I usually can remember what I was saying when I read the transcription, which usually contains a couple of errors. Then there are those added hallucinations which are entire sentences, like: ---- 00:00.000 --> 00:05.000 Also temperaturmäßig ist es recht gut. [So temperature wise, it's pretty good.] 00:05.000 --> 00:09.000 Der eine hat 12 Grad, der andere 10. [One has 12 degrees, the other 10. (I have two temperature sensors mounted on the bike, ESP32 streaming the data to the phone via BLE)] 00:09.000 --> 00:12.000 Also sagen wir mal, 10 Grad. [So let's say 10 degrees.] 00:14.000 --> 00:19.000 Es ist bewölkt und windig. [It's cloudy and windy.] 00:20.000 --> 00:24.000 Aber irgendwie vom Wetter her gut. [But somehow from the weather it's good.] 00:24.000 --> 00:31.000 Ich habe heute überhaupt nichts gegessen und sehr wenig getrunken. [I ate nothing at all today and drank very little.] 00:54.000 --> 00:59.000 Vielen Dank für's Zuschauen! [Thanks for watching!] Transcribed in 77.2 seconds ---- The last sentence, "Thanks for watching!" is a complete hallucination. There were 30 seconds remaining which were me breathing and the wind blowing into the microphone and it came up with that comment. I usually comment on the weather because I take note of what I am wearing, and it allows me to better prepare for future rides. 77 seconds for the 60 second file because my server has no GPU, so I'm running the large model on the CPU (in a VM which has 8 cores assigned to it from a Ryzen 9 5950X). I've been considering buying a small PC with a 3060 RTX only for inferencing, but it may be too expensive. I tried Google Speech-To-Text and it is nowhere as good as Whisper under these conditions (having the wind noise and the heavy breathing). This is Google's result: ---- "Also temperaturmäßig es ist recht gut, der eine hat 12° andere 10. Es ist angemalte 10 Grad. Es ist bewölkt und windig, aber er hat sie vom Wetter her gut, ich wollte überhaupt nichts gegessen und sehr wenig getrunken." ["So temperature-wise it's pretty good, one has 12° other 10. It's painted 10 degrees. It's cloudy and windy, but he has it good from the weather, I did not want to eat anything at all and drank very little."] ---- Also, whisper.cpp doesn't seem to generate the same results, and they appear to be not so good (in this case it was almost just as good). I just tested the same file on whisper.cpp with the large model and it's even funnier: ---- [00:00:00.000 --> 00:00:05.000] also temperaturmäßig ist es recht gut [...] [00:00:05.000 --> 00:00:09.000] der eine hat 12 Grad, der andere 10 [...] [00:00:09.000 --> 00:00:12.000] also sagen wir mal so 10 Grad [...] [00:00:12.000 --> 00:00:19.000] es ist bewölkt und windig [...] [00:00:19.000 --> 00:00:24.000] aber irgendwie vom Wetter her gut [...] [00:00:27.000 --> 00:00:31.000] ich habe heute überhaupt nichts gegessen und sehr wenig getrunken [...] [00:00:31.000 --> 00:00:35.000] das ist der Grund, warum ich so viel auf dem Knie gehe [this is the reason why I go so much on the knee] [00:00:35.000 --> 00:00:39.000] das war's, bis zum nächsten Mal! [that's it, until next time!] [00:00:39.000 --> 00:00:59.000] Danke fürs Zuschauen! [Thanks for watching!] `time` yields 567.63s user 1.99s system 755% cpu 1:15.36 total ---- The first 30 seconds, where the text is clearly understood, is inferenced within ~10-15 seconds. It's the "silence" which makes the AI go crazy on the workload. The idea behind this is to set up a system which then sends me an email with a map and trail of the ride as well as the transcriptions of the notes.

IanCal 4y ago
Instead of setting up a machine for inference, try modal labs (no affiliation): https://modal.com/docs/guide/whisper-transcriber https://modal.com/docs/guide/whisper-transcriber Pay per second GPU processing, with an example of running whisper over 10 GPUs in parallel.
- qwertox 4y ago
  Interesting. I thought that with these offerings I had to rent a VM with GPU and pay the hourly rate for as long as a VM is running. So this is really 0 USD when not in use? I'm also intending to use this for transcribing my phone answering machine recordings, so the transcription requests come in at random times which means that the transcription service should be constantly available.
  - IanCal 4y ago
    Most are, modal is a very different offering where it's $0 when not in use. They have some other very interesting ideas like charging you for CPU time rather than wall time. It's a newer business so I guess you should factor that risk in though.

uncanneyvalley 4y ago

It’s not that your audio is being amplified, it’s that the VAD classifier is poorly tuned. The noise should never even reach the recognition stage. Whisper’s hallucinations are pretty severe, but are improved by adding VAD to its pipeline.