3 ms·

Whisper V3 is released! https://github.com/openai/whisper/commit/c5d42560760a05584c1c79546a098287e5a771eb https://github.com/openai/whisper/commit/c5d42560760a0

Whisper V3 is released! https://github.com/openai/whisper/commit/c5d42560760a05584c1c79546a098287e5a771eb https://github.com/openai/whisper/commit/c5d42560760a05584c1... Looks like it's just a new checkpoint for the large model. It would be nice to have updates for the smaller models too. But it'll be easy to integrate with anything using Whisper V2. I'm excited to add it to my local voice AI (https://www.microsoft.com/store/apps/9NC624PBFGB7 https://www.microsoft.com/store/apps/9NC624PBFGB7) I assume ChatGPT voice has been using Whisper V3 and I've noticed that it still has the classic Whisper hallucinations ("Thank you for watching!"), so I guess it's an incremental improvement but not revolutionary.

ianbicking 3y ago
Do you also get those hallucinations just on silence? I kind of wonder if they had a bunch of training data of video with transcripts, but some of the video/audio was truncated and the transcript still said the last speech, and so now it thinks silence is just another way of signing off from a TV program. IMHO the bottleneck on voice now is all the infrastructure around it. How do you detect speech starting and stopping? How do you play sound/speech while also being ready for the user to speak? This stuff is necessary, but everything kind of works poorly, and you really need hardware/software integration.
- modeless 3y ago
  You're right, I think that's exactly what happened. Silence is when you get the most hallucinations. But there is a trick supported by some implementations that helps a lot. Whisper does have a special <|nospeech|> token that it predicts for silence. You can look at the probability of that token even when it's not picked during sampling. Hallucinations often have a relatively high probability for the nospeech token compared to actual speech, so that can help filter them out. As for all the surrounding stuff like detecting speech starting and stopping and listening for interruptions while talking, give my voice AI a try. It has a rough first pass at all that stuff, and it needs a lot of work but it's a start and it's fun to play with. Ultimately the answer is end-to-end speech-to-speech models, but you can get pretty far with what we have now in open source!
  - azinman2 3y ago
    So why doesn't the model score that higher then? I'm guessing there's an inherent trade off and they picked/trained it with enough silence vs non-silence?
    - modeless 3y ago
      Low quality training data, almost certainly.
  - ianbicking 3y ago
    This is what you mean? https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 But I don't have Windows :(
    - modeless 3y ago
      I plan to do a Linux build when I have time and release it as a Snap or Flatpak or Appimage or something. But it relies on an Nvidia GPU, so I can't release a macOS version right now. I could probably put something together with llama.cpp and whisper.cpp but it wouldn't be as fast, even on Apple's best. I'll probably do it eventually along with AMD support using ROCm or Vulkan, but I have a whole lot of other things to get to first and very limited time to work on it. Right now my target is people with high end gaming PCs, because they can have a really good experience with the right software but most AI stuff is ridiculously hard to install. My goal is one click install with no required dependencies.
- deleted 3y ago
  [deleted]
- depr 3y ago
  >How do you detect speech starting and stopping? https://github.com/snakers4/silero-vad https://github.com/snakers4/silero-vad
Void_ 3y ago
Too bad they didn't upgrade Whisper API yet. Can't wait to make it available in https://whispermemos.com https://whispermemos.com
- dotancohen 3y ago
  If you add an Android version that I could activate from my lock screen, you'll have another customer.
- henryaj 3y ago
  This looks awesome!
dang 3y ago
Related: OpenAI releases Whisper v3, new generation open source ASR model - https://news.ycombinator.com/item?id=38166965 https://news.ycombinator.com/item?id=38166965
prabhasp 3y ago
Love that Sama literally spent 16 seconds of a 45minute announcement on whisper - https://app.reduct.video/o/eca54fbf9f/p/250fab814f/share/9d98094fe58360389b48/d/faf4965b9b/h/da843bde66 https://app.reduct.video/o/eca54fbf9f/p/250fab814f/share/9d9...
petesergeant 3y ago
Does it have diarisation yet?
- RockRobotRock 3y ago
  Check out WhisperX
  - petesergeant 3y ago
    Thanks!

ianbicking 3y ago

Do you also get those hallucinations just on silence? I kind of wonder if they had a bunch of training data of video with transcripts, but some of the video/audio was truncated and the transcript still said the last speech, and so now it thinks silence is just another way of signing off from a TV program. IMHO the bottleneck on voice now is all the infrastructure around it. How do you detect speech starting and stopping? How do you play sound/speech while also being ready for the user to speak? This stuff is necessary, but everything kind of works poorly, and you really need hardware/software integration.

modeless 3y ago
You're right, I think that's exactly what happened. Silence is when you get the most hallucinations. But there is a trick supported by some implementations that helps a lot. Whisper does have a special <|nospeech|> token that it predicts for silence. You can look at the probability of that token even when it's not picked during sampling. Hallucinations often have a relatively high probability for the nospeech token compared to actual speech, so that can help filter them out. As for all the surrounding stuff like detecting speech starting and stopping and listening for interruptions while talking, give my voice AI a try. It has a rough first pass at all that stuff, and it needs a lot of work but it's a start and it's fun to play with. Ultimately the answer is end-to-end speech-to-speech models, but you can get pretty far with what we have now in open source!
- azinman2 3y ago
  So why doesn't the model score that higher then? I'm guessing there's an inherent trade off and they picked/trained it with enough silence vs non-silence?
  - modeless 3y ago
    Low quality training data, almost certainly.
- ianbicking 3y ago
  This is what you mean? https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 But I don't have Windows :(
  - modeless 3y ago
    I plan to do a Linux build when I have time and release it as a Snap or Flatpak or Appimage or something. But it relies on an Nvidia GPU, so I can't release a macOS version right now. I could probably put something together with llama.cpp and whisper.cpp but it wouldn't be as fast, even on Apple's best. I'll probably do it eventually along with AMD support using ROCm or Vulkan, but I have a whole lot of other things to get to first and very limited time to work on it. Right now my target is people with high end gaming PCs, because they can have a really good experience with the right software but most AI stuff is ridiculously hard to install. My goal is one click install with no required dependencies.
deleted 3y ago
[deleted]
depr 3y ago
>How do you detect speech starting and stopping? https://github.com/snakers4/silero-vad https://github.com/snakers4/silero-vad

Void_ 3y ago

Too bad they didn't upgrade Whisper API yet. Can't wait to make it available in https://whispermemos.com https://whispermemos.com

dotancohen 3y ago
If you add an Android version that I could activate from my lock screen, you'll have another customer.
henryaj 3y ago
This looks awesome!

dang 3y ago

Related: OpenAI releases Whisper v3, new generation open source ASR model - https://news.ycombinator.com/item?id=38166965 https://news.ycombinator.com/item?id=38166965

prabhasp 3y ago

Love that Sama literally spent 16 seconds of a 45minute announcement on whisper - https://app.reduct.video/o/eca54fbf9f/p/250fab814f/share/9d98094fe58360389b48/d/faf4965b9b/h/da843bde66 https://app.reduct.video/o/eca54fbf9f/p/250fab814f/share/9d9...

petesergeant 3y ago

Does it have diarisation yet?

RockRobotRock 3y ago
Check out WhisperX
- petesergeant 3y ago
  Thanks!