5 ms·

Do you also get those hallucinations just on silence? I kind of wonder if they had a bunch of training data of video with transcripts, but some of the video/au

Do you also get those hallucinations just on silence? I kind of wonder if they had a bunch of training data of video with transcripts, but some of the video/audio was truncated and the transcript still said the last speech, and so now it thinks silence is just another way of signing off from a TV program. IMHO the bottleneck on voice now is all the infrastructure around it. How do you detect speech starting and stopping? How do you play sound/speech while also being ready for the user to speak? This stuff is necessary, but everything kind of works poorly, and you really need hardware/software integration.

modeless 3y ago
You're right, I think that's exactly what happened. Silence is when you get the most hallucinations. But there is a trick supported by some implementations that helps a lot. Whisper does have a special <|nospeech|> token that it predicts for silence. You can look at the probability of that token even when it's not picked during sampling. Hallucinations often have a relatively high probability for the nospeech token compared to actual speech, so that can help filter them out. As for all the surrounding stuff like detecting speech starting and stopping and listening for interruptions while talking, give my voice AI a try. It has a rough first pass at all that stuff, and it needs a lot of work but it's a start and it's fun to play with. Ultimately the answer is end-to-end speech-to-speech models, but you can get pretty far with what we have now in open source!
- azinman2 3y ago
  So why doesn't the model score that higher then? I'm guessing there's an inherent trade off and they picked/trained it with enough silence vs non-silence?
  - modeless 3y ago
    Low quality training data, almost certainly.
- ianbicking 3y ago
  This is what you mean? https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 But I don't have Windows :(
  - modeless 3y ago
    I plan to do a Linux build when I have time and release it as a Snap or Flatpak or Appimage or something. But it relies on an Nvidia GPU, so I can't release a macOS version right now. I could probably put something together with llama.cpp and whisper.cpp but it wouldn't be as fast, even on Apple's best. I'll probably do it eventually along with AMD support using ROCm or Vulkan, but I have a whole lot of other things to get to first and very limited time to work on it. Right now my target is people with high end gaming PCs, because they can have a really good experience with the right software but most AI stuff is ridiculously hard to install. My goal is one click install with no required dependencies.
deleted 3y ago
[deleted]
depr 3y ago
>How do you detect speech starting and stopping? https://github.com/snakers4/silero-vad https://github.com/snakers4/silero-vad

modeless 3y ago

You're right, I think that's exactly what happened. Silence is when you get the most hallucinations. But there is a trick supported by some implementations that helps a lot. Whisper does have a special <|nospeech|> token that it predicts for silence. You can look at the probability of that token even when it's not picked during sampling. Hallucinations often have a relatively high probability for the nospeech token compared to actual speech, so that can help filter them out. As for all the surrounding stuff like detecting speech starting and stopping and listening for interruptions while talking, give my voice AI a try. It has a rough first pass at all that stuff, and it needs a lot of work but it's a start and it's fun to play with. Ultimately the answer is end-to-end speech-to-speech models, but you can get pretty far with what we have now in open source!

azinman2 3y ago
So why doesn't the model score that higher then? I'm guessing there's an inherent trade off and they picked/trained it with enough silence vs non-silence?
- modeless 3y ago
  Low quality training data, almost certainly.
ianbicking 3y ago
This is what you mean? https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 But I don't have Windows :(
- modeless 3y ago
  I plan to do a Linux build when I have time and release it as a Snap or Flatpak or Appimage or something. But it relies on an Nvidia GPU, so I can't release a macOS version right now. I could probably put something together with llama.cpp and whisper.cpp but it wouldn't be as fast, even on Apple's best. I'll probably do it eventually along with AMD support using ROCm or Vulkan, but I have a whole lot of other things to get to first and very limited time to work on it. Right now my target is people with high end gaming PCs, because they can have a really good experience with the right software but most AI stuff is ridiculously hard to install. My goal is one click install with no required dependencies.

deleted 3y ago

[deleted]

depr 3y ago

>How do you detect speech starting and stopping? https://github.com/snakers4/silero-vad https://github.com/snakers4/silero-vad