4 ms·

How do you get Whisper to be fast? Isn't it quite non-realtime?

• TOMDM 3y ago
  The community upgrades to whisper are far faster than real-time, especially if you have a powerful gpu
  • Jach 3y ago
    There's several faster ones out there. I've been using https://github.com/Softcatala/whisper-ctranslate2 https://github.com/Softcatala/whisper-ctranslate2 which includes a nice --live_transcribe flag. It's not as good as running it on a complete file but it's been helpful to get the gist of foreign language live streams.
• wahnfrieden 3y ago
  use whisper-distil, it's like 5-8x faster
• modeless 3y ago
  Great question! Whisper processes audio in 30 second chunks. But on a fast GPU it can finish in only 100 milliseconds or so. So you can run it 10+ times per second and get around 100ms latency. Even better actually because Whisper will predict past the end of the audio sometimes. This is an advantage of running locally. Running whisper this way is inefficient but I have a whole GPU sitting there dedicated to one user, so it's not a problem as long as it is fast enough. It wouldn't work well for a cloud service trying to optimize GPU use. But there are other ways of doing real time speech recognition that could be used there.