3 ms·

This is really cool. I really enjoyed having this built into my Pixel a few years ago (super useful when you want to watch a video in public but don't have head

This is really cool. I really enjoyed having this built into my Pixel a few years ago (super useful when you want to watch a video in public but don't have headphones). The implementation in Chrome doesn't work that well. It would be great to support both OpenAI's whisper model and a customized vocabulary (I find that a lot of transcription errors are because the set of words I'm exposed to don't necessarily fall inside the most common 50k or 100k words).

NayamAmarshe 4y ago
Is Whisper fast enough on a local machine to be giving live captions? I tried the python module and it takes a long time to process audio files.
- notjulianjaynes 4y ago
  I tried it on a laptop with 8th gen i5 (no dGPU) and transcribing an hour long interview took something like 2+ hours? I didn't time it precisely but it was slower than real time. uploading to otter.ai took a few minutes by comparison. The accuracy was pretty much the same. I have no clue how much faster a consumer device with a dedicated GPU might work though.
- belltaco 4y ago
  Are you on Windows? Chrome has live captions now, not sure if it has it on other OSes. I believe you can play local files in Chrome too.
  - NayamAmarshe 4y ago
    No, I'm on ZorinOS. Takes twice as long as the clip length on small model.
- deet 4y ago
  There are various size options for the models. The choice of which you use trades off accuracy for higher performance. There is also a C++ re-implementation that performs well and can definitely transcribe in realtime on many machines: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
woodson 4y ago
Looks like this isn't based on OpenAI's whisper, but on a model trained with RNNT loss using K2/icefall (https://github.com/k2-fsa/icefall https://github.com/k2-fsa/icefall; K2 is next generation of Kaldi). As such, it should in principle support customized vocabulary and LM for beam search decoding.

NayamAmarshe 4y ago

Is Whisper fast enough on a local machine to be giving live captions? I tried the python module and it takes a long time to process audio files.

notjulianjaynes 4y ago
I tried it on a laptop with 8th gen i5 (no dGPU) and transcribing an hour long interview took something like 2+ hours? I didn't time it precisely but it was slower than real time. uploading to otter.ai took a few minutes by comparison. The accuracy was pretty much the same. I have no clue how much faster a consumer device with a dedicated GPU might work though.
belltaco 4y ago
Are you on Windows? Chrome has live captions now, not sure if it has it on other OSes. I believe you can play local files in Chrome too.
- NayamAmarshe 4y ago
  No, I'm on ZorinOS. Takes twice as long as the clip length on small model.
deet 4y ago
There are various size options for the models. The choice of which you use trades off accuracy for higher performance. There is also a C++ re-implementation that performs well and can definitely transcribe in realtime on many machines: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp

woodson 4y ago

Looks like this isn't based on OpenAI's whisper, but on a model trained with RNNT loss using K2/icefall (https://github.com/k2-fsa/icefall https://github.com/k2-fsa/icefall; K2 is next generation of Kaldi). As such, it should in principle support customized vocabulary and LM for beam search decoding.