3 ms·

Funnily enough, `-small`, `-base` and `-tiny` versions of this would be more exciting to me. `small.en` is the largest of the original whisper models that will

Funnily enough, `-small`, `-base` and `-tiny` versions of this would be more exciting to me. `small.en` is the largest of the original whisper models that will run anywhere near usable speed on a raspberry pi zero 2 with whisper.cpp, and it's still too slow to really bother with for streaming. Anything smaller is too inaccurate for day to day use. If there was a distilled version which had a similar 6x speedup, that would be transformative.

rjwilmsi 3y ago
I understand that, though I think significant speedups can be useful at multiple levels. So for me for example I am using either the base or small model on beam size of 1 with faster-whisper for real-time dictation on a laptop CPU (Rzyen 4500U). The recognition time is just that bit too high when using a larger beam size or is much too high when using the medium model. So if these models offer a decent speed up it means I can either increase beam size or go up a model size which will lead to good improvement in accuracy I think - With real-time dictation I find that small errors are quite annoying to deal with so any improvement in accuracy is really useful. At a larger level, say an exercise to transcribe a back catalogue of audio might need a $1000 GPU with the current model speeds to get the job done in a reasonable time. With models that run 6x faster it might be that a $200 GPU is sufficient. That could be quite a significant saving for a small company or charity etc.
- regularfry 3y ago
  Oh yes, that's absolutely true - faster is better for everyone. It's just that this particular breakpoint would put realtime transcription on a $17 device with an amazing support ecosystem. It's wild. That being said, even with this distillation there's still the aspect that Whisper isn't really designed for streaming. It's fairly simplistic and always deals with 30 second windows. I was expecting there to have been some sort of useful transform you could do to the model to avoid quite so much reprocessing per frame, but other than https://github.com/mit-han-lab/streaming-llm https://github.com/mit-han-lab/streaming-llm (which I'm not even sure directly helps) I haven't noticed anything out there.
abecedarius 3y ago
Another important use is low-latency transcription on a phone, for hard-of-hearing people like me. I've been tempted lately to buy a beefier phone.
- regularfry 3y ago
  I do wonder how low latency can feasibly get. I think there's an interesting barrier to cross with letting speach-to-text correct previously emitted tokens when later context reveals "mishearings" that would help. I know I've seen some systems do that so it's definitely possible, just not something that's readily supported by Whisper as it stands.

rjwilmsi 3y ago

I understand that, though I think significant speedups can be useful at multiple levels. So for me for example I am using either the base or small model on beam size of 1 with faster-whisper for real-time dictation on a laptop CPU (Rzyen 4500U). The recognition time is just that bit too high when using a larger beam size or is much too high when using the medium model. So if these models offer a decent speed up it means I can either increase beam size or go up a model size which will lead to good improvement in accuracy I think - With real-time dictation I find that small errors are quite annoying to deal with so any improvement in accuracy is really useful. At a larger level, say an exercise to transcribe a back catalogue of audio might need a $1000 GPU with the current model speeds to get the job done in a reasonable time. With models that run 6x faster it might be that a $200 GPU is sufficient. That could be quite a significant saving for a small company or charity etc.

regularfry 3y ago
Oh yes, that's absolutely true - faster is better for everyone. It's just that this particular breakpoint would put realtime transcription on a $17 device with an amazing support ecosystem. It's wild. That being said, even with this distillation there's still the aspect that Whisper isn't really designed for streaming. It's fairly simplistic and always deals with 30 second windows. I was expecting there to have been some sort of useful transform you could do to the model to avoid quite so much reprocessing per frame, but other than https://github.com/mit-han-lab/streaming-llm https://github.com/mit-han-lab/streaming-llm (which I'm not even sure directly helps) I haven't noticed anything out there.

abecedarius 3y ago

Another important use is low-latency transcription on a phone, for hard-of-hearing people like me. I've been tempted lately to buy a beefier phone.

regularfry 3y ago
I do wonder how low latency can feasibly get. I think there's an interesting barrier to cross with letting speach-to-text correct previously emitted tokens when later context reveals "mishearings" that would help. I know I've seen some systems do that so it's definitely possible, just not something that's readily supported by Whisper as it stands.