3 ms·

Whisper Jax: Transcribe a 1 hour of audio in under 15 seconds

36 points by mysterybox 3y ago

mysterybox 3y ago
Whisper JAX is an optimised implementation of the Whisper model by OpenAI. It runs on JAX with a TPU v4-8 in the backend. Compared to PyTorch on an A100 GPU, it is over 70x faster, making it the fastest Whisper API available.
- __henil 3y ago
  how does it compare to whisper.cpp?
  - neximo64 3y ago
    70 times faster
  - ttt3ts 3y ago
    Not a good comparison. CPU vs TPU. Also, whisper.cpp perf varies a ton depending on hardware. Avx512 or neural engine?
deleted 3y ago
[deleted]
sebnun 3y ago
I've been looking for faster implementations of Whisper, the main drawback with Whisper Jax is that the performance comes from running on Google TPUs, which are much more expensive than GPUs. On "normal" GPUs the fastest implementation I've found is https://github.com/guillaumekln/faster-whisper https://github.com/guillaumekln/faster-whisper. Whisper.cpp works faster on a CPU, especially on Apple Silicon, but still nowhere near the performance you could get on a GPU (understandably). How does Whisper Jax compares to faster-whisper on a GPU?
- ttt3ts 3y ago
  Yea, whisper runs fairly quickly on really old GPUs and leveraging CTranslate2 (underlying tech in your link) you need < 4GB of GPU RAM to run the large model. This might be faster but an order of magnitude more expensive? Also, huge lockin to one provider.
- zznzz 3y ago
  jax and whisper-jax support running on GPUs in addition to TPUs, this particular demo just runs on a TPU.
toomuchtodo 3y ago
Could this be used on device to generate closed captions of video on the fly?
jax_user123 3y ago
Can Whisper-JAX also translate audio streams in real time from X -> Y, both of which are non-English languages? Additionally I have tried Whisper JAX on JupyterHub and for some reason it does not transcribe/translate under 10 seconds for me. In fact, if an audio is 5 minutes long, it would still take 3-4 minutes for transcribing it, although I have followed the similar steps as the ones in the Kaggle notebook, posted by the Author themself. Any ideas/suggestions why this would be happenind would be really helpful. Thank you!

36 points by mysterybox 3y ago

Whisper JAX is an optimised implementation of the Whisper model by OpenAI. It runs on JAX with a TPU v4-8 in the backend. Compared to PyTorch on an A100 GPU, it is over 70x faster, making it the fastest Whisper API available.

__henil 3y ago
how does it compare to whisper.cpp?
- neximo64 3y ago
  70 times faster
- ttt3ts 3y ago
  Not a good comparison. CPU vs TPU. Also, whisper.cpp perf varies a ton depending on hardware. Avx512 or neural engine?

deleted 3y ago

[deleted]

sebnun 3y ago

I've been looking for faster implementations of Whisper, the main drawback with Whisper Jax is that the performance comes from running on Google TPUs, which are much more expensive than GPUs. On "normal" GPUs the fastest implementation I've found is https://github.com/guillaumekln/faster-whisper https://github.com/guillaumekln/faster-whisper. Whisper.cpp works faster on a CPU, especially on Apple Silicon, but still nowhere near the performance you could get on a GPU (understandably). How does Whisper Jax compares to faster-whisper on a GPU?

ttt3ts 3y ago
Yea, whisper runs fairly quickly on really old GPUs and leveraging CTranslate2 (underlying tech in your link) you need < 4GB of GPU RAM to run the large model. This might be faster but an order of magnitude more expensive? Also, huge lockin to one provider.
zznzz 3y ago
jax and whisper-jax support running on GPUs in addition to TPUs, this particular demo just runs on a TPU.

toomuchtodo 3y ago

Could this be used on device to generate closed captions of video on the fly?

jax_user123 3y ago

Can Whisper-JAX also translate audio streams in real time from X -> Y, both of which are non-English languages? Additionally I have tried Whisper JAX on JupyterHub and for some reason it does not transcribe/translate under 10 seconds for me. In fact, if an audio is 5 minutes long, it would still take 3-4 minutes for transcribing it, although I have followed the similar steps as the ones in the Kaggle notebook, posted by the Author themself. Any ideas/suggestions why this would be happenind would be really helpful. Thank you!