4 ms·

Would you be so kind as to link to a guide for your method or share it in a comment yourself? I installed following the official docs and found it much, much s

by swores 3y ago

Would you be so kind as to link to a guide for your method or share it in a comment yourself? I installed following the official docs and found it much, much slower, although I sadly don't have a 4090, instead a 3080 Ti 12GB (just big enough to load the large whisper model into GPU memory).

modeless 3y ago
I'm running on Linux with a 13900k, 64 GB RAM, and I already have CUDA installed. Install commands directly from the README: pipx install insanely-fast-whisper pipx runpip insanely-fast-whisper install flash-attn --no-build-isolation To transcribe the file: insanely-fast-whisper --flash True --file-name ~/Downloads/podcast_1652_was_jetzt_episode_1289963_update_warum_streiken_sie_schon_wieder_herr_zugchef.mp3 --language german --model-name openai/whisper-large-v3 The file can be downloaded at: https://adswizz.podigee-cdn.net/version/1702050198/media/podcast_1652_was_jetzt_episode_1289963_update_warum_streiken_sie_schon_wieder_herr_zugchef.mp3?awCollectionId=iqd_b64d5d&awEpisodeId=1289963&source=feed&v=1702050198&listeningSessionID=0CD_382_346__8a2f79cfadbaf93f331d0647f8c71d974ffe1ce5 https://adswizz.podigee-cdn.net/version/1702050198/media/pod... I just ran it again and happened to get an even better time, under 7 seconds without loading and 13.08 seconds including loading. In case anyone is curious about the use of Flash Attention, I tried without it and transcription took under 10 seconds, 15.3 including loading.
- owehrens 3y ago
  I just can't get it to work, it errors out with 'NotImplementedError: The model type whisper is not yet supported to be used with BetterTransformer.' Did you happen to run into this problem?
  - modeless 3y ago
    Sorry, I didn't encounter that error. It worked on the first try for me. I have wished many times that the ML community didn't settle on Python for this reason...
- owehrens 3y ago
  Thanks so much again. Got it working. 8 seconds. Nvidia is the king. Updated the blog post.
  - darkteflon 3y ago
    I think insanely-faster-whisper uses batching, so faster-whisper (which doesn’t) might be a fairer comparison for the purposes of your post.
- swores 3y ago
  Thanks! Another question that's only slightly related, but while we're here... Using OAI's paid Whisper API, you can give a text prompt to a) set the tone/style of the transcription and b) teach it technical terms, names etc that it might not be familiar with and should expect in the audio to transcribe. Am I correct that this isn't possible with any released versions of Whisper, or is there a way to do it on my machine that I'm not aware of?
  - modeless 3y ago
    You can definitely do this with the open source version. Many transcription implementations use it to maintain context between the max-30-second chunks Whisper natively supports.
    - swores 3y ago
      I'll try to understand some of how stuff like faster-whisper works when I've got time over the weekend, but I fear it may be too complex for me... I was rather hoping for a guide of just how to either adapt classic whisper usage or adapt one of the optimised ones like faster-whisper (which I've just set up in a docker container but that's used up all the time I've got for playing around right now) to take a text prompt with the audio file.
- sundvor 3y ago
  Cheers, I've been wanting to get into doing something else with my 4090 order than multi monitor simulator gaming, quad screen workstation work - and this will get me kicked off! The 4090 is an absolute beast, runs extremely quiet and simply powers through everything. DCS pushes it to the limit, but the resulting experience is simply stunning. Mine's coupled to a 7800x3d which uses hardly any power at all, absolutely love it.
  - modeless 3y ago
    If you're looking for something easy to try out, try my early demo that hooks Whisper to an LLM and TTS so you can have a real time speech conversation with your local GPU that feels like talking to a person! It's way faster than ChatGPT: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7
    - pixelpoet 3y ago
      This sounds awesome! Will check it out soon

by swores 3y ago

modeless 3y ago

I'm running on Linux with a 13900k, 64 GB RAM, and I already have CUDA installed. Install commands directly from the README: pipx install insanely-fast-whisper pipx runpip insanely-fast-whisper install flash-attn --no-build-isolation To transcribe the file: insanely-fast-whisper --flash True --file-name ~/Downloads/podcast_1652_was_jetzt_episode_1289963_update_warum_streiken_sie_schon_wieder_herr_zugchef.mp3 --language german --model-name openai/whisper-large-v3 The file can be downloaded at: https://adswizz.podigee-cdn.net/version/1702050198/media/podcast_1652_was_jetzt_episode_1289963_update_warum_streiken_sie_schon_wieder_herr_zugchef.mp3?awCollectionId=iqd_b64d5d&awEpisodeId=1289963&source=feed&v=1702050198&listeningSessionID=0CD_382_346__8a2f79cfadbaf93f331d0647f8c71d974ffe1ce5 https://adswizz.podigee-cdn.net/version/1702050198/media/pod... I just ran it again and happened to get an even better time, under 7 seconds without loading and 13.08 seconds including loading. In case anyone is curious about the use of Flash Attention, I tried without it and transcription took under 10 seconds, 15.3 including loading.

owehrens 3y ago
I just can't get it to work, it errors out with 'NotImplementedError: The model type whisper is not yet supported to be used with BetterTransformer.' Did you happen to run into this problem?
- modeless 3y ago
  Sorry, I didn't encounter that error. It worked on the first try for me. I have wished many times that the ML community didn't settle on Python for this reason...
owehrens 3y ago
Thanks so much again. Got it working. 8 seconds. Nvidia is the king. Updated the blog post.
- darkteflon 3y ago
  I think insanely-faster-whisper uses batching, so faster-whisper (which doesn’t) might be a fairer comparison for the purposes of your post.
swores 3y ago
Thanks! Another question that's only slightly related, but while we're here... Using OAI's paid Whisper API, you can give a text prompt to a) set the tone/style of the transcription and b) teach it technical terms, names etc that it might not be familiar with and should expect in the audio to transcribe. Am I correct that this isn't possible with any released versions of Whisper, or is there a way to do it on my machine that I'm not aware of?
- modeless 3y ago
  You can definitely do this with the open source version. Many transcription implementations use it to maintain context between the max-30-second chunks Whisper natively supports.
  - swores 3y ago
    I'll try to understand some of how stuff like faster-whisper works when I've got time over the weekend, but I fear it may be too complex for me... I was rather hoping for a guide of just how to either adapt classic whisper usage or adapt one of the optimised ones like faster-whisper (which I've just set up in a docker container but that's used up all the time I've got for playing around right now) to take a text prompt with the audio file.
sundvor 3y ago
Cheers, I've been wanting to get into doing something else with my 4090 order than multi monitor simulator gaming, quad screen workstation work - and this will get me kicked off! The 4090 is an absolute beast, runs extremely quiet and simply powers through everything. DCS pushes it to the limit, but the resulting experience is simply stunning. Mine's coupled to a 7800x3d which uses hardly any power at all, absolutely love it.
- modeless 3y ago
  If you're looking for something easy to try out, try my early demo that hooks Whisper to an LLM and TTS so you can have a real time speech conversation with your local GPU that feels like talking to a person! It's way faster than ChatGPT: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7
  - pixelpoet 3y ago
    This sounds awesome! Will check it out soon