4 ms·

Is there any overhead from Windows (i.e. codec translation) during the live transcription? kinda surprising the latency is so large... ...anyway this is great,

Is there any overhead from Windows (i.e. codec translation) during the live transcription? kinda surprising the latency is so large... ...anyway this is great, will check it out after work!

Const-me 4y ago
That latency is in my code, not in some Windows component. I’m accumulating several seconds of the audio before running the model to transcribe the buffer with these audio samples. The logic is in that method: https://github.com/Const-me/Whisper/blob/15aea5bc/Whisper/Whisper/ContextImpl.capture.cpp#L212-L213 https://github.com/Const-me/Whisper/blob/15aea5bc/Whisper/Wh... How long is “several seconds” controlled by these user-adjustable parameters: https://github.com/Const-me/Whisper/blob/8648d1d5/Whisper/API/MfStructs.h#L27-L28 https://github.com/Const-me/Whisper/blob/8648d1d5/Whisper/AP...
- CrimsonCape 4y ago
  Thanks I understand now. You needed to send buffered audio because the model wasn't handling short snippets well. Do you have a sample audio clip you would like to add to the repo for benchmarking purposes? I'm going to try it on my 3060ti tonight and could compare times...
  - Const-me 4y ago
    I have uploaded two sample clips: https://github.com/Const-me/Whisper/tree/master/SampleClips https://github.com/Const-me/Whisper/tree/master/SampleClips The text files in that folder contain performance data from two computers: a desktop with nVidia 1080Ti, and a laptop with integrated AMD GPU. If you want just a single number, look at the “RunComplete” value in these text files.
    - CrimsonCape 4y ago
      Here are my benchmarks on a 3060Ti, i7 12700k: Columbia Medium EN: 21.56 seconds Columbia Large: 37.5 seconds JFK Medium EN: 1.89 seconds JFK Large: 3.25 seconds Seems like your optimizations for your native hardware are really good!
      - Const-me 4y ago
        Wikipedia says there're two versions of 3060Ti, one has GDDR6 memory with 448 GB/second bandwidth, another one has GDDR6X memory with 608 GB/second bandwidth: https://en.wikipedia.org/wiki/GeForce_30_series#Desktop https://en.wikipedia.org/wiki/GeForce_30_series#Desktop The GDDR5X VRAM in 1080Ti delivers up to 484 GB/second. I wonder whether are you using GDDR6 or 6X version of 3060Ti?
        CrimsonCape 4y ago
        Founder's Edition, so according to this site, https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-3060-ti-founders-edition-gddr6x.b9802 https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-306... it's the 6X version.
        Const-me 4y ago
        Here's another founders edition on that web site, with GDDR6 memory: https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-3060-ti-founders-edition.b8440 https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-306... They have a tool to find out for sure: https://www.techpowerup.com/download/techpowerup-gpu-z/ https://www.techpowerup.com/download/techpowerup-gpu-z/
        CrimsonCape 4y ago
        Yes, the page I quoted is wrong. Founder's Edition is the lower range GDDR6, 8gb.
        Const-me 4y ago

Is there any overhead from Windows (i.e. codec translation) during the live transcription? kinda surprising the latency is so large... ...anyway this is great, will check it out after work!

Const-me 4y ago

That latency is in my code, not in some Windows component. I’m accumulating several seconds of the audio before running the model to transcribe the buffer with these audio samples. The logic is in that method: https://github.com/Const-me/Whisper/blob/15aea5bc/Whisper/Whisper/ContextImpl.capture.cpp#L212-L213 https://github.com/Const-me/Whisper/blob/15aea5bc/Whisper/Wh... How long is “several seconds” controlled by these user-adjustable parameters: https://github.com/Const-me/Whisper/blob/8648d1d5/Whisper/API/MfStructs.h#L27-L28 https://github.com/Const-me/Whisper/blob/8648d1d5/Whisper/AP...

CrimsonCape 4y ago
Thanks I understand now. You needed to send buffered audio because the model wasn't handling short snippets well. Do you have a sample audio clip you would like to add to the repo for benchmarking purposes? I'm going to try it on my 3060ti tonight and could compare times...
- Const-me 4y ago
  I have uploaded two sample clips: https://github.com/Const-me/Whisper/tree/master/SampleClips https://github.com/Const-me/Whisper/tree/master/SampleClips The text files in that folder contain performance data from two computers: a desktop with nVidia 1080Ti, and a laptop with integrated AMD GPU. If you want just a single number, look at the “RunComplete” value in these text files.
  - CrimsonCape 4y ago
    Here are my benchmarks on a 3060Ti, i7 12700k: Columbia Medium EN: 21.56 seconds Columbia Large: 37.5 seconds JFK Medium EN: 1.89 seconds JFK Large: 3.25 seconds Seems like your optimizations for your native hardware are really good!
    - Const-me 4y ago
      Wikipedia says there're two versions of 3060Ti, one has GDDR6 memory with 448 GB/second bandwidth, another one has GDDR6X memory with 608 GB/second bandwidth: https://en.wikipedia.org/wiki/GeForce_30_series#Desktop https://en.wikipedia.org/wiki/GeForce_30_series#Desktop The GDDR5X VRAM in 1080Ti delivers up to 484 GB/second. I wonder whether are you using GDDR6 or 6X version of 3060Ti?
      - CrimsonCape 4y ago
        Founder's Edition, so according to this site, https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-3060-ti-founders-edition-gddr6x.b9802 https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-306... it's the 6X version.
        Const-me 4y ago
        Here's another founders edition on that web site, with GDDR6 memory: https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-3060-ti-founders-edition.b8440 https://www.techpowerup.com/gpu-specs/nvidia-geforce-rtx-306... They have a tool to find out for sure: https://www.techpowerup.com/download/techpowerup-gpu-z/ https://www.techpowerup.com/download/techpowerup-gpu-z/
        CrimsonCape 4y ago
        Yes, the page I quoted is wrong. Founder's Edition is the lower range GDDR6, 8gb.
        Const-me 4y ago