4 ms·

Someone else in this thread[0] said Whisper was running at 17x real time for them. So, even a weak machine might be able to do an acceptable approximation of re

Someone else in this thread[0] said Whisper was running at 17x real time for them. So, even a weak machine might be able to do an acceptable approximation of real time with Whisper. Also, I feel like shipping to the cloud and back has been shown to be just as fast as on device transcription in a lot of scenarios. Doing it on device is primarily a benefit for privacy and offline, not necessarily latency. (Although, increasingly powerful smartphone hardware is starting to give the latency edge to local processing.) Siri's dictation has had such terrible accuracy for me (an American English speaker without a particularly strong regional accent) and everyone else I know for so many years that it is just a joke in my family. Google and Microsoft have much higher accuracy in their models. The bar is so low for Siri that I automatically wonder how much Whisper is beating Siri in accuracy... because I assume it has to be better than that. I really wish there was an easy demo for Whisper that I could try out. [0]: https://news.ycombinator.com/item?id=32928207 https://news.ycombinator.com/item?id=32928207

lunixbochs 4y ago
17x realtime on a 3090 I did some basic tests on CPU, the "small" Whisper model is in the ballpark of 0.5x realtime, which is probably not great for interactive use. My models in Talon run closer to 100x realtime on CPU.
- coder543 4y ago
  “CPU” isn’t necessarily the benchmark, though. Most smartphones going back years have ML inference accelerators built in, and both Intel and AMD are starting to build in instructions to accelerate inference. Apple’s M1 and M2 have the same inference accelerator hardware as their phones and tablets. The question is whether this model is a good fit for those inference accelerators, and how well it works there, or how well it works running on the integrated GPUs these devices all have. Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow. I have no experience on the accuracy of Talon, but I’ve heard that most open source models are basically overfit to the test datasets… so their posted accuracy is often misleading. If Whisper is substantially better in the real world, that’s the important thing, but I have no idea if that’s the case.
  - lunixbochs 4y ago
    See https://news.ycombinator.com/item?id=32929029 https://news.ycombinator.com/item?id=32929029 re accuracy, I'm working on a wider comparison. My models are generally more robust than open-source models such as Vosk and Silero, but I'm definitely interested in how my stuff compares to Whisper on difficult held-out data. > Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow. It's not that simple. Many of the mobile ML accelerators are more targeted for conv net image workloads, and current-gen Intel and Apple CPUs have dedicated hardware to accelerate matrix math (which helps quite a bit here, and these instructions were in use in my tests). Also, not sure which model they were using at 17x realtime on the 3090. (If it's one of the smaller models, that bodes even worse for non-3090 performance.) The 3090 is one of the fastest ML inference chips in the world, so it doesn't necessarily set realistic expectations. There are also plenty of optimizations that aren't applied to the code we're testing, but I think it's fairly safe to say the Large model is likely to be slow on anything but a desktop-gpu-class accelerator just due to the sheer parameter size.
  - lunixbochs 4y ago
    Ok, my test harness is ready. My A40 box will be busy until later tonight, but on an NVIDIA A2 [1], this is the batchsize=1 throughput I'm seeing. Common Voice, default Whisper settings, card is staying at 97-100% utilization: tiny.en: ~18 sec/sec base.en: ~14 sec/sec small.en: ~6 sec sec/sec medium.en: ~2.2 sec/sec large: ~1.0 sec/sec (fairly wide variance when ramping up as this is slow to process individual clips) [1] https://www.nvidia.com/en-us/data-center/products/a2/ https://www.nvidia.com/en-us/data-center/products/a2/
    - coder543 4y ago
      Isn’t the A2 much weaker than a 3090? So those results are promising. EDIT: for what it's worth, Nvidia rated the A2 at 18 TFLOPS of FP16, and Apple rates the current A16 Neural Engine at 17 TFLOPS of FP16. I'm sure it's not an "apples to apples" comparison.
      - lunixbochs 4y ago
MacsHeadroom 4y ago
> I really wish there was an easy demo for Whisper that I could try out. Like the colab notebook linked on the official Whisper github project page?
- coder543 4y ago
  Sure, but I did see one linked in another thread here on HN after posting that comment.

lunixbochs 4y ago

17x realtime on a 3090 I did some basic tests on CPU, the "small" Whisper model is in the ballpark of 0.5x realtime, which is probably not great for interactive use. My models in Talon run closer to 100x realtime on CPU.

coder543 4y ago
“CPU” isn’t necessarily the benchmark, though. Most smartphones going back years have ML inference accelerators built in, and both Intel and AMD are starting to build in instructions to accelerate inference. Apple’s M1 and M2 have the same inference accelerator hardware as their phones and tablets. The question is whether this model is a good fit for those inference accelerators, and how well it works there, or how well it works running on the integrated GPUs these devices all have. Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow. I have no experience on the accuracy of Talon, but I’ve heard that most open source models are basically overfit to the test datasets… so their posted accuracy is often misleading. If Whisper is substantially better in the real world, that’s the important thing, but I have no idea if that’s the case.
- lunixbochs 4y ago
  See https://news.ycombinator.com/item?id=32929029 https://news.ycombinator.com/item?id=32929029 re accuracy, I'm working on a wider comparison. My models are generally more robust than open-source models such as Vosk and Silero, but I'm definitely interested in how my stuff compares to Whisper on difficult held-out data. > Brute forcing the model with just traditional CPU instructions is fine, but… obviously going to be pretty slow. It's not that simple. Many of the mobile ML accelerators are more targeted for conv net image workloads, and current-gen Intel and Apple CPUs have dedicated hardware to accelerate matrix math (which helps quite a bit here, and these instructions were in use in my tests). Also, not sure which model they were using at 17x realtime on the 3090. (If it's one of the smaller models, that bodes even worse for non-3090 performance.) The 3090 is one of the fastest ML inference chips in the world, so it doesn't necessarily set realistic expectations. There are also plenty of optimizations that aren't applied to the code we're testing, but I think it's fairly safe to say the Large model is likely to be slow on anything but a desktop-gpu-class accelerator just due to the sheer parameter size.
- lunixbochs 4y ago
  Ok, my test harness is ready. My A40 box will be busy until later tonight, but on an NVIDIA A2 [1], this is the batchsize=1 throughput I'm seeing. Common Voice, default Whisper settings, card is staying at 97-100% utilization: tiny.en: ~18 sec/sec base.en: ~14 sec/sec small.en: ~6 sec sec/sec medium.en: ~2.2 sec/sec large: ~1.0 sec/sec (fairly wide variance when ramping up as this is slow to process individual clips) [1] https://www.nvidia.com/en-us/data-center/products/a2/ https://www.nvidia.com/en-us/data-center/products/a2/
  - coder543 4y ago
    Isn’t the A2 much weaker than a 3090? So those results are promising. EDIT: for what it's worth, Nvidia rated the A2 at 18 TFLOPS of FP16, and Apple rates the current A16 Neural Engine at 17 TFLOPS of FP16. I'm sure it's not an "apples to apples" comparison.
    - lunixbochs 4y ago

MacsHeadroom 4y ago

> I really wish there was an easy demo for Whisper that I could try out. Like the colab notebook linked on the official Whisper github project page?

coder543 4y ago
Sure, but I did see one linked in another thread here on HN after posting that comment.