3 ms·

Thanks! Yes, "realtime multiple" is audio/speech length divided by actual inference time. You got it! The demo video is showing the slowest response times bec

Thanks! Yes, "realtime multiple" is audio/speech length divided by actual inference time. You got it! The demo video is showing the slowest response times because it is using the highest quality/accuracy settings available with Whisper (large-v2, beam 5). Willow devices use medium 1 by default for comparison and those responses are measured in the 500 milliseconds or less range (again depending on speech length) across a wide variety of new and old CUDA hardware. Some sample benchmarks here[0]. Applications using WIS (including Willow) can provide model settings on a per-request basis to balance quality vs latency depending on the task. [0] - https://github.com/toverainc/willow-inference-server#benchmarks https://github.com/toverainc/willow-inference-server#benchma...

morehamish 3y ago
Great stuff. Which video card are you running in the demo video?
- kkielhofner 3y ago
  The fancy: RTX 3090 The not-so-fancy: it's a 10 year old Xeon... The benchmarks for the other GPUs were across a pretty wide range of hardware. The GTX 1060, for example, is in a REALLY old Xeon that doesn't even support AVX... I have hardware arriving this week that I purchased from eBay for $320 with shipping and tax - Dell Precision with i7-7700, 16GB RAM, 512GB SSD[0] and GTX 1070[1]. Seems to be the current "best bang for the buck" if you don't have any of this stuff just sitting around. The idea is because Willow devices are so cost-effective you can buy this hardware to host WIS, HA, other homelab stuff, etc and still come out ahead (even with power) compared to Raspberry Pi: Qty 5 Willows - $270 with power supplies (support wake word, far-field audio, LCD display, speaker, mics, etc) WIS hardware - $320 (or less - it's up to you!) Total cost: $590 Six Raspberry Pis (no wake word, poor audio quality, unusably slow, cumbersome, very DIY): $720 (retail kit with board, SD, LCD display, mic array, speaker, enclosure). MSRP - can't actually be purchased for that. I expect this hardware to do well (returning well below 500ms for Willow speech locally with excellent quality) and I'll be documenting the power consumption optimization work I'm doing over the weekend. [0] - https://www.ebay.com/itm/234908676168 https://www.ebay.com/itm/234908676168 [1] - https://www.ebay.com/itm/115536328587 https://www.ebay.com/itm/115536328587
  - morehamish 3y ago
    Brilliant. Willow looks like a game-changer to me, thanks a million!
    - kkielhofner 3y ago
      Thanks! We believe we have a practical, viable approach to provide an Alexa equivalent (or better) experience with self-hosted privacy and control. Not to mention all of the other stuff you can do with WIS :).

morehamish 3y ago

Great stuff. Which video card are you running in the demo video?

kkielhofner 3y ago
The fancy: RTX 3090 The not-so-fancy: it's a 10 year old Xeon... The benchmarks for the other GPUs were across a pretty wide range of hardware. The GTX 1060, for example, is in a REALLY old Xeon that doesn't even support AVX... I have hardware arriving this week that I purchased from eBay for $320 with shipping and tax - Dell Precision with i7-7700, 16GB RAM, 512GB SSD[0] and GTX 1070[1]. Seems to be the current "best bang for the buck" if you don't have any of this stuff just sitting around. The idea is because Willow devices are so cost-effective you can buy this hardware to host WIS, HA, other homelab stuff, etc and still come out ahead (even with power) compared to Raspberry Pi: Qty 5 Willows - $270 with power supplies (support wake word, far-field audio, LCD display, speaker, mics, etc) WIS hardware - $320 (or less - it's up to you!) Total cost: $590 Six Raspberry Pis (no wake word, poor audio quality, unusably slow, cumbersome, very DIY): $720 (retail kit with board, SD, LCD display, mic array, speaker, enclosure). MSRP - can't actually be purchased for that. I expect this hardware to do well (returning well below 500ms for Willow speech locally with excellent quality) and I'll be documenting the power consumption optimization work I'm doing over the weekend. [0] - https://www.ebay.com/itm/234908676168 https://www.ebay.com/itm/234908676168 [1] - https://www.ebay.com/itm/115536328587 https://www.ebay.com/itm/115536328587
- morehamish 3y ago
  Brilliant. Willow looks like a game-changer to me, thanks a million!
  - kkielhofner 3y ago
    Thanks! We believe we have a practical, viable approach to provide an Alexa equivalent (or better) experience with self-hosted privacy and control. Not to mention all of the other stuff you can do with WIS :).