5 ms·
Show HN: Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST
Hey HN!
Willow Inference Server (WIS) is a focused and highly optimized language inference server implementation. Our goal is to "automagically" enable performant, cost-effective self-hosting of released state of the art/best of breed models to enable speech and language tasks:
Primarily targeting CUDA (works on CPU too) with support for low-end (cheap) devices such as the Tesla P4, GTX 1060, and up. Don't worry - it screams on an RTX 4090 too! (See benchmarks on Github).
Memory optimized - all three default Whisper (base, medium, large-v2) models loaded simultaneously with TTS support inside of 6GB VRAM. LLM support defaults to int4 quantization (conversion scripts included). ASR/STT + TTS + Vicuna 13B require roughly 18GB VRAM. Less for 7B, of course!
ASR. Heavy emphasis - Whisper optimized for very high quality as-close-to-real-time-as-possible speech recognition via a variety of means (Willow, WebRTC, POST a file, integration with devices and client applications, etc). Results in hundreds of milliseconds or less for most intended speech tasks. See YouTube WebRTC demo[0].
TTS. Primarily provided for assistant tasks (like Willow!) and visually impaired users.
LLM. Optionally pass input through a provided/configured LLM for question answering, chatbot, and assistant tasks. Currently supports LLaMA deriviates with strong preference for Vicuna (I like 13B). Built in support for quantization to int4 to conserve GPU memory.
Support for a variety of transports. REST, WebRTC, Web Sockets (primarily for LLM).
Performance and memory optimized. Leverages CTranslate2 for Whisper support and AutoGPTQ for LLMs.
Willow support. WIS powers the Tovera hosted best-effort example server Willow users enjoy.
Support for WebRTC - stream audio in real-time from browsers or WebRTC applications to optimize quality and response time. Heavily optimized for long-running sessions using WebRTC audio track management. Leave your session open for days at a time and have self-hosted ASR transcription within hundreds of milliseconds while conserving network bandwidth and CPU!
Support for custom TTS voices. With relatively small audio recordings WIS can create and manage custom TTS voices. See API documentation for more information.
Much like the release of Willow[1] last week this is an early release but we had a great response from HN and are looking forward to hearing what everyone thinks!
[0] - https://www.youtube.com/watch?v=PxCO5eONqSQ https://www.youtube.com/watch?v=PxCO5eONqSQ
[1] - https://github.com/toverainc/willow https://github.com/toverainc/willow
- deleted 3y ago[deleted]
- morehamish 3y agoLooks fantastic! A question re. benchmarks: the "realtime multiple" means "how many times faster the model is processing the audio compared to real time" here? (I.e. the opposite of "real-time factor" sometimes used in speech recognition contexts?) Also bigger beam sizes give better quality I guess? Tnx!
- kkielhofner 3y agoThanks! Yes, "realtime multiple" is audio/speech length divided by actual inference time. You got it! The demo video is showing the slowest response times because it is using the highest quality/accuracy settings available with Whisper (large-v2, beam 5). Willow devices use medium 1 by default for comparison and those responses are measured in the 500 milliseconds or less range (again depending on speech length) across a wide variety of new and old CUDA hardware. Some sample benchmarks here[0]. Applications using WIS (including Willow) can provide model settings on a per-request basis to balance quality vs latency depending on the task. [0] - https://github.com/toverainc/willow-inference-server#benchmarks https://github.com/toverainc/willow-inference-server#benchma...
- morehamish 3y agoGreat stuff. Which video card are you running in the demo video?
- kkielhofner 3y agoThe fancy: RTX 3090 The not-so-fancy: it's a 10 year old Xeon... The benchmarks for the other GPUs were across a pretty wide range of hardware. The GTX 1060, for example, is in a REALLY old Xeon that doesn't even support AVX... I have hardware arriving this week that I purchased from eBay for $320 with shipping and tax - Dell Precision with i7-7700, 16GB RAM, 512GB SSD[0] and GTX 1070[1]. Seems to be the current "best bang for the buck" if you don't have any of this stuff just sitting around. The idea is because Willow devices are so cost-effective you can buy this hardware to host WIS, HA, other homelab stuff, etc and still come out ahead (even with power) compared to Raspberry Pi: Qty 5 Willows - $270 with power supplies (support wake word, far-field audio, LCD display, speaker, mics, etc) WIS hardware - $320 (or less - it's up to you!) Total cost: $590 Six Raspberry Pis (no wake word, poor audio quality, unusably slow, cumbersome, very DIY): $720 (retail kit with board, SD, LCD display, mic array, speaker, enclosure). MSRP - can't actually be purchased for that. I expect this hardware to do well (returning well below 500ms for Willow speech locally with excellent quality) and I'll be documenting the power consumption optimization work I'm doing over the weekend. [0] - https://www.ebay.com/itm/234908676168 https://www.ebay.com/itm/234908676168 [1] - https://www.ebay.com/itm/115536328587 https://www.ebay.com/itm/115536328587