Shameful plug for my project Willow: https://heywillow.io/ https://heywillow.io/ Note that it's important to understand the realities here - short of somethin

Shameful plug for my project Willow: https://heywillow.io/ https://heywillow.io/ Note that it's important to understand the realities here - short of something like an RTX 3090/4090 with LLama and every performance optimization available when it comes to responsiveness and accuracy competitive with commercial solutions this is a big challenge. Even with the potential improvements of this work and optimizations like Ctranslate (used by our Willow Inference Server and faster-whisper) getting sub one second response times like Echo/Alexa more-or-less automatically calls for GPU even with every performance trick available. As I like to say when it comes to ML/AI/speech rec/speech synthesis if you bring a CPU to a GPU fight you're going to lose - and all of the commercial implementations are certainly using GPU/TPU on top of who knows what else they've come up with over the years and their immense spending. To get an idea of how dramatic this is you can see the benchmarks with Ctranslate2/faster-whisper and our Willow Inference Server here[0]. Looking at those real-world numbers even at a claimed 6x performance improvement a mighty Threadripper PRO 5955WX can barely meet this goal with the models needed for voice assistant use cases under real world conditions (medium/large). Throw an LLM in the mix and you're sitting around waiting at least several seconds for a response, even with ridiculous hardware. On anything less than ridiculous hardware (including GPU) that becomes at least 10s of seconds very quickly. At the fundamental level a seven year old $100 used GTX 1070 is approximately 5x faster than a monster CPU like the Threadripper PRO 5955WX - at a fraction of the cost and power. That's just for the first step (speech rec), to get something approaching Alexa-level you're in RTX 3090 territory because performance and VRAM. Amazon has spent hundreds of millions of dollars (minimum) over the better part of a decade developing Echo/Alexa. The open source world has a long way to go to catch up. [0] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...