2 ms·
this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice
by zuzululu 2mo ago
this is cool but for agent scenarios unless an LLM bakes in the speech tokens directly, the latency is lost to inference, and this is what makes openai's voice model so interesting
also sweet spot is under 150ms so the remainder is inference latency turn around, a 50ms turnaround including tts-stt would ofc be the dream
that is "this ai agent is indistinguishably present and sentient" area
- toebee 2mo agoQwen3 TTS has input streaming mode: you can stream LLM output into the speech model. So don’t need to wait for a full sentence. We also implement this websocket variant, and it also runs at sub 50 ms. LLM TTFT is still a big issue, and we might tackle that problem as well.
- zuzululu 2mo agohuh that is crazy fast, demo ?
- toebee 2mo agowill try to record a video soon, in the meantime you can grab a h100 from somewhere like runpod or modal and test it out: docker run --rm --gpus all \ -p 8000:8000 \ -e HF_TOKEN \ -e QWEN3_TTS_PROFILE=ttfa \ -v nari-qwen3-tts-cache:/home/nari/.cache \ ghcr.io/nari-labs/nari-qwen3-tts:latest
- totetsu 2mo agoI tried it out on runpod for about $4 using this template. you might need to override the environment varrs and put in a hugging face token for it to pull model. https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswpjk https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswp...