4 ms·
time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for product
by toebee 1mo ago
time-to-first-audio (TTFA) is critical for realtime voice applications. open source implementations (e.g. vLLM-Omni, SGLang-Omni) are often too slow for production and can have issues with realtime playback if you push for lower latency. we wanted to fix that.
we optimized qwen3-tts, a popular OSS TTS model, to achieve 34 ms p95 TTFA at 10 requests per second on 1 x H100. we open source the implementation and benchmark, as well as a breakdown of how it was done.
github: https://github.com/nari-labs/nari-qwen3-tts https://github.com/nari-labs/nari-qwen3-tts
- bityard 1mo agoHow fast is it on consumer-level hardware?
- toebee 1mo agoWe got a rtx 4090 handling around 10 concurrent requests at 50 ms TTFA after some config changes / adjustment as it doesn’t have FP8. So this 50 ms TTFA thing is very much possible on consumer hardware.
- bt1a 1mo agoI'm going to see how this shakes out on my machine with a few 3090s. I see you all are leveraging some custom cuda kernels, so it may not work out of the box on Ampere (30xx) architecture yeah?
- toebee 1mo agoYep, might need some changes.
- jubilanti 1mo agoI can buy a used car for the price of a used RTX 4090 ($2500-$3000), I wouldn't consider it consumer hardware. Prosumer maybe. Almost no consumer needs 10 concurrent requests. How fast does this run on a 3060 or CPU/iGPU only like an Intel Iris or AMD Navi? Or is your priority more commercial cloud services instead of local self hosted?
- kamranjon 1mo agoHi there! I actually thought your Dia models were amazing and very natural sounding, I haven’t tried qwen 3 tts yet - has your focus shifted away from building your Dia models and shifted more towards hosting and infrastructure?
- toebee 1mo agoHey thank you for your kind words! Yes, we’ve shifted to inference but will also continue doing finetuning etc. on top of open models. Don’t have plans to do pretraining though.
- narrationbox 1mo agoHaven't read the full report yet, just a quick question. Are your numbers for cold start without pre fill or is it after warmed cache?
- toebee 1mo agoWe do graph capture etc at startup (same as vLLM) but this model variant doesn’t require prefix caching - the prefix is just 10 tokens.
- kshmir 1mo agoThanks for your article I set it in my 5090 server and took some ideas to improve the whisper inference I also have. Running asr+llm+tts now :)
- sciencesama 1mo agointeresting
- rullelito 1mo agoFeels like caching the top 1000 most common beginnings would solve this for 99% of cases?
- totetsu 1mo agoI just tried it out on runpod [1] on the lady of shalott. Its got a funny sing song affect https://pastewaves.com/player/cdc4693a-a818-4d34-8566-f0207d502127 https://pastewaves.com/player/cdc4693a-a818-4d34-8566-f0207d... that one took maybe 12.677831s to generate.. DNS: 0.003747s Connect: 0.044571s TLS: 0.115428s First byte: 12.677831s Total curl: 26.268993s HTTP status: 200 Downloaded: 7695404 bytes [1] with this template https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswpjk https://console.runpod.io/hub/template/pahlszv0ul?ref=0xiswp...
- totetsu 1mo agoIs this kind of failure typical? On the websockts mode it ends up turning into gibberish. (after "such that") https://pastewaves.com/player/4786c14f-3370-4b71-ab1b-1aa422216fe7 https://pastewaves.com/player/4786c14f-3370-4b71-ab1b-1aa422... but not in POST mode. https://pastewaves.com/player/aa61a44f-5abb-475c-8df4-b2194480091e https://pastewaves.com/player/aa61a44f-5abb-475c-8df4-b21944...