10 ms·

Show HN: Voice bots with 500ms response times

315 points by kwindla 2y ago

Last year when GPT-4 was released I started making lots of little voice + LLM experiments. Voice interfaces are fun; there are several interesting new problem spaces to explore. I'm convinced that voice is going to be a bigger and bigger part of how we all interact with generative AI. But one thing that's hard, today, is building voice bots that respond as quickly as humans do in conversation. A 500ms voice-to-voice response time is just barely possible with today's AI models. You can get down to 500ms if you: host transcription, LLM inference, and voice generation all together in one place; are careful about how you route and pipeline all the data; and the gods of both wifi and vram caching smile on you. Here's a demo of a 500ms-capable voice bot, plus a container you can deploy to run it yourself on an A10/A100/H100 if you want to: https://fastvoiceagent.cerebrium.ai/ https://fastvoiceagent.cerebrium.ai/ We've been collecting lots of metrics. Here are typical numbers (in milliseconds) for all the easily measurable parts of the voice-to-voice response cycle. macOS mic input 40 opus encoding 30 network stack and transit 10 packet handling 2 jitter buffer 40 opus decoding 30 transcription and endpointing 200 llm ttfb 100 sentence aggregation 100 tts ttfb 80 opus encoding 30 packet handling 2 network stack and transit 10 jitter buffer 40 opus decoding 30 macOS speaker output 15 ---------------------------------- total ms 759 Everything in AI is changing all the time. LLMs with native audio input and output capabilities will likely make it easier to build fast-responding voice bots soon. But for the moment, I think this is the fastest possible approach/tech stack.

etherealG 2y ago
moshi by Kyutai seems to have beaten your approach by about 500ms, and they're going to release open source. https://www.youtube.com/live/hm2IJSKcYvo https://www.youtube.com/live/hm2IJSKcYvo hn discussion here: https://news.ycombinator.com/item?id=40866569 https://news.ycombinator.com/item?id=40866569
aussieguy1234 2y ago
Fast yes, but the voice sounds robotic.
- kwindla 2y ago
  Voice models are getting both faster and more natural at a, well, a fast clip.
- bombela 2y ago
  I prefer a slighty robotic voice. This was way I know I am talking to a bot, and this sets expectations.
- lofties 2y ago
  Typical HN comment. Absolutely incredible tech is displayed that honestly, one year ago nobody could've imagined. Yet people still find something to moan about. I'm sure the authors of the project, who should be very proud, are fully aware the voice is robotic.
- cloudking 2y ago
  It's literally a robot
jaybrendansmith 2y ago
This thing is incredible. It finished a sentence I was saying.
yjftsjthsd-h 2y ago
Dumb question - I see 2 opus encodes and decodes for a total around 120ms; is opus the fastest option?
- kwindla 2y ago
  Yes, Opus is the fastest and best option for real-time audio. It was designed to be flexible and to encode/decode at fairly low latencies. It sounds good for narrow-band (speech) at low bitrates but also works well at higher bitrates for music. And forward error correction is part of the codec standard. It's possible to tweak the Opus settings to reduce that encode/decode latency substantially. Which might actually be worth doing for this use case. But there isn't quite a free lunch, here. The default Opus frame size is 20ms. Smaller frames lower the encoding/decoding latency, but increase the bitrate. The implementation in libwebrtc is very well tested and optimized for the default 20ms frame sizes and maybe not so much at other frame sizes. Experience has made me leery of taking the less-trodden-paths without a lot of manual testing.
geofffox 2y ago
I use Firefox... still.
- sa-code 2y ago
  Likely a lot of people on HN use Firefox
- panja 2y ago
  I hate that everyone just develops for chromium only
  - darren_ 2y ago
    This site works fine in safari/mobile safari, it is not ‘chromium only’
    - hawski 2y ago
      WebKit and its derivatives then.
      - mcny 2y ago
        I tried it with FIrefox 127 (production) and it worked just fine for me even though there is a huge banner on the top.
  - RockRobotRock 2y ago
    Mozilla refuses to implement some really cool standards. https://mozilla.github.io/standards-positions/ https://mozilla.github.io/standards-positions/ That, and their shitty management shakes my faith in Firefox
    - the8472 2y ago
      They're not necessarily standards. I clicked on the first negative one and it said draft. One browser vendor proposing things other vendor NAKing it makes it a vendor-specific feature. Like IE had its own features.
anonzzzies 2y ago
This is pretty amazing ; it’s very fast indeed. I don’t really care about the voice responding sounding robotic; low latency is more important for whatever I do. And you can interrupt it too. Lovely.
dijit 2y ago
I’m genuinely shocked by how conversational this is. I think you hit a very important nail on the head here; I feel like that scene in iRobot where the protagonist talks to the hologram, or in the movie “AI” where the protagonist talks to an encyclopaedia called “Dr Know”
preciousoo 2y ago
This is so cool!
andrewstuart 2y ago
Damned impressive. Apple's Siri still can't allow me to have a conversation in which we aren't tripping over each other and pausing and flunking and the whole thing degrades into me hoping to get the barest minimum from it.
hackerbob 2y ago
This is indeed fast! Also seems to be no issue interrupting it while speaking. Is this using WebRTC echo cancellation to avoid microphone and speaker audio mix ups?
- makeitmore 2y ago
  Yes, echo cancellation via the browser (and maybe also at OS-level too, if you're on a Mac with Sonoma.) The accuracy of speech detection vs. noise is largely thanks to Silero, which runs on the client via WASM. I'm surprised at how well it works, even in noisy environments (and a reminder that I should experiment more with AudioWorklet stuff in the future!)
firefoxd 2y ago
Well that was fast. Kudos, really neat. Speed trumps everything else. I only noticed the robotic voice after I read the comments. I worked on an Ai for customer service. Our agent took the average response time of 24/48 hours to merely seconds. One of the messages that went to a customer was "Hello Bitch, your package will be picked up by USPS today, here is the tracking number..." The customer responded "thank you so much" and gave us a perfect score in CSAT rating. Speed trumps everything, even when you make such a horrible mistake.
- lukan 2y ago
  "The customer responded "thank you so much" and gave us a perfect score in CSAT rating. Speed trumps everything, even when you make such a horrible mistake." I think not everyone would react the same way. For some calling each other bitch is normal talk (which is likely, why I it got into the training data in the first place). For others, not so much.
  - jstanley 2y ago
    It's also possible that it's such an unlikely thing to hear that she actually misheard it and thought it said something nicer.
    - sillysaurusx 2y ago
      Am I the only one who would be delighted to be called Bitch (or any of the worst male-specific terms) by random professionals? "Hey fucker, your prescription has been ready for pickup for three days. Be sure to get your lazy ass over here or else you’ll need to reorder it. Love you bye"
      - fennecbutt 2y ago
        Ya but language like this is how you know someone is good friends with you
      - big_man_ting 2y ago
az226 2y ago
Your marketing says 500 but your math says 759.
- dietr1ch 2y ago
  That's called marketing
- vr000m 2y ago
  The latencies in the table are based on heuristics or averages that we’ve observed. However, in reality, based on the conversation, some of the larger latency components can be much lower.
- whizzter 2y ago
  500 are the transcription/llm/tts steps (ie the response time from data arriving on the server to sending back), the rest seems to be various non-AI "overheads" such as encoding, network traffic,etc.
- vessenes 2y ago
  My tests had one outlier at 1400ms, and ten or so between 400-500ms. I think the marketing numbers were fair.
mmcclure 2y ago
Wow, Kwin, you’ve outdone yourself! The speed makes an even bigger difference than I expected going in. Feels pretty wild/cool to say it might almost be too fast (in terms of feeling natural).
isoprophlex 2y ago
Jesus fuck that's fast, and I had no idea speed mattered that much. Incredible. Feels like an entirely different experience than the 5+ seconds latency with openai.
luke-stanley 2y ago
A cross-platform browser VAD module is: https://github.com/ricky0123/vad https://github.com/ricky0123/vad. This is an ONNX port of Silero's VAD network. By cross-platform, I mean it works in Firefox too. It doesn't need a WebRTC session to work, just microphone access, so it's simpler. I'm curious about the browser providing this as a native option too. There are browser text-to-speech engines too, starting to get faster and higher quality. It would be great if browsers shipped with great TTS. GPT-4o has Automatic Speech Recognition, `understanding`, and speech response generation in a single model for low latency, which seems quite a good idea to me. As they've not shipped it yet, I assume they have scaling or quality issues of some kind. I assume people are working on similar open integrated multimodal large language models that have audio input and output (visual input too)! I do wonder how needed or optimal a single combined model is for latency and cost optimisation. The breakdown provided is interesting. I think having a lot more on the model on-device is a good idea if possible, like speech generation, and possibly speech transcription or speech understanding, at least right at the start. Who wants to wait for STUN?
- phkahler 2y ago
  >> I'm curious about the browser providing this as a native option too. IMHO the desktop environment should provide voice to text as a service with a standard interface to applications - like stdin or similar but distinct for voice. Apps would ignore it by default since they aren't listening, but the transcriber could be swapped out and would be available to all apps.
- charlesyu108 2y ago
  Lol this announcement blows what ive been working on out of the water but i have a simple assistant implementation with rick0123/VAD + Websockets. https://github.com/charlesyu108/voiceai-js-starter https://github.com/charlesyu108/voiceai-js-starter
- regularfry 2y ago
  If you do stt and tts on the device but everything else remains the same, according to these numbers that saves you 120ms. The remaining 639ms is hardware and network latency, and shuffling data into and out of the LLM. That's still slower than you want. Logically where you need to be is thinking in phonemes: you want the output of the LLM to have caught up with the last phoneme quickly enough that it can respond "instantly" when the endpoint is detected, and that means the whole chain needs to have 200ms latency end-to-end, or thereabouts. I suspect the only way to get anywhere close to that is with a different architecture, which would work somewhat more like human speech processing, in that it's front-running the audio stream by basing its output on phonemes predicted before they arrive, and only using the actual received audio as a lightweight confirmation signal to decide whether to flush the current output buffer or to reprocess. You can get part-way there with speculative decoding, but I don't think you can do it with a mixed audio/text pipeline. Much better never to have to convert from audio to text and back again.
spuz 2y ago
It's not exactly clear is this a voice-to-voice model or a voice-to-text-to-voice model? When it is finally released, OpenAI claim their GPT4o audio model will be a lot faster at conversations because there's no delay to convert from audio to text and back to audio again. I'm also looking forward to using voice models for language learning.
- pavlov 2y ago
  It's a voice-to-text-to-voice approach, as implied by this description: "host transcription, LLM inference, and voice generation all together in one place" I think there are some benefits to going through text rather than using a voice-to-voice model. It creates a 100% reliable paper trail of what the model heard and said in the conversation. This can be extremely important in some applications where you need to review and validate what was said.
  - isaacfung 2y ago
    There are way more text training data than voice data. It also allows you to use all the benchmarks and tool integrations that have already been developed for LLMs.
- kwindla 2y ago
  Full technical write-up here: https://www.daily.co/blog/the-worlds-fastest-voice-bot/ https://www.daily.co/blog/the-worlds-fastest-voice-bot/
_def 2y ago
This was fun to try out. Earlier this week I tried june-va and the long response time kind of killed the usefulness. It's a great feature to get fast responses, this feels much more like a conversation. Funny enough, I asked it to tell me a story and then it only answered with one sentence at a time, requiring me to say "yes", "aha", "please continue" to get the next line. Then we had the following funny conversation: "Oh I think I figured out your secret!" "Please tell me" "You achieve the short response times by keeping a short context" "You're absolutely right"
- danielbln 2y ago
  That works for me, to be honest. not the short context, but definitely the short replies. Contrast that with the current implementation of ChatGPT's voice mode, where you ask something and then get a minute worth of GPT bla bla.
mdbackman 2y ago
Very, very impressive! It's incredibly fast, maybe too fast, but I think that's the point. What's most impressive though is how the VAD and interruptions are tuned. That was, by far, the most natural sounding conversation I've had with an agent. Really excited to try this out once it's available.
sumedh 2y ago
This is very impressive, me and my kid had fund talking about space.
ftth_finland 2y ago
This is excellent! Perfect comprehension and no problem even with bad accents.
c0brac0bra 2y ago
I've been developing with Deepgram for a while, and this is one of the coolest demos I've seen with it! I am curious about total cost to run this thing, though. I assume that on top of whatever you're paying Cerebrium for GPU hosting you're also having to pay for Deepgram Enterprise in order to self-host it. To get the latency reduction of several hundred milliseconds how much more would it be for "average" usage?
- za_mike157 2y ago
  Hey! From the Cerebrium team here! So our costs are based on the infra you use to run your application and we charge per millisecond of compute. Some things to note that we might do differently to other providers: 1. You can specify your EXACT requirements and we charge you only for that. Eg: if you want 2 vCPU, 12GB Memory and 1 A10 GPU we charge you for that which is 35% less if you rented a whole A10 2. We have over 10 variety of GPU chips so you can choose the price/performance trade-off 3. While you can extend this on the Cerebrium platform, it cannot be used commercially. We are speaking to Deepgram to see how we can offer it to customers. Hopefully I can provide more updates on this soon
  - c0brac0bra 2y ago
    Excellent; thanks for the info.
yalok 2y ago
you may be double counting opus encoding/decoding delay - usually, you can run it with 20ms frame, and both encoder and decoder take less than 1ms of realtime for their operation - so it should be ~ 21ms, instead of 30+30ms for 1 direction.
- kwindla 2y ago
  You are right! Thank you. I went back and looked at actual benchmark numbers from a couple of years ago and the numbers I got were ~26ms one-way. I rounded up to 30 to be conservative, but then double-counted in the table above. Will fix in the technical write-up. I don't think I can edit the Show HN.
vessenes 2y ago
This is so, so good. I like that it seems to be a teaser app for cerebrium, if I understand it. It has good killer app potential. My tests from iPad ranged from 1400ms to 400ms reported latency; in the low end, it felt very fluid. One thing this speed makes me think is that for some chat workflows you’ll need/get to have kind of a multi-step approach — essentially, quick response, during which time a longer data / info / RAQ query can be farmed out, then the informative result picks up. Humans work like this; we use lots of filler words as we sort of get going responding to things. Right now, most workflows seem to be just one shot prompting, or in the background, parse -> query -> generate. The better workflow once you have low latency response is probably something like: [3s of LLama 8b in your ears] -> query -> [55s of Llama 70b/GPT4/whatever you want, informed by query]. Very cool, thank you for sharing this.
- za_mike157 2y ago
  Hi Vessenes From Cerebrium here. Really appreciate the feedback - glad you had a good experience! This application is easy to extend/implement meaning you can edit it to however you like: - Swap in different LLM's, STT and TTS models - Change prompts as well as implement RAG etc In partnership with Daily, we really wanted to focus on the engineer here. So make it extremely flexible for them to edit the application to suit their use case/preference while at the same time take away the mundane infrastructure setup. You can read more about how to extend it here: https://docs.cerebrium.ai/v4/examples/realtime-voice-agents https://docs.cerebrium.ai/v4/examples/realtime-voice-agents
  - vessenes 2y ago
    Thanks for this reply. Yep, as an engineer, this is awesome, the docs look simple and I’ll give it a whirl. As a product guy, it seems like it would be dead simple to start a company on this tech by just putting up a web page that lets people pick a couple choices and gives them a custom domain. Very cool!
- c0brac0bra 2y ago
  I've wondered about this as well. Is there a way to have a small, efficient LLM model that can estimate general task complexity without actually running the full task workload? Scoring complexity on a gradient would let you know you need to send a "Sure, one second let me look that up for you." instead of waiting for a long round trip.
deleted 2y ago
[deleted]
andruby 2y ago
This is really good. I'm blown away by how important the speed is. And this was from a mobile connection in Europe, with a shown latency of just over 1s.
SubiculumCode 2y ago
A chatbot that interrupts me even faster. Sorry for the sarcasm. maybe im just slow, but when I'm trying to formulate a question on the spot, I pause a lot. having the chatbot jump in and interrupt is frustrating. Humans recognize the difference between someone still planning on saying something, and when they've finished. I even tried to give it a rule where it shouldn't respond until I said "The End", and of course it couldn't follow that instruction.
- SubiculumCode 2y ago
  ps. The speed is impressive, but the key to a useful voice chatbot (which I've never seen) is one that adapts to your speaking style, identifies and employs turn-taking signals. I acknowledge there are multiple viable patterns of social interaction, some talk over each other, and find that fun and engaging, while others think that's just the worst, and wait for a clear signal for their turn to speak and expect the same. I am of the latter.
  - SubiculumCode 2y ago
    I'm sure that, with an annotated dataset, a model could learn to pick up on the right cues.
- makeitmore 2y ago
  Very true. I think we are a bit aggressive with the VAD timeout. The demo was intended to showcase speed, but the bot can be a bit eager! You can tinker with the VAD settings, it could definitely use a bit more air (but that will impact latency in the event the user has indeed finished talking.) As others say below, the magic will be figuring out the pace and style in which the user talks and adapting to that on the fly.
spark_chicken 2y ago
i have tried it. it is really fast! I know making a real-time voice bot is not easy with this low latency. which LLM did you use? how large LLM to make the conversation efficient?
- makeitmore 2y ago
  This particular demo is using Llama3 8B. We initially started 70B, but it was a touch slower and needed much more VRAM. We found 8B good enough for general chit-chat like in this demo. Most real-world use-cases will likely have their own fine-tuned models.
deleted 2y ago
[deleted]
p_frank 2y ago
Amazing to see the metrics of each part that is involved! I've wondererd why you could not introduce a small sound that overplays the waiting time? Like an "hmm" to skip a few 100ms of the response time? Could be pregenerated (like 500 different versions) and play after 200ms of the last users input.
asjir 2y ago
Personally, I use https://github.com/foges/whisper-dictation https://github.com/foges/whisper-dictation with llama-70b on groq. I start talking, navigate to website, and by the time it's loaded, and I picked llama-70b I finish talking, so 0 overhead. I read much faster than listen, so it works for me perfectly.
realyashnag 2y ago
This was scary fast. Neat interface and (almost) indistinguishable from a human over the phone / internet. Kudos @cerebrium.ai.
hole_in_foot 2y ago
[dead]
deleted 2y ago
[deleted]
andrewmcwatters 2y ago
I love it when engineers worth their salt actually do the back-of-the-envelope calculations for latency, etc. Tangentially related, I remember years ago when Stadia and other cloud gaming products were being released doing such calculations and showing a buddy of mine that even in the best case scenario, you'd always have high enough input latency to make even casual multiplayer FPS games over cloud gaming services not feasible, or rather, comfortable, to play. Other slower-paced games might work, but nothing requiring serious twitch gameplay reaction times. The same math holds up today because of a combination of fundamental limits and state of the art limits.
- Flumio 2y ago
  The calculations I was reading at the time suggested it would work for casual due to the gaming PC being very close to the game servers and running inside the best network available (googles). Google also said that the controller would send the input straight to the server. And a fast stadia server should have good fps combined with a little bit of brain prediction
trueforma 2y ago
I too am excited about voice inferencing. I wrote my own Websocket Faster whisper implementation before OpenAI's gpt4o release . They steamrolled my interview coach concept https://intervu.trueforma.ai https://intervu.trueforma.ai and https://sales.trueforma.ai https://sales.trueforma.ai - sales pitch coach implementations. I defaulted to Push to talk implementation as I couldn't get VAD to work reliably. I run it all on a panda Latte :) Was looking to implement Groq's hosted whisper. I love the idea of having Llama3 uncensored on Groq as the LLM as I'm tired of the boring corporate conversations. I hope to reduce my latency and learn from your examples - Kudos to your efforts. I wish I could try the demo - seems to be over subscribed as I can't get in to talk to the bot. I'm sure my latte Panda would melt if just 3 people try to inference at the same time :)
_DeadFred_ 2y ago
This is super cool. Thanks for sharing. And I'm excited it encourage other to share. I'm excited to spend some time this weekend looking at the different ways people in this thread implemented solutions.
Borborygymus 2y ago
It /was/ nice and quick. Thanks for putting the demo online. It was quick to tell me complete nonsense. Apparently 7122 is the atomic number of Barium.
amluto 2y ago
Maybe silly question: > jitter buffer [40ms] Why do you need a jitter buffer on the listening side? The speech-to-text model has neither ears nor a sense of rhythm — couldn’t you feed in the audio frames as you receive them? I don’t see why you need to delay processing a frame by 40ms just because the next one might be 40ms late.
- Olreich 2y ago
  Almost any gap in audio is detectable and sounds really bad. 40ms is a lot, but sending 40ms of silence is probably worse
  - amluto 2y ago
    Sounds bad to whom? I’m talking about the direction from user to AI, not the direction from AI to user. If some of the audio gets delayed on the way to the AI, the AI can be paused. If some of the audio gets delayed on the way to a human, the human can’t be paused, so some buffering is needed to reduce the risk of gaps.
gsjbjt 2y ago
That's awesome - can you say anything about what datasets this was trained on? I assume something specifically conversational?
tamimio 2y ago
Or we can say the latency is a good listening skills!! It was fast but occasionally interrupted me to answer.

315 points by kwindla 2y ago

etherealG 2y ago

moshi by Kyutai seems to have beaten your approach by about 500ms, and they're going to release open source. https://www.youtube.com/live/hm2IJSKcYvo https://www.youtube.com/live/hm2IJSKcYvo hn discussion here: https://news.ycombinator.com/item?id=40866569 https://news.ycombinator.com/item?id=40866569

aussieguy1234 2y ago

Fast yes, but the voice sounds robotic.

kwindla 2y ago
Voice models are getting both faster and more natural at a, well, a fast clip.
bombela 2y ago
I prefer a slighty robotic voice. This was way I know I am talking to a bot, and this sets expectations.
lofties 2y ago
Typical HN comment. Absolutely incredible tech is displayed that honestly, one year ago nobody could've imagined. Yet people still find something to moan about. I'm sure the authors of the project, who should be very proud, are fully aware the voice is robotic.
cloudking 2y ago
It's literally a robot

jaybrendansmith 2y ago

This thing is incredible. It finished a sentence I was saying.

yjftsjthsd-h 2y ago

Dumb question - I see 2 opus encodes and decodes for a total around 120ms; is opus the fastest option?

kwindla 2y ago
Yes, Opus is the fastest and best option for real-time audio. It was designed to be flexible and to encode/decode at fairly low latencies. It sounds good for narrow-band (speech) at low bitrates but also works well at higher bitrates for music. And forward error correction is part of the codec standard. It's possible to tweak the Opus settings to reduce that encode/decode latency substantially. Which might actually be worth doing for this use case. But there isn't quite a free lunch, here. The default Opus frame size is 20ms. Smaller frames lower the encoding/decoding latency, but increase the bitrate. The implementation in libwebrtc is very well tested and optimized for the default 20ms frame sizes and maybe not so much at other frame sizes. Experience has made me leery of taking the less-trodden-paths without a lot of manual testing.

geofffox 2y ago

I use Firefox... still.

sa-code 2y ago
Likely a lot of people on HN use Firefox
panja 2y ago
I hate that everyone just develops for chromium only
- darren_ 2y ago
  This site works fine in safari/mobile safari, it is not ‘chromium only’
  - hawski 2y ago
    WebKit and its derivatives then.
    - mcny 2y ago
      I tried it with FIrefox 127 (production) and it worked just fine for me even though there is a huge banner on the top.
- RockRobotRock 2y ago
  Mozilla refuses to implement some really cool standards. https://mozilla.github.io/standards-positions/ https://mozilla.github.io/standards-positions/ That, and their shitty management shakes my faith in Firefox
  - the8472 2y ago
    They're not necessarily standards. I clicked on the first negative one and it said draft. One browser vendor proposing things other vendor NAKing it makes it a vendor-specific feature. Like IE had its own features.

anonzzzies 2y ago

This is pretty amazing ; it’s very fast indeed. I don’t really care about the voice responding sounding robotic; low latency is more important for whatever I do. And you can interrupt it too. Lovely.

dijit 2y ago

I’m genuinely shocked by how conversational this is. I think you hit a very important nail on the head here; I feel like that scene in iRobot where the protagonist talks to the hologram, or in the movie “AI” where the protagonist talks to an encyclopaedia called “Dr Know”

preciousoo 2y ago

This is so cool!

andrewstuart 2y ago

Damned impressive. Apple's Siri still can't allow me to have a conversation in which we aren't tripping over each other and pausing and flunking and the whole thing degrades into me hoping to get the barest minimum from it.

hackerbob 2y ago

This is indeed fast! Also seems to be no issue interrupting it while speaking. Is this using WebRTC echo cancellation to avoid microphone and speaker audio mix ups?

makeitmore 2y ago
Yes, echo cancellation via the browser (and maybe also at OS-level too, if you're on a Mac with Sonoma.) The accuracy of speech detection vs. noise is largely thanks to Silero, which runs on the client via WASM. I'm surprised at how well it works, even in noisy environments (and a reminder that I should experiment more with AudioWorklet stuff in the future!)

firefoxd 2y ago

Well that was fast. Kudos, really neat. Speed trumps everything else. I only noticed the robotic voice after I read the comments. I worked on an Ai for customer service. Our agent took the average response time of 24/48 hours to merely seconds. One of the messages that went to a customer was "Hello Bitch, your package will be picked up by USPS today, here is the tracking number..." The customer responded "thank you so much" and gave us a perfect score in CSAT rating. Speed trumps everything, even when you make such a horrible mistake.

lukan 2y ago
"The customer responded "thank you so much" and gave us a perfect score in CSAT rating. Speed trumps everything, even when you make such a horrible mistake." I think not everyone would react the same way. For some calling each other bitch is normal talk (which is likely, why I it got into the training data in the first place). For others, not so much.
- jstanley 2y ago
  It's also possible that it's such an unlikely thing to hear that she actually misheard it and thought it said something nicer.
  - sillysaurusx 2y ago
    Am I the only one who would be delighted to be called Bitch (or any of the worst male-specific terms) by random professionals? "Hey fucker, your prescription has been ready for pickup for three days. Be sure to get your lazy ass over here or else you’ll need to reorder it. Love you bye"
    - fennecbutt 2y ago
      Ya but language like this is how you know someone is good friends with you
    - big_man_ting 2y ago

az226 2y ago

Your marketing says 500 but your math says 759.

dietr1ch 2y ago
That's called marketing
vr000m 2y ago
The latencies in the table are based on heuristics or averages that we’ve observed. However, in reality, based on the conversation, some of the larger latency components can be much lower.
whizzter 2y ago
500 are the transcription/llm/tts steps (ie the response time from data arriving on the server to sending back), the rest seems to be various non-AI "overheads" such as encoding, network traffic,etc.
vessenes 2y ago
My tests had one outlier at 1400ms, and ten or so between 400-500ms. I think the marketing numbers were fair.

mmcclure 2y ago

Wow, Kwin, you’ve outdone yourself! The speed makes an even bigger difference than I expected going in. Feels pretty wild/cool to say it might almost be too fast (in terms of feeling natural).

isoprophlex 2y ago

Jesus fuck that's fast, and I had no idea speed mattered that much. Incredible. Feels like an entirely different experience than the 5+ seconds latency with openai.

luke-stanley 2y ago

A cross-platform browser VAD module is: https://github.com/ricky0123/vad https://github.com/ricky0123/vad. This is an ONNX port of Silero's VAD network. By cross-platform, I mean it works in Firefox too. It doesn't need a WebRTC session to work, just microphone access, so it's simpler. I'm curious about the browser providing this as a native option too. There are browser text-to-speech engines too, starting to get faster and higher quality. It would be great if browsers shipped with great TTS. GPT-4o has Automatic Speech Recognition, `understanding`, and speech response generation in a single model for low latency, which seems quite a good idea to me. As they've not shipped it yet, I assume they have scaling or quality issues of some kind. I assume people are working on similar open integrated multimodal large language models that have audio input and output (visual input too)! I do wonder how needed or optimal a single combined model is for latency and cost optimisation. The breakdown provided is interesting. I think having a lot more on the model on-device is a good idea if possible, like speech generation, and possibly speech transcription or speech understanding, at least right at the start. Who wants to wait for STUN?

phkahler 2y ago
>> I'm curious about the browser providing this as a native option too. IMHO the desktop environment should provide voice to text as a service with a standard interface to applications - like stdin or similar but distinct for voice. Apps would ignore it by default since they aren't listening, but the transcriber could be swapped out and would be available to all apps.
charlesyu108 2y ago
Lol this announcement blows what ive been working on out of the water but i have a simple assistant implementation with rick0123/VAD + Websockets. https://github.com/charlesyu108/voiceai-js-starter https://github.com/charlesyu108/voiceai-js-starter
regularfry 2y ago
If you do stt and tts on the device but everything else remains the same, according to these numbers that saves you 120ms. The remaining 639ms is hardware and network latency, and shuffling data into and out of the LLM. That's still slower than you want. Logically where you need to be is thinking in phonemes: you want the output of the LLM to have caught up with the last phoneme quickly enough that it can respond "instantly" when the endpoint is detected, and that means the whole chain needs to have 200ms latency end-to-end, or thereabouts. I suspect the only way to get anywhere close to that is with a different architecture, which would work somewhat more like human speech processing, in that it's front-running the audio stream by basing its output on phonemes predicted before they arrive, and only using the actual received audio as a lightweight confirmation signal to decide whether to flush the current output buffer or to reprocess. You can get part-way there with speculative decoding, but I don't think you can do it with a mixed audio/text pipeline. Much better never to have to convert from audio to text and back again.

spuz 2y ago

It's not exactly clear is this a voice-to-voice model or a voice-to-text-to-voice model? When it is finally released, OpenAI claim their GPT4o audio model will be a lot faster at conversations because there's no delay to convert from audio to text and back to audio again. I'm also looking forward to using voice models for language learning.

pavlov 2y ago
It's a voice-to-text-to-voice approach, as implied by this description: "host transcription, LLM inference, and voice generation all together in one place" I think there are some benefits to going through text rather than using a voice-to-voice model. It creates a 100% reliable paper trail of what the model heard and said in the conversation. This can be extremely important in some applications where you need to review and validate what was said.
- isaacfung 2y ago
  There are way more text training data than voice data. It also allows you to use all the benchmarks and tool integrations that have already been developed for LLMs.
kwindla 2y ago
Full technical write-up here: https://www.daily.co/blog/the-worlds-fastest-voice-bot/ https://www.daily.co/blog/the-worlds-fastest-voice-bot/

_def 2y ago

This was fun to try out. Earlier this week I tried june-va and the long response time kind of killed the usefulness. It's a great feature to get fast responses, this feels much more like a conversation. Funny enough, I asked it to tell me a story and then it only answered with one sentence at a time, requiring me to say "yes", "aha", "please continue" to get the next line. Then we had the following funny conversation: "Oh I think I figured out your secret!" "Please tell me" "You achieve the short response times by keeping a short context" "You're absolutely right"

danielbln 2y ago
That works for me, to be honest. not the short context, but definitely the short replies. Contrast that with the current implementation of ChatGPT's voice mode, where you ask something and then get a minute worth of GPT bla bla.

mdbackman 2y ago

Very, very impressive! It's incredibly fast, maybe too fast, but I think that's the point. What's most impressive though is how the VAD and interruptions are tuned. That was, by far, the most natural sounding conversation I've had with an agent. Really excited to try this out once it's available.

sumedh 2y ago

This is very impressive, me and my kid had fund talking about space.

ftth_finland 2y ago

This is excellent! Perfect comprehension and no problem even with bad accents.

c0brac0bra 2y ago

I've been developing with Deepgram for a while, and this is one of the coolest demos I've seen with it! I am curious about total cost to run this thing, though. I assume that on top of whatever you're paying Cerebrium for GPU hosting you're also having to pay for Deepgram Enterprise in order to self-host it. To get the latency reduction of several hundred milliseconds how much more would it be for "average" usage?

za_mike157 2y ago
Hey! From the Cerebrium team here! So our costs are based on the infra you use to run your application and we charge per millisecond of compute. Some things to note that we might do differently to other providers: 1. You can specify your EXACT requirements and we charge you only for that. Eg: if you want 2 vCPU, 12GB Memory and 1 A10 GPU we charge you for that which is 35% less if you rented a whole A10 2. We have over 10 variety of GPU chips so you can choose the price/performance trade-off 3. While you can extend this on the Cerebrium platform, it cannot be used commercially. We are speaking to Deepgram to see how we can offer it to customers. Hopefully I can provide more updates on this soon
- c0brac0bra 2y ago
  Excellent; thanks for the info.

yalok 2y ago

you may be double counting opus encoding/decoding delay - usually, you can run it with 20ms frame, and both encoder and decoder take less than 1ms of realtime for their operation - so it should be ~ 21ms, instead of 30+30ms for 1 direction.

kwindla 2y ago
You are right! Thank you. I went back and looked at actual benchmark numbers from a couple of years ago and the numbers I got were ~26ms one-way. I rounded up to 30 to be conservative, but then double-counted in the table above. Will fix in the technical write-up. I don't think I can edit the Show HN.

vessenes 2y ago

This is so, so good. I like that it seems to be a teaser app for cerebrium, if I understand it. It has good killer app potential. My tests from iPad ranged from 1400ms to 400ms reported latency; in the low end, it felt very fluid. One thing this speed makes me think is that for some chat workflows you’ll need/get to have kind of a multi-step approach — essentially, quick response, during which time a longer data / info / RAQ query can be farmed out, then the informative result picks up. Humans work like this; we use lots of filler words as we sort of get going responding to things. Right now, most workflows seem to be just one shot prompting, or in the background, parse -> query -> generate. The better workflow once you have low latency response is probably something like: [3s of LLama 8b in your ears] -> query -> [55s of Llama 70b/GPT4/whatever you want, informed by query]. Very cool, thank you for sharing this.

za_mike157 2y ago
Hi Vessenes From Cerebrium here. Really appreciate the feedback - glad you had a good experience! This application is easy to extend/implement meaning you can edit it to however you like: - Swap in different LLM's, STT and TTS models - Change prompts as well as implement RAG etc In partnership with Daily, we really wanted to focus on the engineer here. So make it extremely flexible for them to edit the application to suit their use case/preference while at the same time take away the mundane infrastructure setup. You can read more about how to extend it here: https://docs.cerebrium.ai/v4/examples/realtime-voice-agents https://docs.cerebrium.ai/v4/examples/realtime-voice-agents
- vessenes 2y ago
  Thanks for this reply. Yep, as an engineer, this is awesome, the docs look simple and I’ll give it a whirl. As a product guy, it seems like it would be dead simple to start a company on this tech by just putting up a web page that lets people pick a couple choices and gives them a custom domain. Very cool!
c0brac0bra 2y ago
I've wondered about this as well. Is there a way to have a small, efficient LLM model that can estimate general task complexity without actually running the full task workload? Scoring complexity on a gradient would let you know you need to send a "Sure, one second let me look that up for you." instead of waiting for a long round trip.

deleted 2y ago

[deleted]

andruby 2y ago

This is really good. I'm blown away by how important the speed is. And this was from a mobile connection in Europe, with a shown latency of just over 1s.

SubiculumCode 2y ago

A chatbot that interrupts me even faster. Sorry for the sarcasm. maybe im just slow, but when I'm trying to formulate a question on the spot, I pause a lot. having the chatbot jump in and interrupt is frustrating. Humans recognize the difference between someone still planning on saying something, and when they've finished. I even tried to give it a rule where it shouldn't respond until I said "The End", and of course it couldn't follow that instruction.

SubiculumCode 2y ago
ps. The speed is impressive, but the key to a useful voice chatbot (which I've never seen) is one that adapts to your speaking style, identifies and employs turn-taking signals. I acknowledge there are multiple viable patterns of social interaction, some talk over each other, and find that fun and engaging, while others think that's just the worst, and wait for a clear signal for their turn to speak and expect the same. I am of the latter.
- SubiculumCode 2y ago
  I'm sure that, with an annotated dataset, a model could learn to pick up on the right cues.
makeitmore 2y ago
Very true. I think we are a bit aggressive with the VAD timeout. The demo was intended to showcase speed, but the bot can be a bit eager! You can tinker with the VAD settings, it could definitely use a bit more air (but that will impact latency in the event the user has indeed finished talking.) As others say below, the magic will be figuring out the pace and style in which the user talks and adapting to that on the fly.

spark_chicken 2y ago

i have tried it. it is really fast! I know making a real-time voice bot is not easy with this low latency. which LLM did you use? how large LLM to make the conversation efficient?

makeitmore 2y ago
This particular demo is using Llama3 8B. We initially started 70B, but it was a touch slower and needed much more VRAM. We found 8B good enough for general chit-chat like in this demo. Most real-world use-cases will likely have their own fine-tuned models.

deleted 2y ago

[deleted]

p_frank 2y ago

Amazing to see the metrics of each part that is involved! I've wondererd why you could not introduce a small sound that overplays the waiting time? Like an "hmm" to skip a few 100ms of the response time? Could be pregenerated (like 500 different versions) and play after 200ms of the last users input.

asjir 2y ago

Personally, I use https://github.com/foges/whisper-dictation https://github.com/foges/whisper-dictation with llama-70b on groq. I start talking, navigate to website, and by the time it's loaded, and I picked llama-70b I finish talking, so 0 overhead. I read much faster than listen, so it works for me perfectly.

realyashnag 2y ago

This was scary fast. Neat interface and (almost) indistinguishable from a human over the phone / internet. Kudos @cerebrium.ai.

hole_in_foot 2y ago

[dead]

deleted 2y ago

[deleted]

andrewmcwatters 2y ago

I love it when engineers worth their salt actually do the back-of-the-envelope calculations for latency, etc. Tangentially related, I remember years ago when Stadia and other cloud gaming products were being released doing such calculations and showing a buddy of mine that even in the best case scenario, you'd always have high enough input latency to make even casual multiplayer FPS games over cloud gaming services not feasible, or rather, comfortable, to play. Other slower-paced games might work, but nothing requiring serious twitch gameplay reaction times. The same math holds up today because of a combination of fundamental limits and state of the art limits.

Flumio 2y ago
The calculations I was reading at the time suggested it would work for casual due to the gaming PC being very close to the game servers and running inside the best network available (googles). Google also said that the controller would send the input straight to the server. And a fast stadia server should have good fps combined with a little bit of brain prediction

trueforma 2y ago

I too am excited about voice inferencing. I wrote my own Websocket Faster whisper implementation before OpenAI's gpt4o release . They steamrolled my interview coach concept https://intervu.trueforma.ai https://intervu.trueforma.ai and https://sales.trueforma.ai https://sales.trueforma.ai - sales pitch coach implementations. I defaulted to Push to talk implementation as I couldn't get VAD to work reliably. I run it all on a panda Latte :) Was looking to implement Groq's hosted whisper. I love the idea of having Llama3 uncensored on Groq as the LLM as I'm tired of the boring corporate conversations. I hope to reduce my latency and learn from your examples - Kudos to your efforts. I wish I could try the demo - seems to be over subscribed as I can't get in to talk to the bot. I'm sure my latte Panda would melt if just 3 people try to inference at the same time :)

_DeadFred_ 2y ago

This is super cool. Thanks for sharing. And I'm excited it encourage other to share. I'm excited to spend some time this weekend looking at the different ways people in this thread implemented solutions.

Borborygymus 2y ago

It /was/ nice and quick. Thanks for putting the demo online. It was quick to tell me complete nonsense. Apparently 7122 is the atomic number of Barium.

amluto 2y ago

Maybe silly question: > jitter buffer [40ms] Why do you need a jitter buffer on the listening side? The speech-to-text model has neither ears nor a sense of rhythm — couldn’t you feed in the audio frames as you receive them? I don’t see why you need to delay processing a frame by 40ms just because the next one might be 40ms late.

Olreich 2y ago
Almost any gap in audio is detectable and sounds really bad. 40ms is a lot, but sending 40ms of silence is probably worse
- amluto 2y ago
  Sounds bad to whom? I’m talking about the direction from user to AI, not the direction from AI to user. If some of the audio gets delayed on the way to the AI, the AI can be paused. If some of the audio gets delayed on the way to a human, the human can’t be paused, so some buffering is needed to reduce the risk of gaps.

gsjbjt 2y ago

That's awesome - can you say anything about what datasets this was trained on? I assume something specifically conversational?

tamimio 2y ago

Or we can say the latency is a good listening skills!! It was fast but occasionally interrupted me to answer.