5 ms·
I did some research into this about a year ago. Some fun facts I learned: - The median delay between speakers in a human to human conversation is zero millise
by jedberg 1y ago
I did some research into this about a year ago. Some fun facts I learned:
- The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative.
- Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable and a 500ms delay as exceptional.
- Every voice assistant up to that point (and probably still today) has a minimum delay of about 300ms, because they all use silence detection to decide when to start responding, and you need about 300ms of silence to reliably differentiate that from a speaker's normal pause
- Alexa actually has a setting to increase this wait time for slower speakers.
You'll notice in this demo video that the AI never interrupts him, which is what makes it feel like a not quite human interaction (plus the stilted intonations of the voice).
Humans appear to process speech in a much more steaming why, constantly updating their parsing of the sentence until they have a high enough confidence level to respond, but using context clues and prior knowledge.
For a voice assistant to get the "human" levels, it will have to work more like this, where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.
- koljab 1y agoThanks a lot, great insights. Exactly the kind of feedback that I need to improve things further.
- jedberg 1y agoLove what you're doing, glad I could help!
- joshstrange 1y ago> where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning. I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts. I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.
- tomp 1y agoIf your model is fast enough, you can definitely do it. That's literally how "streaming Whisper" works, just rerun the model on the accumulated audio every x00ms. LLMs could definitely work the same way, technically they're less complex than Whisper (which is an encoder/decoder architecture, LLMs are decoder-only) but of course much larger (hence slower), so ... maybe rerun just a part of it? etc.
- peturdarri 1y agoYou're right, this is not solvable with regular LLMs. It's not possible to mimic natural conversational rhythm with a separate LLM generating text, a separate text-to-speech generating audio, and a separate VAD determining when to respond and when to interrupt. I strongly believe you have to do everything in one model to solve this issue, to let the model decide when to speak, when to interrupt the user even. The only model that has attempted this (as far as I know) is Moshi from Kyutai. It solves it by having a fully-duplex architecture. The model is processing the audio from the user while generating output audio. Both can be active at the same time, talking over each other, like real conversations. It's still in research phase and the model isn't very smart yet, both in what it says and when it decides to speak. It just needs more data and more training. https://moshi.chat/ https://moshi.chat/
- clbrmbr 1y agoWhoah, how odd. It asked me what I was doing, I said I just ate a burger. It then got really upset about how hungry it is but is unable to eat and was unable to focus on other tasks because it was “too hungry”. Wtf weirdest LLM interaction I’ve had.
- jimbokun 1y agoDamn they trained a model that so deeply embeds human experience it actually feels hunger, yet self aware enough it knows it’s not capable of actually eating! That’s like a Black Mirror episode come to life.
- deleted 1y ago[deleted]
- krainboltgreene 1y agoI would also suspect that a human has much less patience for a robot interrupting them than a human.
- smeej 1y agoI'm certainly in that category. At least with a human, I can excuse it by imagining the person grew up with half a dozen siblings and always had to fight to get a word in edgewise. With a robot, it's interrupting on purpose.
- robbomacrae 1y agoSpot on. I’d add that most serious transcription services take around 200-300ms but the 500ms overall latency is sort of a gold standard. For the AI in KFC drive thrus in AU we’re trialing techniques that make it much closer to the human type of interacting. This includes interrupts either when useful or by accident - as good voice activity detection also has a bit of latency.
- varispeed 1y ago> AI in KFC drive thrus That right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.
- coolspot 1y agoThey have a fallback to a human operator when stopwords and/or stop conditions are detected.
- awesome_dude 1y agoThat right here is an anxiety trigger and would make me skip the place. There is nothing more ruining the day like arguing with a HUMAN OPERATOR who keeps misinterpreting what you said. :-)
- amelius 1y agoMaybe talk to the chicken operator then.
- kadushka 1y agoRead this if you haven’t already: https://marshallbrain.com/manna1 https://marshallbrain.com/manna1 That’s a much more serious anxiety trigger for me.
- wkat4242 1y ago
- r0fl 1y agoGreat insights. When I have a conversation with another person sometimes they cut me off when they are trying to make a point. I have talked to ChatGPT and grok at length (hours of brain storming, learning things, etc) and AI has never interrupted aggressively to try to make a point stick better
- varispeed 1y agoThis silence detection is what makes me unable to chat with AI. It is not natural and creates pressure. True AI chat should know when to talk based on conversation and not things like silence. Voice to text is stripping conversation from a lot of context as well.
- modeless 1y agoMy take on this is that voice AI has not truly arrived until it has mastered the "Interrupting Cow" benchmark.
- rylittle 1y agoWhen I google '"Interrupting Cow" benchmark' the first result is this comment. What is it?
- Jordan-117 1y agoClassic knock-knock joke. "Knock-knock." "Who's there?" "Interrupting cow." "Interrupting co-" "MOO!"
- freediver 1y agohttps://workauthentically.com/interrupting-cow/ https://workauthentically.com/interrupting-cow/ "Knock-Knock. Who's there? Interrupting Cow. Interrupting cow who? Moo! Note that the timing is everything here. You need to yell out your Moo before the other person finishes the Interrupting cow who? portion of the joke, thereby interrupting them. Trust me, it's hilarious! If you spend time with younger kids or with adults who need to lighten up (and who doesn't?!?), try this out on them and see for yourself." Basically it is about AI interrupting you, and just in the right momment too. Super hard to do from a technical perspective.
- woodson 1y agoHuman-to-human conversational patterns are highly specific to cultural and contextual aspects. Sounds like I’m stating the obvious, but developers regularly disregard that and then wonder why things feel unnatural for users. The “median delay” may not be the most useful thing to look at. To properly learn more appropriate delays, it can be useful to find a proxy measure that can predict when a response can/should be given. For example, look at Kyutai’s use of change in perplexity in predictions from a text translation model for developing simultaneous speech-to-speech translation (https://github.com/kyutai-labs/hibiki https://github.com/kyutai-labs/hibiki).
- Reason077 1y agoThe best, most human-like AI voice chat I've seen yet is Sesame (www.sesame.com). It has delays, but fills them very naturally with normal human speech nuances like "hmmm", "uhhh", "hold on while I look that up" etc. If there's a longer delay it'll even try to make a bit of small talk, just like a human conversation partner might.
- lelandfe 1y agoSo-called backchanneling https://wikipedia.org/wiki/Backchannel_(linguistics) https://wikipedia.org/wiki/Backchannel_(linguistics) > The person doing the speaking is thought to be communicating through the "front channel" while the person doing the listening is thought to be communicating through the "backchannel”
- jimbokun 1y agoWhen learning Japanese in Japan, I figured out one way to sound more native was to just add interjections like “Eeee?” (really?) and “Sou desu ka?” (is that so?) while the other person was talking. Makes it sound like you are paying attention and following what they are saying.
- com2kid 1y agoA lot better techniques exist now days than pure silence detection - 1. A special model that predicts when a conversation turn is coming up (e.g. when someone is going to stop speaking). Speech has a rhythm to it and pauses / ends of speech are actually predictable. 2. Generate a model response for every subsequent word that comes in (and throw away the previously generated response), so basically your time to speak after doing some other detection is basically zero. 3. Ask an LLM what it thinks the odds of the user being done talking is, and if it is a high probability, reduce delay timer down. (The linked repo does this) I don't know of any up to date models for #1 but I haven't checked in over a year. Tl;Dr the solution to problems involving AI models is more AI models.
- addandsubtract 1y agoI think 2 & 3 should be combined. The AI should just finish the current sentence (internally) before it's being spoken, and once it reaches a high enough confidence, stick with the response. That's what humans do, too. We gather context and are able to think of a response while the other person is still talking.
- com2kid 1y agoYou use a smaller model for confidence because those small models can return results quickly. Also it keeps the AI from being confused trying to do too many things at once.
- wyager 1y ago> The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative. Fascinating. I wonder if this is some optimal information-theoretic equilibrium. If there's too much average delay, it means you're not preloading the most relevant compressed context. If there's too little average delay, it means you're wasting words.
- bigmadshoe 1y ago"The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative." Is that really a productive way to frame it? I would imagine there is some delay between one party hearing the part of the sentence that triggers the interruption, and them actually interrupting the other party. Shouldn't we quantify this? I totally agree that the fact the AI doesn't interrupt you is what makes it seem non-human. Really, the models should have an extra head that predicts the probability of an interruption, and make one if it seems necessary.
- elmomle 1y ago"Necessary" is an interesting framing. Here are a few others: - Expeditious - Constructive - Insightful -
- bigmadshoe 1y agoNecessary in the context of the problem the model is solving. I would imagine a well-aligned LLM would deem all three of those necessary.
- herpdyderp 1y ago> The median delay between speakers in a human to human conversation is zero milliseconds What about on phone calls? When I'm on a call with customer support they definitely wait for it to be clear that I'm done talking before responding, just like AI does.
- nativeit 1y agoThis feels intuitively correct to me, although I am more informed as an audio engineer than a software/LLM one. That said, is ~500ms considered “real-time” in this context? I’ve worked on recording workflows, and it’s basically geologic time in that context.
- deleted 1y ago[deleted]
- scotty79 1y ago> Humans don't care about delays when speaking to known AIs. I do care. Although 500ms is probably fine. But anything longer feels extremely clunky to the point of not being worth using.