3 ms·

100%. I made this demo in 2-3 hours, and I did use the "wait until the dictation results are finalized" technique which is safer (i.e. the dictation transcript

100%. I made this demo in 2-3 hours, and I did use the "wait until the dictation results are finalized" technique which is safer (i.e. the dictation transcription is more robust) but slower. For another demo - https://www.youtube.com/watch?v=fxS7OKh_4vc https://www.youtube.com/watch?v=fxS7OKh_4vc - I kept feeding the "in progress" transcription results into GPT and that was super super awesome & fast. It would just require more work to deal with all of the different timings going on (i.e. there's the speech itself from the person, the time to transcribe, sending the request to GPT, "sync'ing" it to where the person is (mentally/in their speech) at the point where GPT replies, etc.) But yeah. Real time/continuous talk is absolutely where it's at. Should GPT be available as a websocket...?!

modeless 3y ago
I have a rough demo of real time continuous voice chat here, ~1 second response latency: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 Basically it starts computing a response every time a word comes out of the speech recognizer, and if it is able to finish its response before it hears another word then it starts speaking. If more words come in then it stops speaking immediately; in other words, you can interrupt it. It feels so much more natural in conversation than ChatGPT's voice mode due to the low latency and continuous listening with the ability to interrupt. There are a lot of things that need improvement. Most important is probably that the speech recognition system (Whisper) wasn't designed for real time and is not that reliable or efficient in a real time mode. I think some more tweaking could improve reliability considerably. But also very important is that it doesn't know when not to respond. It will always jump in if you stop speaking for a second, and it will always try to get the last word. A first pass at fixing that would be to fine tune a language model to predict whose turn it is to speak. There are also a lot of things that this architecture will never be able to do. It will never be able to correct your pronunciation (e.g. for language learning), it will never be able to identify your emotions based on vocal cues or express proper emotions in its voice, it will never be able to hear the tone of a singing voice or produce singing itself. The future is in eliminating the boundaries between speech-to-text and LLM and text-to-speech, with one unified model trained end-to-end. Such a system would be able to do everything I mentioned and more, if trained on enough data. And further integrating vision will help with conversation too, allowing it to see the emotions on your face and take conversational cues from your gaze direction and hand gestures, in addition to all the other obvious things you can do with vision such as chat about something the camera can see or something displayed on your screen.
- thom 3y ago
  What's the horizon after which you reset the input instead of appending to it? Does that happen if the user lets the system finish speaking?
  - modeless 3y ago
    Great question. Right now that happens, somewhat arbitrarily, if the user lets the system finish speaking the first sentence of its response. If the user interrupts before that, then it's considered a continuation of the previous input. If the user interrupts after that, it's still an interruption (and, importantly, the language model's response must be truncated in the conversation context because the user didn't hear it all), but it starts a new input to the LLM. This could be handled better as well. Basically any heuristics like this that are in the system should eventually be subsumed into the AI models so that they can be responsive to the conversation context.
- leminimal 3y ago
  Do you have a version that doesn't need Windows and/or a Microsoft account? Or an uncut video of someone using it?
xiphias2 3y ago
This would be super cool with Mistral on local machine

modeless 3y ago

I have a rough demo of real time continuous voice chat here, ~1 second response latency: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 Basically it starts computing a response every time a word comes out of the speech recognizer, and if it is able to finish its response before it hears another word then it starts speaking. If more words come in then it stops speaking immediately; in other words, you can interrupt it. It feels so much more natural in conversation than ChatGPT's voice mode due to the low latency and continuous listening with the ability to interrupt. There are a lot of things that need improvement. Most important is probably that the speech recognition system (Whisper) wasn't designed for real time and is not that reliable or efficient in a real time mode. I think some more tweaking could improve reliability considerably. But also very important is that it doesn't know when not to respond. It will always jump in if you stop speaking for a second, and it will always try to get the last word. A first pass at fixing that would be to fine tune a language model to predict whose turn it is to speak. There are also a lot of things that this architecture will never be able to do. It will never be able to correct your pronunciation (e.g. for language learning), it will never be able to identify your emotions based on vocal cues or express proper emotions in its voice, it will never be able to hear the tone of a singing voice or produce singing itself. The future is in eliminating the boundaries between speech-to-text and LLM and text-to-speech, with one unified model trained end-to-end. Such a system would be able to do everything I mentioned and more, if trained on enough data. And further integrating vision will help with conversation too, allowing it to see the emotions on your face and take conversational cues from your gaze direction and hand gestures, in addition to all the other obvious things you can do with vision such as chat about something the camera can see or something displayed on your screen.

thom 3y ago
What's the horizon after which you reset the input instead of appending to it? Does that happen if the user lets the system finish speaking?
- modeless 3y ago
  Great question. Right now that happens, somewhat arbitrarily, if the user lets the system finish speaking the first sentence of its response. If the user interrupts before that, then it's considered a continuation of the previous input. If the user interrupts after that, it's still an interruption (and, importantly, the language model's response must be truncated in the conversation context because the user didn't hear it all), but it starts a new input to the LLM. This could be handled better as well. Basically any heuristics like this that are in the system should eventually be subsumed into the AI models so that they can be responsive to the conversation context.
leminimal 3y ago
Do you have a version that doesn't need Windows and/or a Microsoft account? Or an uncut video of someone using it?

xiphias2 3y ago

This would be super cool with Mistral on local machine