4 ms·

The "magic" of the fake Gemini demo was the way it seemed like the LLM was continually receiving audio + video input and knew when to jump in with a response.

• famouswaffles 3y ago
  I think probably training on pause tokens or something similar would be the key to something like this. Maybe it's not even necessary. Maybe if you just tell GPT-4 to output something like .... every time it thinks it should wait for a response (you wouldn't need to wait for the user to finish then), things would be a lot smoother.
  • phire 3y ago
    Yes, you could probably fine-tune (or even zero-shot) a LLM to handle the "knowing when to jump in" use case. The real problem is that it's simply too computationally expensive to continually feed audio and video into it one of these massive LLMs just in case it might decide to jump in. I was wondering if you could train a lightweight monitoring model that continually watching the audio/video input and only tried to work out when the full-sized LLM might want to jump in and generate a response.
    • b112 3y ago
      As the human brain is a clump if regions all interconnected and interacting, for example, one may focus their attention elsewhere until their name is called, having a ight model wait for an important queue makes sense more than fiscally. One time I was so distracted, I missed an entire paragraph someone said to me, walked to my car, drove away, and 5 minutes later processed it.
      • Ajedi32 3y ago
        Yeah, one thing I've noticed myself do is that when I'm focused on something else and someone suddenly gets my attention I'll replay the last few seconds of the conversation in my head to get context on what was being talked about before I respond. That seems pretty trivial to do with a LLM; it doesn't need to be using 100% of its "brainpower" at all times.
• nextworddev 3y ago
  One easy improvement would be to stop the video capture automatically via a combination of silence detection and motion detection
• gregsadetsky 3y ago
  100%. I made this demo in 2-3 hours, and I did use the "wait until the dictation results are finalized" technique which is safer (i.e. the dictation transcription is more robust) but slower. For another demo - https://www.youtube.com/watch?v=fxS7OKh_4vc https://www.youtube.com/watch?v=fxS7OKh_4vc - I kept feeding the "in progress" transcription results into GPT and that was super super awesome & fast. It would just require more work to deal with all of the different timings going on (i.e. there's the speech itself from the person, the time to transcribe, sending the request to GPT, "sync'ing" it to where the person is (mentally/in their speech) at the point where GPT replies, etc.) But yeah. Real time/continuous talk is absolutely where it's at. Should GPT be available as a websocket...?!
  • modeless 3y ago
    I have a rough demo of real time continuous voice chat here, ~1 second response latency: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 Basically it starts computing a response every time a word comes out of the speech recognizer, and if it is able to finish its response before it hears another word then it starts speaking. If more words come in then it stops speaking immediately; in other words, you can interrupt it. It feels so much more natural in conversation than ChatGPT's voice mode due to the low latency and continuous listening with the ability to interrupt. There are a lot of things that need improvement. Most important is probably that the speech recognition system (Whisper) wasn't designed for real time and is not that reliable or efficient in a real time mode. I think some more tweaking could improve reliability considerably. But also very important is that it doesn't know when not to respond. It will always jump in if you stop speaking for a second, and it will always try to get the last word. A first pass at fixing that would be to fine tune a language model to predict whose turn it is to speak. There are also a lot of things that this architecture will never be able to do. It will never be able to correct your pronunciation (e.g. for language learning), it will never be able to identify your emotions based on vocal cues or express proper emotions in its voice, it will never be able to hear the tone of a singing voice or produce singing itself. The future is in eliminating the boundaries between speech-to-text and LLM and text-to-speech, with one unified model trained end-to-end. Such a system would be able to do everything I mentioned and more, if trained on enough data. And further integrating vision will help with conversation too, allowing it to see the emotions on your face and take conversational cues from your gaze direction and hand gestures, in addition to all the other obvious things you can do with vision such as chat about something the camera can see or something displayed on your screen.
• TaylorAlexander 3y ago
  Yeah my friend and I were just talking about continuous stream input multimodal LLMs. Does anyone know if there is a technical limitation preventing continuous stream input data? Like it’s listening to you practice guitar and then when you get to a certain point it says “okay let’s go back and practice that section again”. It seems the normal approach of next token prediction falls flat when there is a continuous stream of tokens and it only sometimes needs to produce output. What is that type of input called in the literature and what research has been done on it? Thanks!
  • profile53 3y ago
    At a purely technical level, no, as long as the model can output a null token. E.g. imagine training using a transcript of two people talking. What would be a single text token is a tuple of two tokens, one per person. Each segment where a person is not talking is a series of null tokens, one per ‘tick’ of time. In an actual conversation, one token in the tuple is user input and one is GPT prediction. Just disregard the user half of the tuple when determining whether the GPT should ‘speak’. The real world challenge is threefold. First, null tokens would be massively over represented in training and by extent, in outputs. Second, at a computational level, outputting a continuous stream of tokens would be absurdly expensive. Third, there is not nearly as much training data of interspersed conversations as of monologues (e.g. research papers, this comment, etc.).
    • TaylorAlexander 3y ago
      Yeah it seems the notion of time is sort of not built in conceptually to current systems. You could pick a fixed time constant like 0.1 seconds or 1 second, but it's clear that it's sort of missing something more fundamental.
      • radarsat1 3y ago
        I think if the same LLM were trained on audio and video input instead of text, and produced audio output, including silence tokens, then the notion of time would get "built in". Audio continuation without translation to text has been shown to work. Mixing it with text is also possible. But all this would require a massive network that maybe even be difficult for the world's biggest companies to train and serve at any kind of scale. So it's more of an engineering problem than a theoretical one imho. Also imho, I think until the context/memory problem is fully solved we won't really see the AI as having any kind of agency. But continuous, low latency interaction would certainly feel like a step towards that.
• ycdxvjp 3y ago
  As a deaf person I have been watching "live" speech recognition demos for 20-30 years. All look great. Using it in day to day life is crazy cause if you have 1 mistake per 10 words it builds up over time to be supremely annoying.
• LelouBil 3y ago
  I wanted to plug a GPT4 chatbot into a group chat, so it could react to what people said. In the end I abandoned the idea because it was too hard for me to figure out when it should talk vs let people talk between them.
  • cubefox 3y ago
    Couldn't you instruct the model to only say something when it is important or when it's being addressed directly, and otherwise output just empty response which isn't rendered?
• BiteCode_dev 3y ago
  For now LLM can only answer, but they will soon be able to prompt YOU. True conversation is going to be very interesting.