4 ms·

This is really impressive engineering. I thought real time agents would completely change the way we're going to interact with large models but it would take 1~

This is really impressive engineering. I thought real time agents would completely change the way we're going to interact with large models but it would take 1~2 more years. I wonder what kind of new techs are developed to enable this, but OpenAI is fairly secretive so we won't be able to know their sauce. On the other hand, this also feels like a signal that reasoning capability has probably already been plateaued at GPT-4 level and OpenAI knew it so they decided to focus on research that matters to delivering product engineering rather than long-term research to unlock further general (super)intelligence.

nopinsight 2y ago
Reliable agents in diverse domains need better reasoning ability and fewer hallucinations. If the rumored GPT-5 and Q* capabilities are true, such agents could become available soon after it’s launched.
- summerlight 2y ago
  Sam has been pretty clear on denying GPT-5 rumors, so I don't think it will come anytime soon.
  - nopinsight 2y ago
    Sam mentioned on several occasions that GPT-5 will be much smarter than GPT-4. On Lex Fridman’s podcast, he even said the gap between GPT-5 and 4 will be as wide as GPT-4 and 3 (not 3.5). He did remain silent on when it’s going to be launched.
    - valine 2y ago
      OpenAI has been open about their ability to predict model performance prior to training. When Sam talks about GPT-5 he could very easily be talking about the hypothetical performance of a model given their internal projections. I think it’s very unlikely a fully trained GPT-5 exists yet.
      - bigyikes 2y ago
        Sam has stated that he knows the month GPT-5 will be released. Given the amount of time and uncertainty involved in training and red-teaming these models, we can assume GPT-5 exists if we take Altman at his word.
    - Atotalnoob 2y ago
      It’s going to be launched this year. My buddy’s company had a private demo of gpt5
  - jasondigitized 2y ago
    Ben Horowitz and Andreesen just complimented Sam on their podcast on how smart he is. They then went onto compliment how adept he is at competitive strategy. I wouldn’t trust a word he says about when they will arrive at agi or milestones along the way.
MVissers 2y ago
Why would reasoning have plateau’d? I think reasoning ability is not the largest bottleneck for improvement in usefulness right now. Cost is a bigger one IMO. Running these models as agents is hella expensive, and agents or agent-like recurrent reasoning (like humans do) is the key to improved performance if you look at any type of human intelligence. Single-shot performance only gets you so far. For example- If it can write code 90% of the way, and then debug in a loop, it’d be much more performant than any single shot algorithm. And OpenAI has these huge models in their basement probably. But they might not be much more useful than GPT-4 when used as single-shot. I mean, what could it do what we can’t do today with gpt-4? It’s agents and recurrent reasoning we need for more usefulness. At least- That’s my humble opinion as an amateur neuroscientist that plays around with these models.
- Jensson 2y ago
  > Running these models as agents is hella expensive Because they are dumb so you need to over compute so many things to get anything useful. Smarter models would solve this problem. Making the current model cheaper is like trying to solve Go by scaling up Deep Blue, it doesn't work to just hardcode dumb pieces together, the model needs to get smarter.
  - cchance 2y ago
    You mean like our dumb ass brains? Theirs a reason "saying the first thing out of your mind" is a bad fucking idea, thats what AI's currently do, they don't take a moment think about the answer and then formulate a response, they spit out their first "thought" thats why multi-shot works so much better, just like our own dumb brains.
    - Jensson 2y ago
      My brain can navigate a computer interface without using word tokens, since I have tokens for navigating OS and browsers and tabs etc. That way I don't have to read a million tokens of text to figure out where buttons are or how to navigate to places, since my brain is smart enough to not use words for it. ChatGPT doesn't have that sort of thing currently, and until it does it will always be really bad at that sort of thing. You are using a hand to hammer a nail, that will never go well, the solution isn't to use more hands the solution is to wield a hammer.
CuriouslyC 2y ago
This isn't really new tech, it's just an async agent in front of a multimodal model. It seems from the demo that the improvements have been in response latency and audio generation. Still, it looks like they're building a solid product, which has been their big issue so far.
- searealist 2y ago
  No, audio is fed directly into the model. There is no text to speech transformer in front of it like there was with chatgpt-4.
- cchance 2y ago
  Its 200-300ms for a multimodal response, thats REALLY a big step forward, especially given it's doing it with full voice response, not just text.
  - valine 2y ago
    It has to be a separate model doing the voice output right? I can’t imagine they’ve solved true multimodal output from a single model, they’d be bragging about it. They’re probably predicting tone of voice tokens. Feed that into an audio transformer along with some speculative decoding to keep latency low.
    - throwthrowuknow 2y ago
      The old voice mode was but everyone including gdb is saying that this one is natively multimodal once it’s fully rolled out, audio in audio out. This has been in the works for a while, you can look up papers on things like OCR-free document understanding and the like but the basic idea is you just train it and evaluate it on whatever media you want it to understand. As long as you can tokenize it it’ll work.
      - valine 2y ago
        It’s definitely multimodal input. Passing Clip embeddings to an LLM is nothing new, and that’s really all you need for document understanding. It’s almost certainly the same thing for audio. They would have trained a dual encoder that maps both audio and text to a shared embedding space. What’s not at all clear to me is if they’re doing something special for output. Are you saying OpenAI has moved beyond next token prediction and just hasn’t bothered to mention it?
cchance 2y ago
Ya so sad that OpenAI isn't more Open imagine if OpenAI was still sharing their thought processes and papers with the overall commity, really wish we saw collaborations between OpenAI and Meta for instance to really have helped push the open source arena further ahead, i love that their latest models are so great but the fact they aren't helping the Open source arena to progress is sad. Imagine how far we'd be if OpenAI was still as open as they once were and we saw collaborations betweeen Meta, OpenAI and Anthropic all working and sharing growth and tech to reduce double work and help each other not go down failed paths.

nopinsight 2y ago

Reliable agents in diverse domains need better reasoning ability and fewer hallucinations. If the rumored GPT-5 and Q* capabilities are true, such agents could become available soon after it’s launched.

summerlight 2y ago
Sam has been pretty clear on denying GPT-5 rumors, so I don't think it will come anytime soon.
- nopinsight 2y ago
  Sam mentioned on several occasions that GPT-5 will be much smarter than GPT-4. On Lex Fridman’s podcast, he even said the gap between GPT-5 and 4 will be as wide as GPT-4 and 3 (not 3.5). He did remain silent on when it’s going to be launched.
  - valine 2y ago
    OpenAI has been open about their ability to predict model performance prior to training. When Sam talks about GPT-5 he could very easily be talking about the hypothetical performance of a model given their internal projections. I think it’s very unlikely a fully trained GPT-5 exists yet.
    - bigyikes 2y ago
      Sam has stated that he knows the month GPT-5 will be released. Given the amount of time and uncertainty involved in training and red-teaming these models, we can assume GPT-5 exists if we take Altman at his word.
  - Atotalnoob 2y ago
    It’s going to be launched this year. My buddy’s company had a private demo of gpt5
- jasondigitized 2y ago
  Ben Horowitz and Andreesen just complimented Sam on their podcast on how smart he is. They then went onto compliment how adept he is at competitive strategy. I wouldn’t trust a word he says about when they will arrive at agi or milestones along the way.

MVissers 2y ago

Why would reasoning have plateau’d? I think reasoning ability is not the largest bottleneck for improvement in usefulness right now. Cost is a bigger one IMO. Running these models as agents is hella expensive, and agents or agent-like recurrent reasoning (like humans do) is the key to improved performance if you look at any type of human intelligence. Single-shot performance only gets you so far. For example- If it can write code 90% of the way, and then debug in a loop, it’d be much more performant than any single shot algorithm. And OpenAI has these huge models in their basement probably. But they might not be much more useful than GPT-4 when used as single-shot. I mean, what could it do what we can’t do today with gpt-4? It’s agents and recurrent reasoning we need for more usefulness. At least- That’s my humble opinion as an amateur neuroscientist that plays around with these models.

Jensson 2y ago
> Running these models as agents is hella expensive Because they are dumb so you need to over compute so many things to get anything useful. Smarter models would solve this problem. Making the current model cheaper is like trying to solve Go by scaling up Deep Blue, it doesn't work to just hardcode dumb pieces together, the model needs to get smarter.
- cchance 2y ago
  You mean like our dumb ass brains? Theirs a reason "saying the first thing out of your mind" is a bad fucking idea, thats what AI's currently do, they don't take a moment think about the answer and then formulate a response, they spit out their first "thought" thats why multi-shot works so much better, just like our own dumb brains.
  - Jensson 2y ago
    My brain can navigate a computer interface without using word tokens, since I have tokens for navigating OS and browsers and tabs etc. That way I don't have to read a million tokens of text to figure out where buttons are or how to navigate to places, since my brain is smart enough to not use words for it. ChatGPT doesn't have that sort of thing currently, and until it does it will always be really bad at that sort of thing. You are using a hand to hammer a nail, that will never go well, the solution isn't to use more hands the solution is to wield a hammer.

CuriouslyC 2y ago

This isn't really new tech, it's just an async agent in front of a multimodal model. It seems from the demo that the improvements have been in response latency and audio generation. Still, it looks like they're building a solid product, which has been their big issue so far.

searealist 2y ago
No, audio is fed directly into the model. There is no text to speech transformer in front of it like there was with chatgpt-4.
cchance 2y ago
Its 200-300ms for a multimodal response, thats REALLY a big step forward, especially given it's doing it with full voice response, not just text.
- valine 2y ago
  It has to be a separate model doing the voice output right? I can’t imagine they’ve solved true multimodal output from a single model, they’d be bragging about it. They’re probably predicting tone of voice tokens. Feed that into an audio transformer along with some speculative decoding to keep latency low.
  - throwthrowuknow 2y ago
    The old voice mode was but everyone including gdb is saying that this one is natively multimodal once it’s fully rolled out, audio in audio out. This has been in the works for a while, you can look up papers on things like OCR-free document understanding and the like but the basic idea is you just train it and evaluate it on whatever media you want it to understand. As long as you can tokenize it it’ll work.
    - valine 2y ago
      It’s definitely multimodal input. Passing Clip embeddings to an LLM is nothing new, and that’s really all you need for document understanding. It’s almost certainly the same thing for audio. They would have trained a dual encoder that maps both audio and text to a shared embedding space. What’s not at all clear to me is if they’re doing something special for output. Are you saying OpenAI has moved beyond next token prediction and just hasn’t bothered to mention it?

cchance 2y ago

Ya so sad that OpenAI isn't more Open imagine if OpenAI was still sharing their thought processes and papers with the overall commity, really wish we saw collaborations between OpenAI and Meta for instance to really have helped push the open source arena further ahead, i love that their latest models are so great but the fact they aren't helping the Open source arena to progress is sad. Imagine how far we'd be if OpenAI was still as open as they once were and we saw collaborations betweeen Meta, OpenAI and Anthropic all working and sharing growth and tech to reduce double work and help each other not go down failed paths.