3 ms·

What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the v

What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3, and Sam Altman has directly said there are no plans for a GPT 5. I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming.

famouswaffles 3y ago
>What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Past the introduction of the transformer in 2017, There is no big "innovation". It is just scale. Bigger models are better. The last 4 years can be summed up that simply. >Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? What is existing technology here ? Open ai aren't doing anything so alien you couldn't guess at if you knew what you were doing but image training at the scale of GPT-4 is new and it's not even the cleanest way to do it. We still don't have a "trained from scratch" large scale multimodal LLM yet. >The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3 Definitely not lol. The OG GPT-3 was pulling sub 50 on MMLU. Even benchmarks aside, there is a massive gap in utility between 3.5 and 4, never mind 3. 4 was finished training august 2022. It's only 2 years apart from 3. >I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming. There was no special leap (in terms of theory and engineering). This is scale plainly laid out and there's more of it to go. >and Sam Altman has directly said there are no plans for a GPT 5. the same that sat on 4 for 8 months and said absolutely nothing about it ? Take anything altman says about new iterations with a grain of salt.
- WendyTheWillow 3y ago
  Firstly no, the gap between 3 and 4 is not anything as large as the gap between 2 and 3. Secondly, nothing you said here changed as of this announcement. Nothing here makes it any more or less likely LLMs will risk software engineering jobs. Thirdly, you can take what Sam Altman says with as many grains of salt as you like, if there really was no innovation at all as you claim, then there will be a limit hit at computing capability and cost.
  - famouswaffles 3y ago
    >the gap between 3 and 4 is not anything as large as the gap between 2 and 3. We'll just have to agree to disagree. 3 was a signal of things to come but it was ultimately a bit of a toy, a research curiosity. Utility wise, they are worlds apart. >if there really was no innovation at all as you claim, then there will be a limit hit at computing capability and cost. computing capability and cost are just about the one thing you can bank on to reduce. already training gpt-4 today would be a fraction of the cost than it was when open ai did it and that was just over a year ago. Today's GPU's take ML into account to some degree but they are nowhere near as calibrated for it as they could be. That work has just begun to start. Of any of the possible barriers, compute is exactly the kind you want. It will fall.
    - WendyTheWillow 3y ago
      Do you realize I'm not disagreeing with you about the difference between 3 and 4? Reread what I wrote. I contrasted 3 and 4 with 2 and 3, which you seem to be entirely ignoring. 3 and 4 could be worlds apart, but wouldn't matter if 2 and 3 were two worlds apart, for example. And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. You would need a matching exponential growth in processing capability which is most certainly not occurring. So compute will not fall at the rate you would need it to for LLMs to actually compete in any meaningful way with human software engineers. We are not guaranteed to continue to progress in anything just because we have in the past.

famouswaffles 3y ago

>What actually was the innovation in LLMs that produced the kind of AI we're seeing now? Is that innovation ongoing or did it happen, and now we're seeing the various optimizations of that innovation? Past the introduction of the transformer in 2017, There is no big "innovation". It is just scale. Bigger models are better. The last 4 years can be summed up that simply. >Is voice and image integration with ChatGPT a whole new capability of LLMs or is the "product" here a clean and intuitive interface through which to use the already existent technology? What is existing technology here ? Open ai aren't doing anything so alien you couldn't guess at if you knew what you were doing but image training at the scale of GPT-4 is new and it's not even the cleanest way to do it. We still don't have a "trained from scratch" large scale multimodal LLM yet. >The difference between GPT 3, 3.5, and 4 is substantially smaller than the difference between GPT 2 and GPT 3 Definitely not lol. The OG GPT-3 was pulling sub 50 on MMLU. Even benchmarks aside, there is a massive gap in utility between 3.5 and 4, never mind 3. 4 was finished training august 2022. It's only 2 years apart from 3. >I don't think progress is linear here. Rather, it seems more likely that we made the leap about a year or so ago, and are currently in the process of applying that leap in many different ways. But the leap happened, and there isn't seemingly another one coming. There was no special leap (in terms of theory and engineering). This is scale plainly laid out and there's more of it to go. >and Sam Altman has directly said there are no plans for a GPT 5. the same that sat on 4 for 8 months and said absolutely nothing about it ? Take anything altman says about new iterations with a grain of salt.

WendyTheWillow 3y ago
Firstly no, the gap between 3 and 4 is not anything as large as the gap between 2 and 3. Secondly, nothing you said here changed as of this announcement. Nothing here makes it any more or less likely LLMs will risk software engineering jobs. Thirdly, you can take what Sam Altman says with as many grains of salt as you like, if there really was no innovation at all as you claim, then there will be a limit hit at computing capability and cost.
- famouswaffles 3y ago
  >the gap between 3 and 4 is not anything as large as the gap between 2 and 3. We'll just have to agree to disagree. 3 was a signal of things to come but it was ultimately a bit of a toy, a research curiosity. Utility wise, they are worlds apart. >if there really was no innovation at all as you claim, then there will be a limit hit at computing capability and cost. computing capability and cost are just about the one thing you can bank on to reduce. already training gpt-4 today would be a fraction of the cost than it was when open ai did it and that was just over a year ago. Today's GPU's take ML into account to some degree but they are nowhere near as calibrated for it as they could be. That work has just begun to start. Of any of the possible barriers, compute is exactly the kind you want. It will fall.
  - WendyTheWillow 3y ago
    Do you realize I'm not disagreeing with you about the difference between 3 and 4? Reread what I wrote. I contrasted 3 and 4 with 2 and 3, which you seem to be entirely ignoring. 3 and 4 could be worlds apart, but wouldn't matter if 2 and 3 were two worlds apart, for example. And it is not true that computing power will continue to reduce; Moore's Law has been dead for some time now, and if incremental growth in LLMs require exponential growth in computing power the marginal difference won't matter. You would need a matching exponential growth in processing capability which is most certainly not occurring. So compute will not fall at the rate you would need it to for LLMs to actually compete in any meaningful way with human software engineers. We are not guaranteed to continue to progress in anything just because we have in the past.