3 ms·

More than that, I think people overestimate how much AI will progress as you throw more compute at it. It’s the “9 women can’t deliver a baby in a month” equiva

More than that, I think people overestimate how much AI will progress as you throw more compute at it. It’s the “9 women can’t deliver a baby in a month” equivalent of AI. Additional compute won’t magically give you AGI.

paytonjjones 3mo ago
Maybe not AGI, but if you look at the differences between, say, GPT-2 and GPT 5.5, it's remarkable how well it works to mostly just throw scale at the problem.
- codemog 3mo ago
  They already tried that with GPT-4 and GPT-4.5 They were allegedly massive but the cost and returns were not worth it.
- root_axis 3mo ago
  The difference is a lot more than just throwing scale at it, pretty much everything useful comes from an evolving landscape of post-training techniques. Of course, param count and context length are also important because they increase the model's overall fidelity, but a base model without SFT, RHLF etc is effectively useless.
  - darth_avocado 3mo ago
    Correct. That is what I was trying to hint at. Yes, massive compute is needed to train ai, but it isn’t the only thing. A lot of research and experimentation goes into moving the marker just a little bit. Innovation can’t be forced into weekly sprints, it takes its own time.
    - paytonjjones 3mo ago
      Research and experimentation on neural nets has been going on since the 70s (arguably much earlier even), but the lions share of capability changes has all been in the last couple years. Scale was really the unlock; the new pre and post training techniques and architectures are very cool and useful but they definitely aren't the differentiators when comparing to the previous era of NLP.
      - 8note 3mo ago
        which non-transformer neural networks are matching frontier performance using compute scale?
        paytonjjones 3mo ago
        BERT is a transformer! The unlock happened within transformers, yes, but they were not exactly super new or innovative architectures at that time. The scale was the main innovation that brought us from BERT to today's frontier.
      - emp17344 3mo ago
        I think the “unlock” is that AI firms were given trillions of dollars to discover new techniques. In fact, there are very few industries where a sudden influx of that much money would not lead to rapid advancements. It’s not really unique to the AI field.
        paytonjjones 3mo ago
        Your timeline is a bit backwards: they were given trillions of dollars only after they'd demonstrated a few pretty remarkable advancements. The fact that their advancement suggested that pouring more compute would continue working was also especially attractive to investors: it made a massive R&D budget feel like less of a risk.
      - ehnto 3mo ago
        I think the unlock only happens once though. I think that's where people are misled at the moment, the technology was there but required huge compute and data ingest to show improvement, but we have done that now. What's next for a giant leap is not more compute, and what new data we can provide now pales in comparison to that first ingest.
- bigstrat2003 3mo ago
  Not really. I didn't use GPT-2, but I don't think there was much difference between we got what out of GPT-3.5 and 5.5. It's still an unreliable tool which will go off the rails the second you aren't watching it like a hawk. It still has zero intelligence or ability to reason as to why its output is flawed. Just throwing more compute hasn't gotten us anything worth using.
- zdragnar 3mo ago
  I was under the impression that the deltas between versions were shrinking- i.e. gpt 4 -> 5 was much less impactful than 3 -> 4 or 2 -> 3. If the growth is getting diminishing returns, I can't say I'm optimistic without finding a drastically different approach.
  - esperent 3mo ago
    GPT 4/5 etc. are just marketing names though.
sroerick 3mo ago
The Allegorical Agent Aeon
IshKebab 3mo ago
How do you know though? People thought ANNs and perceptrons were basically useless until pretty much around the time of AlexNet and while there were some architectural improvements (CNNs, transformers), the biggest thing that changed was that we simply threw more compute at it. It's definitely too early to declare that more compute won't make a difference.
- noosphr 3mo ago
  It was data. Alexnet worked because imagenet existed. It took less than a year between the dataset becoming public and the first version of Alexnet being trained.

paytonjjones 3mo ago

Maybe not AGI, but if you look at the differences between, say, GPT-2 and GPT 5.5, it's remarkable how well it works to mostly just throw scale at the problem.

codemog 3mo ago
They already tried that with GPT-4 and GPT-4.5 They were allegedly massive but the cost and returns were not worth it.
root_axis 3mo ago
The difference is a lot more than just throwing scale at it, pretty much everything useful comes from an evolving landscape of post-training techniques. Of course, param count and context length are also important because they increase the model's overall fidelity, but a base model without SFT, RHLF etc is effectively useless.
- darth_avocado 3mo ago
  Correct. That is what I was trying to hint at. Yes, massive compute is needed to train ai, but it isn’t the only thing. A lot of research and experimentation goes into moving the marker just a little bit. Innovation can’t be forced into weekly sprints, it takes its own time.
  - paytonjjones 3mo ago
    Research and experimentation on neural nets has been going on since the 70s (arguably much earlier even), but the lions share of capability changes has all been in the last couple years. Scale was really the unlock; the new pre and post training techniques and architectures are very cool and useful but they definitely aren't the differentiators when comparing to the previous era of NLP.
    - 8note 3mo ago
      which non-transformer neural networks are matching frontier performance using compute scale?
      - paytonjjones 3mo ago
        BERT is a transformer! The unlock happened within transformers, yes, but they were not exactly super new or innovative architectures at that time. The scale was the main innovation that brought us from BERT to today's frontier.
    - emp17344 3mo ago
      I think the “unlock” is that AI firms were given trillions of dollars to discover new techniques. In fact, there are very few industries where a sudden influx of that much money would not lead to rapid advancements. It’s not really unique to the AI field.
      - paytonjjones 3mo ago
        Your timeline is a bit backwards: they were given trillions of dollars only after they'd demonstrated a few pretty remarkable advancements. The fact that their advancement suggested that pouring more compute would continue working was also especially attractive to investors: it made a massive R&D budget feel like less of a risk.
    - ehnto 3mo ago
      I think the unlock only happens once though. I think that's where people are misled at the moment, the technology was there but required huge compute and data ingest to show improvement, but we have done that now. What's next for a giant leap is not more compute, and what new data we can provide now pales in comparison to that first ingest.
bigstrat2003 3mo ago
Not really. I didn't use GPT-2, but I don't think there was much difference between we got what out of GPT-3.5 and 5.5. It's still an unreliable tool which will go off the rails the second you aren't watching it like a hawk. It still has zero intelligence or ability to reason as to why its output is flawed. Just throwing more compute hasn't gotten us anything worth using.
zdragnar 3mo ago
I was under the impression that the deltas between versions were shrinking- i.e. gpt 4 -> 5 was much less impactful than 3 -> 4 or 2 -> 3. If the growth is getting diminishing returns, I can't say I'm optimistic without finding a drastically different approach.
- esperent 3mo ago
  GPT 4/5 etc. are just marketing names though.

sroerick 3mo ago

The Allegorical Agent Aeon

IshKebab 3mo ago

How do you know though? People thought ANNs and perceptrons were basically useless until pretty much around the time of AlexNet and while there were some architectural improvements (CNNs, transformers), the biggest thing that changed was that we simply threw more compute at it. It's definitely too early to declare that more compute won't make a difference.

noosphr 3mo ago
It was data. Alexnet worked because imagenet existed. It took less than a year between the dataset becoming public and the first version of Alexnet being trained.