5 ms·

how is this a plateau since gpt-4? this is significantly better

kenjackson 2y ago
People act as if GPT-4 came out 10 years ago.
csomar 2y ago
First, this model is yet to be released. This is a momentum "announcement". When the O1 was "announced", it was announced as a "breakthrough" but I use Claude/O1 daily and 80% of the time Claude beats it. I also see it as a highly fine-tuned/targeted GPT-4 rather than something that has complex understanding. So we'll find out if this model is real or not by 2-3 months. My guess is that it'll turn out to be another flop like O1. They needed to release something big because they are momentum based and their ability to raise funding is contingent on their AGI claims.
- XenophileJKO 2y ago
  I thought o1 was a fine-tune of GPT-4o. I don't think o3 is though. Likely using the same techniques on what would have been the "GPT-5" base model.
Jensson 2y ago
> how is this a plateau since gpt-4? this is significantly better Significantly better at what? A benchmark? That isn't necessarily progress. Many report preferring gpt-4 to the newer o1 models with hidden text. Hidden text makes the model more reliable, but more reliable is bad if it is reliably wrong at something since then you can't ask it over and over to find what you want. I don't feel it is significantly smarter, it is more like having the same dumb person spend more thinking than the model getting smarter.
peepeepoopoo97 2y ago
O3 is multiple orders of magnitude more expensive to realize a marginal performance gain. You could hire 50 full time PhDs for the cost of using O3. You're witnessing the blowoff top of the scaling hype bubble.
- whynotminot 2y ago
  What they’ve proven here is that it can be done. Now they just have to make it cheap. Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. Are you seriously going to assume that won’t happen here?
  - Jensson 2y ago
    > What they’ve proven here is that it can be done. No they haven't, these results do not generalize, as mentioned in the article: "Furthermore, early data points suggest that the upcoming ARC-AGI-2 benchmark will still pose a significant challenge to o3, potentially reducing its score to under 30% even at high compute" Meaning, they haven't solved AGI, and the task itself do not represent programming well, these model do not perform that well on engineering benchmarks.
    - whynotminot 2y ago
      Sure, AGI hasn’t been solved today. But what they’ve done is show that progress isn’t slowing down. In fact, it looks like things are accelerating. So sure, we’ll be splitting hairs for a while about when we reach AGI. But the point is that just yesterday people were still talking about a plateau.
      - peepeepoopoo97 2y ago
        About 10,000 times the cost for twice the performance sure looks like progress is slowing to me.
        whynotminot 2y ago
        Just to be clear — your position is that the cost of inference for o3 will not go down over time (which would be the first time that has happened for any of these models).
crazylogger 2y ago
Intelligence has not been LLM's major limiting factor since GPT4. The original GPT4 reports in late-2022 & 2023 already established that it's well beyond an average human in professional fields: https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/ https://www.microsoft.com/en-us/research/publication/sparks-.... They failed to outright replaced humans at work not because of lacking intelligence. We may have progressed from a 99%-accurate chatbot to one that's 99.9%-accurate, and you'd have a hard time telling them apart in normal real world (dumb) applications. A paradigm shift is needed from the current chatbot interface to a long-lived stream of consciousness model (e.g. a brain that constantly reads input and produces thoughts at 10ms refresh rate; remembers events for years and keep the context window from exploding; paired with a cerebellum to drive robot motors, at even higher refresh rates.) As long as we're stuck at chatbots, LLM's impact on the real world will be very limited, regardless of how intelligent they become.

People act as if GPT-4 came out 10 years ago.

First, this model is yet to be released. This is a momentum "announcement". When the O1 was "announced", it was announced as a "breakthrough" but I use Claude/O1 daily and 80% of the time Claude beats it. I also see it as a highly fine-tuned/targeted GPT-4 rather than something that has complex understanding. So we'll find out if this model is real or not by 2-3 months. My guess is that it'll turn out to be another flop like O1. They needed to release something big because they are momentum based and their ability to raise funding is contingent on their AGI claims.

XenophileJKO 2y ago
I thought o1 was a fine-tune of GPT-4o. I don't think o3 is though. Likely using the same techniques on what would have been the "GPT-5" base model.

Jensson 2y ago

> how is this a plateau since gpt-4? this is significantly better Significantly better at what? A benchmark? That isn't necessarily progress. Many report preferring gpt-4 to the newer o1 models with hidden text. Hidden text makes the model more reliable, but more reliable is bad if it is reliably wrong at something since then you can't ask it over and over to find what you want. I don't feel it is significantly smarter, it is more like having the same dumb person spend more thinking than the model getting smarter.

peepeepoopoo97 2y ago

O3 is multiple orders of magnitude more expensive to realize a marginal performance gain. You could hire 50 full time PhDs for the cost of using O3. You're witnessing the blowoff top of the scaling hype bubble.

whynotminot 2y ago
What they’ve proven here is that it can be done. Now they just have to make it cheap. Tell me, what has this industry been good at since its birth? Driving down the cost of compute and making things more efficient. Are you seriously going to assume that won’t happen here?
- Jensson 2y ago
  > What they’ve proven here is that it can be done. No they haven't, these results do not generalize, as mentioned in the article: "Furthermore, early data points suggest that the upcoming ARC-AGI-2 benchmark will still pose a significant challenge to o3, potentially reducing its score to under 30% even at high compute" Meaning, they haven't solved AGI, and the task itself do not represent programming well, these model do not perform that well on engineering benchmarks.
  - whynotminot 2y ago
    Sure, AGI hasn’t been solved today. But what they’ve done is show that progress isn’t slowing down. In fact, it looks like things are accelerating. So sure, we’ll be splitting hairs for a while about when we reach AGI. But the point is that just yesterday people were still talking about a plateau.
    - peepeepoopoo97 2y ago
      About 10,000 times the cost for twice the performance sure looks like progress is slowing to me.
      - whynotminot 2y ago
        Just to be clear — your position is that the cost of inference for o3 will not go down over time (which would be the first time that has happened for any of these models).

crazylogger 2y ago

Intelligence has not been LLM's major limiting factor since GPT4. The original GPT4 reports in late-2022 & 2023 already established that it's well beyond an average human in professional fields: https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/ https://www.microsoft.com/en-us/research/publication/sparks-.... They failed to outright replaced humans at work not because of lacking intelligence. We may have progressed from a 99%-accurate chatbot to one that's 99.9%-accurate, and you'd have a hard time telling them apart in normal real world (dumb) applications. A paradigm shift is needed from the current chatbot interface to a long-lived stream of consciousness model (e.g. a brain that constantly reads input and produces thoughts at 10ms refresh rate; remembers events for years and keep the context window from exploding; paired with a cerebellum to drive robot motors, at even higher refresh rates.) As long as we're stuck at chatbots, LLM's impact on the real world will be very limited, regardless of how intelligent they become.