4 ms·

"It's a lemon"–OpenAI's largest AI model ever arrives to mixed reviews

5 points by kiyanwang 2y ago

deleted 2y ago
[deleted]
_mitterpach 2y ago
I think you could see this coming as soon as they implemented chain-of-thought in their previous models. Chain-of-thought is well understood to be the way to squeeze out performance out of these models. Slow down, go step by step, use more tokens to get a little bit better output. This is so useful, that often in the comparision graphs you see between different models, the best one (the one I want to sell) is using chain of thought while others are not. Not always, but companies have been caught using this technique to hype up their models. We’ve already seen that CoT (usually!) improves a model’s performance. So does prompting the model with examples of correct question-answer pairs (called in-context examples). But in reports, some models are evaluated with CoT, while others aren’t. The number of in-context examples is often different, and the prompts are almost always different. https://asteriskmag.com/issues/07/can-you-trust-an-ai-press-release https://asteriskmag.com/issues/07/can-you-trust-an-ai-press-... As soon as OpenAI was desparate for performance upgrade, they implemented this as an actual feature.
lompad 2y ago
So where's that promised exponential improvement? Oh right, it doesn't exist. The only thing that exponentially increases are the costs and no real, sustainable business model is in sight. AI companies have no moat and are ironically consistently threatened to be made redundant. Businesses keep running around with the solution that is genAI but can barely find any problems to use it on. We won't be getting what sama promised, that much is clear, I'd say. Thankfully. I do think it's time to prepare for the post-AI-bubble age. Big changes are coming, after all, hundreds of billions have been wasted for what's essentially a toy.
- _mitterpach 2y ago
  Honestly, imagine being in a regular software field and improving your product 2x. It is now two times better than it was previously. That's amazing! You could probably beat out all the competition, steal all the customers, quadruple your profits. OpenAI and every other peddler keeps promising upgrades in the magnitude of 100-200x. That's impossible. To be affordable and usable enough, they would have to come up with actual miracles. At this point, they've picked off all the low hanging fruit. At best, we're going to be seeing marginal improvements at the cost of a large price increase (see latest OpenAI flagship). There is some usefulness, but after so many dollars wasted, all companies are going to be left with is a bunch of useless GPUs. I can't wait.
Ukv 2y ago
> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scaling laws" cited by many for years have possibly met their natural end. The commonly-cited scaling laws[0] predict diminishing returns from scale. Fairly uncontroversial for instance that going from 1 GPU to 2 GPUs gives a larger improvement than going from 101 GPUs to 102 GPUs. Same with, say, computer graphics. I feel scaling laws are getting conflated with the theory of an exponential "hard takeoff" of self-improvement (which isn't particuarly well-founded in my opinion). [0]: https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361

5 points by kiyanwang 2y ago

deleted 2y ago

[deleted]

I think you could see this coming as soon as they implemented chain-of-thought in their previous models. Chain-of-thought is well understood to be the way to squeeze out performance out of these models. Slow down, go step by step, use more tokens to get a little bit better output. This is so useful, that often in the comparision graphs you see between different models, the best one (the one I want to sell) is using chain of thought while others are not. Not always, but companies have been caught using this technique to hype up their models. We’ve already seen that CoT (usually!) improves a model’s performance. So does prompting the model with examples of correct question-answer pairs (called in-context examples). But in reports, some models are evaluated with CoT, while others aren’t. The number of in-context examples is often different, and the prompts are almost always different. https://asteriskmag.com/issues/07/can-you-trust-an-ai-press-release https://asteriskmag.com/issues/07/can-you-trust-an-ai-press-... As soon as OpenAI was desparate for performance upgrade, they implemented this as an actual feature.

lompad 2y ago

So where's that promised exponential improvement? Oh right, it doesn't exist. The only thing that exponentially increases are the costs and no real, sustainable business model is in sight. AI companies have no moat and are ironically consistently threatened to be made redundant. Businesses keep running around with the solution that is genAI but can barely find any problems to use it on. We won't be getting what sama promised, that much is clear, I'd say. Thankfully. I do think it's time to prepare for the post-AI-bubble age. Big changes are coming, after all, hundreds of billions have been wasted for what's essentially a toy.

_mitterpach 2y ago
Honestly, imagine being in a regular software field and improving your product 2x. It is now two times better than it was previously. That's amazing! You could probably beat out all the competition, steal all the customers, quadruple your profits. OpenAI and every other peddler keeps promising upgrades in the magnitude of 100-200x. That's impossible. To be affordable and usable enough, they would have to come up with actual miracles. At this point, they've picked off all the low hanging fruit. At best, we're going to be seeing marginal improvements at the cost of a large price increase (see latest OpenAI flagship). There is some usefulness, but after so many dollars wasted, all companies are going to be left with is a bunch of useless GPUs. I can't wait.

Ukv 2y ago

> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scaling laws" cited by many for years have possibly met their natural end. The commonly-cited scaling laws[0] predict diminishing returns from scale. Fairly uncontroversial for instance that going from 1 GPU to 2 GPUs gives a larger improvement than going from 101 GPUs to 102 GPUs. Same with, say, computer graphics. I feel scaling laws are getting conflated with the theory of an exponential "hard takeoff" of self-improvement (which isn't particuarly well-founded in my opinion). [0]: https://arxiv.org/pdf/2001.08361 https://arxiv.org/pdf/2001.08361