5 ms·
There seem to still be a lot of people who look at results like this and evaluate them purely based on the current state. I don't know how you can look at this
by modeless 8mo ago
There seem to still be a lot of people who look at results like this and evaluate them purely based on the current state. I don't know how you can look at this and not realize that it represents a huge improvement over just a few months ago, there have been continuous improvements for many years now, and there is no reason to believe progress is stopping here. If you project out just one year, even assuming progress stops after that, the implications are staggering.
- nozzlegear 8mo agoEvery S-curve looks like an exponential until you hit the bend.
- NitpickLawyer 8mo agoWe've been hearing this for 3 years now. And especially 25 was full of "they've hit a wall, no more data, running out of data, plateau this, saturated that". And yet, here we are. Models keep on getting better, at more broad tasks, and more useful by the month.
- nozzlegear 8mo ago> We've been hearing this for 3 years now Not from me you haven't! > "they've hit a wall, no more data, running out of data, plateau this, saturated that" Everyone thought Moore's Law was infallible too, right until they hit that bend. What hubris to think these AI models are different! But you've probably been hearing that for 3 years too (though not from me). > Models keep on getting better, at more broad tasks, and more useful by the month. If you say so, I'll take your word for it.
- Cyphase 8mo ago25 is 2025.
- nozzlegear 8mo agoOh my bad, the way it was worded made me read it as the name of somebody's model or something.
- torginus 8mo agoExcept for Moore's law, everyone knew decades ahead of what the limits of Dennard scaling are (shrinking geometry through smaller optical feature sizes), and roughly when we would get to the limit. Since then, all improvements came at a tradeoff, and there was a definite flattening of progress.
- nozzlegear 8mo ago> Since then, all improvements came at a tradeoff, and there was a definite flattening of progress. Idk, that sounds remarkably similar to these AI models to me.
- kijiki 8mo agoEveryone? Intel, at the time the unquestioned world leader in semiconductor fabrication was so unable to accurately predict the end of Dennard scaling that they rolled out the Pentium 4. "10Ghz by 2010!" was something they predicted publicly in earnest! It, uhhh, didn't quite work out that way.
- fmbb 8mo ago> And yet, here we are. I dunno. To me it doesn’t even look exponential any more. We are at most on the straight part of the incline.
- deleted 8mo ago[deleted]
- sdf2erf 8mo agoPersonally my usage has fell off a cliff the past few months. Im not a SWE. SWE's may be seeing benefit. But in other areas? Doesnt seem to be the case. Consumers may use it as a more preferred interface for search - but this is a different discussion.
- kelnos 8mo agoYes, and Moore's law took decades to start to fail to be true. Three years of history isn't even close to enough to predict whether or not we'll see exponential improvement, or an unsurmountable plateau. We could hit it in 6 months or 10 years, who knows. And at least with Moore's law, we had some understanding of the physical realities as transistors would get smaller and smaller, and reasonably predict when we'd start to hit limitations. With LLMs, we just have no idea. And that could be go either way.
- LinXitoW 8mo agoModel improvement is very much slowing down, if we actually use fair metrics. Most improvements in the last year or so comes down to external improvements, like better tooling, or the highly sophisticated practice of throwing way more tokens at the same problem (reasoning and agents). Don't get me wrong, LLMs are useful. They just aren't the kind of useful that Sam et al. sold investors. No AGI, no full human worker replacement, no massive reduction in cost for SOTA.
- raincole 8mo agoThis quote would be more impactful if people haven't been repeating it since gpt-4 time.
- nozzlegear 8mo agoI agree, I have been informed that people have been repeating it for three years. Sadly I'm not involved in the AI hype bubble so I wasn't aware. What an embarrassing faux pas.
- kimixa 8mo agoPeople have also been saying we'd be seeing the results of 100x quality improvements in software with corresponding decease in cost since gpt-4 time. So where is that?
- famouswaffles 8mo agoCool I guess. Kind of a meaningless statement yeah? Let's hit the bend, then we'll talk. Until then repeating, 'It's an S Curve guys and what's more, we're near the bend! trust me" ad infinitum is pointless. It's not some wise revelation lol.
- smj-edison 8mo agoMaybe the best thing to say is we can only really forecast about 3 months out accurately, and the rest is wild speculation :) History has a way of being surprisingly boring, so personally I'm not betting on the world order being transformed in five years, but I also have to take my own advice and take things a day at a time.
- nozzlegear 8mo ago> Kind of a meaningless statement yeah? If you say so. It's clear you think these marketing announcements are still "exponential improvements" for some reason, but hey, I'm not an AI hype beast so by all means keep exponentialing lol
- famouswaffles 8mo agoI'm not asking you to change your belief. By all means, think we're just around the corner of a plateau, but like I said, your statement is nothing meaningful or profound. It's your guess that things are about to slow down, that's all. It's better to just say that rather than talking about S curves and bends like you have any more insight than OP.
- deleted 8mo ago[deleted]
- esafak 8mo agoWhat if it plateaus smarter than us? You wouldn't be able to discern where it stopped. I'm not convinced it won't be able to create its own training data to keep improving. I see no ceiling on the horizon, other than energy.
- nozzlegear 8mo agoAre we talking Terminator or Matrix here? I need to know which shitty future to prepare for.
- esafak 8mo agoUsing humans as batteries makes no sense. I expect robots will know better than that.
- zamadatix 8mo agoThe improvements in tool use and agentic loops have been fast and furious lately, delivering great results. The model growth itself is feeling more "slow and linear" lately, but what you can do with models as part of an overall system has been increasing in growth rate and that has been delivering a lot of value. It matters less if the model natively can keep infinite context or figure things out on its own in one shot so long as it can orchestrate external tools to achieve that over time.
- chasd00 8mo agoi have to admit, even if model and tooling progress stopped dead today the world of software development has forever changed and will never go back.
- uywykjdskn 8mo agoYea the software engineering profession is over, even if all improvements stop now.
- LinXitoW 8mo agoThe main issue with improvements in the last year is that a lot of it is based not on the models strictly becoming better, but on tooling being better, and simply using a fuckton more tokens for the same task. Remember that all these companies can only exist because of massive (over)investments in the hope of insane returns and AGI promises. While all these improvements (imho) prove the exact opposite: AGI is absolutely not coming, and the investments aren't going to generate these outsized returns. The will generate decent returns, and the tools are useful.
- modeless 8mo agoI disagree. A year ago the models would not come close to doing this, no matter what tools you gave them or how many tokens you generated. Even three months ago. Effectively using tools to complete long tasks required huge improvements in the models themselves. These improvements were driven not by pretraining like before, but by RL with verifiable rewards. This can continue to scale with training compute for the foreseeable future, eliminating the "data wall" we were supposed to be running into.