6 ms·
> I don't understand why Hacker News is so dismissive about the coming of LLMs I find LLMs incredibly useful, but if you were following along the last few year
by crystal_revenge 10mo ago
> I don't understand why Hacker News is so dismissive about the coming of LLMs
I find LLMs incredibly useful, but if you were following along the last few years the promise was for “exponential progress” with a teaser world destroying super intelligence.
We objectively are not on that path. There is no “coming of LLMs”. We might get some incremental improvement, but we’re very clearly seeing sigmoid progress.
I can’t speak for everyone, but I’m tired of hyperbolic rants that are unquestionably not justified (the nice thing about exponential progress is you don’t need to argue about it)
- aoeusnth1 10mo agoWe're very clearly seeing exponential progress - even above trend, on METR, whose slope keeps getting revised to a higher and higher estimate each time. Explain your perspective on the objective evidence against exponential progress?
- llmslave2 10mo agoPretty neat how this exponential progress hasn't resulted in exponential productivity. Perhaps you could explain your perspective on that?
- viraptor 10mo agoWriting the code itself was never the main bottleneck. Designing the bigger solution, figuring out tradeoffs, taking to affected teams, etc. takes as much time as it used to. But still, there's definitely a significant improvement in code production part in many areas.
- aoeusnth1 10mo agoIt has! CLs/engineer increased by 10% this year. LLMs from late 2024 were nearly worthless as coding agents, so given they have quadrupled in capability since then (exponential growth, btw), it's not surprising to see a modestly positive impact on SWE work. Also, I'm noticing you're not explaining yourself :)
- llmslave2 10mo agoHey, I'm not the OG commentator, why do I have to explain myself! :) When Fernando Alonso (best rookie btw) goes from 0-60 in 2.4 seconds in his Aston Martin, is it reasonable to assume he will near the speed of light in 20 seconds?
- aoeusnth1 10mo agoIf you're not going to explain yourself, at least stay on topic. We're talking about exponential growth, so address the points I'm making.
- lopatin 10mo ago> Hey, I'm not the OG commentator, why do I have to explain myself! :) The issue is that you're not acknowledging or replying to people's explanations for _why_ they see this as exponential growth. It's almost as if you skimmed through the meat of the comment and then just re-phrased your original idea. > When Fernando Alonso (best rookie btw) goes from 0-60 in 2.4 seconds in his Aston Martin, is it reasonable to assume he will near the speed of light in 20 seconds? This comparison doesn't make sense because we know the limits of cars but we don't yet know the limits of LLMs. It's an open question. Whether or not an F1 engine can make it the speed of light in 20 seconds is not an open question.
- llmslave2 10mo agoIt's not in me to somehow disprove claims of exponential growth when there isn't even evidence provided of it. My point with the F1 comparison is to say that a short period of rapid improvement doesn't imply exponential growth and it's about as weird to expect that as it is for an f1 car to reach the speed of light. It's possible you know, the regulations are changing for next season - if Leclerc sets a new lap record in Australia by .1 ms we can just assume exponential improvements and surely Ferrari will be lapping the rest of the field by the summer right?
- aoeusnth1 10mo ago
- HPMOR 10mo agoI think this is an open question still and very interesting. Ilya discussed this on the Dwarkesh podcast. But the capabilities of LLMs is clearly exponential and perhaps super exponential. We went from something that could string together incoherent text in 2022 to general models helping people like Terrance Tao and Scott Aaronson write new research papers. LLMs also beat IMO and the ICPC. We have entered the John Henry era for intellectual tasks...
- llmslave2 10mo ago> But the capabilities of LLMs is clearly exponential and perhaps super exponential By what metric?
- utopiah 10mo agoBS metric... /s
- jennyholzer3 10mo agoChat GPT told him it was true
- aspenmartin 10mo ago- Scaling laws (Chinchilla type) - METR task horizon It's a mix, performance gains are bursty but we have been getting a lot of bursts (RLVR, test-time compute, agentic breakthroughs)
- tsimionescu 10mo ago> LLMs also beat IMO and the ICPC Very spurious claims, given that there was no effort made to check whether the IMO or ICPC problems were in the training set or not, or to quantify how far problems in the training set were from the contest problems. IMO problems are supposed to be unique, but since it's not at the frontier of math research, there is no guarantee that the same problem, or something very similar, was not solved in some obscure manual.
- mgfist 10mo agoBecause that requires adoption. Devs on hackernews are already the most up to date folks in the industry and even here adoption of LLMs is incredibly slow. And a lot of the adoption that does happen is still with older tech like ChatGPT or Cursor.
- belmont_sup 10mo agoWhat’s the newer tech?
- TeodorDyakov 10mo agoClaude Code With Opus 4.5
- scotty79 10mo agoHow long before introduction of computers lead to increases in average productivity? How long for the internet? Business is just slow to figure out how to use anything for its benefit, but it eventually gets there.
- fmbb 10mo ago> How long before introduction of computers lead to increases in average productivity? I think it never did. Still has not. https://en.wikipedia.org/wiki/Productivity_paradox https://en.wikipedia.org/wiki/Productivity_paradox
- spectralista 10mo agoThe best example is that even ATM machines didn't reduce bank teller jobs. Why? Because even the bank teller is doing more than taking and depositing money. IMO there is an ontological bias that pervades our modern society that confuses the map for the territory and has a highly distorted view of human existence through the lens of engineering. We don't see anything in this time series, because this time series itself is meaningless nonsense that reflects exactly this special kind of ontological stupidity: https://fred.stlouisfed.org/series/PRS85006092 https://fred.stlouisfed.org/series/PRS85006092 As if the sum of human interaction in an economy is some kind of machine that we just need to engineer better parts for and then sum the outputs. Any non-careerist, thinking person that studies economics would conclude we don't and will probably not have the tools to properly study this subject in our lifetimes. The high dimensional interaction of biology, entropy and time. We have nothing. The career economist is essentially forced to sing for their supper in a type of time series theater. Then there is the method acting of pretending to be surprised when some meaningless reductionist aspect of human interaction isn't reflected in the fake time series.
- barrenko 10mo agoSir, we're in a modern economy, we don't ever ever look at productivity graphs (this is not to disparage LLMs, just a comment on productivity in general)
- viraptor 10mo ago> exponential progress First you need to define what it means. What's the metric? Otherwise it's very much something you can argue about.
- noodletheworld 10mo ago> What's the metric? Language model capability at generating text output. The model progress this year has been a lot of: - “We added multimodal” - “We added a lot of non AI tooling” (ie agents) - “We put more compute into inference” (ie thinking mode) So yes, there is still rapid progress, but these ^ make it clear, at least to me, that next gen models are significantly harder to build. Simultaneously we see a distinct narrowing between players (openai, deepseek, mistral, google, anthropic) in their offerings. Thats usually a signal that the rate of progress is slowing. Remind me what was so great about gpt 5? How about gpt4 from from gpt 3? Do you even remember the releases? Yeah. I dont. I had to look it up. Just another model with more or less the same capabilities. “Mixed reception” That is not what exponential progress looks like, by any measure. The progress this year has been in the tooling around the models, smaller faster models with similar capabilities. Multimodal add ons that no one asked for, because its easier to add image and audio processing than improve text handling. That may still be on a path to AGI, but it not an exponential path to it.
- dragonwriter 10mo ago> Language model capability at generating text output. That's not a metric, that's a vague non-operationalized concept, that could be operationalized into an infinite number of different metrics. And an improvement that was linear in one of those possible metrics would be exponential in another one (well, actually, one that is was linear in one would also be linear in an infinite number of others, as well as being exponential in an infinite number of others. That’s why you have to define an actual metric, not simply describe a vague concept of a kind of capacity of interest, before you can meaningfully discuss whether improvement is exponential. Because the answer is necessarily entirely dependent on the specific construction of the metric.
- viraptor 10mo ago
- scotty79 10mo ago> but we’re very clearly seeing sigmoid progress. Yeah, probably. But no chart actually shows it yet. For now we are firmly in exponential zone of the signoid curve and can't really tell if it's going to end in a year, decade or a century.
- utopiah 10mo agoDoesn't even matter if the goal is extremely high. Talking about exponential when we clearly see matching energy needs proves there is no way we can maintain that pace without radical (and thus unpredictable) improvements. My own "feeling" is that it's definitely not exponential but again, doesn't matter if it's unsustainable.
- jennyholzer3 10mo ago[flagged]
- fullstackchris 10mo agoI wrote an article complaining about the whole hype over a year ago: https://chrisfrewin.medium.com/why-llms-will-never-be-agi-70335d452bd7 https://chrisfrewin.medium.com/why-llms-will-never-be-agi-70... Seems to be playing out that way.
- senordevnyc 10mo agoI’ve been reading this comment multiple times a week for the last couple years. Constant assertions that we’re starting to hit limits, plateau, etc. But a cursory glance at where we are today vs a year ago, let alone two years ago, makes it wildly obvious that this is bullshit. The pace of improvement of both models and tooling has been breathtaking. I could give a shit whether you think it’s “exponential”, people like you were dismissing all of this years ago, meanwhile I just keep getting more and more productive.
- qualifck 10mo agoPeople keep saying stuff like this. That the improvements are so obvious and breathtaking and astronomical and then I go check out the frontier LLMs again and they're maybe a tiny bit better than they were last year but I can't actually be sure bcuz it's hard to tell. sometimes it seems like people are just living in another timeline.
- senordevnyc 10mo agoI’m genuinely curious what your “checking the frontier LLMs” looks like, especially if you haven’t used AI since last year.
- jennyholzer3 10mo ago"maybe a tiny bit better" is what you say when you've been tricked by snake oil salesman This shit has gotten worse since 2023.
- aspenmartin 10mo ago> This shit has gotten worse since 2023. I would really appreciate it if people could be specific when they say stuff like this because it's so crazy out of line with all measurement efforts. There are an insane amount of serious problems with current LLM / agentic paradigms, but the idea that things have gotten worse since 2023? I mean come on.
- 10mo ago
- aspenmartin 10mo agoI'm not sure I understand: we are _objectively on that path_ -- we are increasing exponentially on a number of metrics that may be imperfect but seem to paint a pretty consistent picture. Scaling laws are exponential. METR's time horizon benchmark is exponential. Lots of performance measures are exponential, so why do you say we're objectively not on that path? > We might get some incremental improvement, but we’re very clearly seeing sigmoid progress. again, if it is "very clear" can you point to some concrete examples to illustrate what you mean? > I can’t speak for everyone, but I’m tired of hyperbolic rants that are unquestionably not justified (the nice thing about exponential progress is you don’t need to argue about it) OK but what specifically do you have an issue with here?
- tim333 10mo ago>following along the last few years the promise was for “exponential progress” I've been following for many years and the main exponential thing has been the Moore's law like growth in compute. Compute per dollar is probably the best tracking one and has done a steady doubling every couple of years or so for decades. It's exponential but quite a leisurely exponential. The recent hype of the last couple of years is more dot com bubble like and going ahead of trend but will quite likely drop back.