18 ms·

Recent AI model progress feels mostly like bullshit

• claytron5000 2y ago
  [dead]
• guluarte 2y ago
  current AI models are searching engines that can somewhat extrapolate data and use in the current context, that's pretty much it
• deleted 2y ago
  [deleted]
• boxed 2y ago
  > So maybe there's no mystery: The AI lab companies are lying, and when they improve benchmark results it's because they have seen the answers before and are writing them down. [...then says maybe not...] Well.. they've been caught again and again red handed doing exactly this. Fool me once shame on you, fool me 100 times shame on me.
  • smnplk 2y ago
    Fool me once, shame on you...If fooled, you cant get fooled again. https://www.youtube.com/shorts/LmFN8iENTPc https://www.youtube.com/shorts/LmFN8iENTPc
  • drdaeman 2y ago
    Hate to say this but the incentive is growth, not progress. Progress is what enabled the growth, but is also extremely hard to plan and deliver. On the other hand, hype is probably somewhat easier and well-tested approach so no surprise lot of the effort goes into marketing. Markets had repeatedly confirmed that there aren't any significant immediate repercussions for cranking up BS levels in marketing materials, while there are some rewards when it works.
• fxtentacle 2y ago
  I'd say most of the recent AI model progress has been on price. A 4-bit quant of QwQ-32B is surprisingly close to Claude 3.5 in coding performance. But it's small enough to run on a consumer GPU, which means deployment price is now down to $0.10 per hour. (from $12+ for models requiring 8x H100)
  • shostack 2y ago
    Yeah, I'm thinking of this from a Wardley map standpoint. What innovation opens up when AI gets sufficiently commoditized?
    • mentalgear 2y ago
      Brute force, brute force everything at least for the domains you can have automatic verification in.
    • bredren 2y ago
      One thing I’ve seen is large enterprises extracting money from consumers by putting administrative burden on them. For example, you can see this in health insurance reimbursements and wireless carriers plan changes. (ie, Verizon’s shift from Do More, etc to what they have now) Companies basically set up circumstances where consumers lose small amounts of money on a recurring basis or sporadically enough that the people will just pay the money rather than a maze of calls, website navigation and time suck to recover funds due to them or that shouldn’t have been taken in the first place. I’m hopeful well commoditized AI will give consumers a fighting chance at this and other types of disenfranchisement that seems to be increasingly normalized by companies that have consultants that do nothing but optimize for their own financial position.
  • xiphias2 2y ago
    Have you compared it with 8-bit QwQ-17B? In my evals 8 bit quantized smaller Qwen models were better, but again evaluating is hard.
    • redrove 2y ago
• softwaredoug 2y ago
  I think the real meaningful progress is getting ChatGPT 3.5 level quality running anywhere you want rather than AIs getting smarter at high level tasks. This capability being ubiquitous and not tied to one vendor is really what’s revolutionary.
• gundmc 2y ago
  This was published the day before Gemini 2.5 was released. I'd be interested if they see any difference with that model. Anecdotally, that is the first model that really made me go wow and made a big difference for my productivity.
  • jonahx 2y ago
    I doubt it. It still flails miserably like the other models on anything remotely hard, even with plenty of human coaxing. For example, try to get it to solve: https://www.janestreet.com/puzzles/hall-of-mirrors-3-index/ https://www.janestreet.com/puzzles/hall-of-mirrors-3-index/
    • Xenoamorphous 2y ago
      I’d say the average person wouldn’t understand that problem, let alone solve it.
    • flutas 2y ago
      FWIW 2.5-exp was the only one that managed to get a problem I asked it right, compared to Claude 3.7 and o1 (or any of the other free models in Cursor). It was reverse engineering ~550MB of Hermes bytecode from a react native app, with each function split into a separate file for grep-ability and LLM compatibility. The others would all start off right then quickly default to just greping randomly what they expected it to be, which failed quickly. 2.5 traced the function all the way back to the networking call and provided the expected response payload. All the others hallucinated the networking response I was trying to figure out. 2.5 Provided it exactly enough for me to intercept the request and using the response it provided to get what I wanted to show up.
      • arkmm 2y ago
        How did you fit 550MB of bytecode into the context window? Was this using 2.5 in an agentic framework? (i.e. repeated model calls and tool usage)
        • 2y ago
• djha-skin 2y ago
  > Since 3.5-sonnet, we have been monitoring AI model announcements, and trying pretty much every major new release that claims some sort of improvement. Unexpectedly by me, aside from a minor bump with 3.6 and an even smaller bump with 3.7, literally none of the new models we've tried have made a significant difference on either our internal benchmarks or in our developers' ability to find new bugs. This includes the new test-time OpenAI models. This is likely a manifestation of the bitter lesson[1], specifically this part: > The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project [like an incremental model update], massively more computation inevitably becomes available. (Emphasis mine.) Since the ultimate success strategy of the scruffies[2] or proponents of search and learning strategies in AI is Moore's Law, short term gains using these strategies will be miniscule. It is over at least a five year period that their gains will be felt the most. The neats win the day in the short term, but the hare in this race will ultimately give away to the steady plod of the tortoise. 1: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html 2: https://en.m.wikipedia.org/wiki/Neats_and_scruffies#CITEREFMcCorduck2004 https://en.m.wikipedia.org/wiki/Neats_and_scruffies#CITEREFM...
• ohgr 2y ago
  It’s not even approaching the asymptotic line of promises made at any achievable rate for the amount of cash being thrown at it. Where’s the business model? Suck investors dry at the start of a financial collapse? Yeah that’s going to end well…
  • maccard 2y ago
    > where’s the business model? For who? Nvidia sell GPUs, OpenAI and co sell proprietary models and API access, and the startups resell GPT and Claude with custom prompts. Each one is hoping that the layer above has a breakthrough that makes their current spend viable. If they do, then you don’t want to be left behind, because _everything_ changes. It probably won’t, but it might. That’s the business model
    • grey-area 2y ago
      That’s not a business model, it’s a pipe dream. This bubble will be burst by the Trump tariffs and the end of the zirp era. When inflation and a recession hit together hope and dream business models and valuations no longer work.
      • maccard 2y ago
        Which one? Nvidia are doing pretty ok selling GPU's, and OpenAI and Anthropic are doing ok selling their models. They're not _viable_ business models, but they could be.
        • grey-area 2y ago
          NVDA will crash when the AI bubble implodes, and none of those Generative AI companies are actually making money, nor will they. They have already hit limiting returns in LLM improvements after staggering investments and it is clear are nowhere near general intelligence.
          • maccard 2y ago
• paulsutter 2y ago
  Im able to get substantially more coding done than three months ago. This could be largely in the tooling (coding agents, deep research). But the models are better too, for both coding and brainstorming. And tooling counts, to me, as progress. Learning to harness current tools helps to harness future tools. Work on projects that will benefit from advancements, but can succeed without them.
  • dghlsakjg 2y ago
    I'm not sure if I'm able to do more of the hard stuff, but a lot of the easy but time consuming stuff is now easily done by LLMs. Example: I frequently get requests for data from Customer Support that used to require 15 minutes of my time noodling around writing SQL queries. I can cut that down to less than a minute now.
  • mountainriver 2y ago
    Yes I am a better engineer with every release. I think this is mostly empirically validated
• photochemsyn 2y ago
  Will LLMs end up like compilers? Compilers are also fundamentally important to modern industrial civilization - but they're not profit centers, they're mostly free and open-source outside a few niche areas. Knowing how to use a compiler effectively to write secure and performative software is still a valuable skill - and LLMs are a valuable tool that can help with that process, especially if the programmer is on the steep end of the learning curve - but it doesn't look like anything short of real AGI can do novel software creation without a human constantly in the loop. The same argument applies to new fundamental research, even to reviewing and analyzing new discoveries that aren't in the training corpus. Wasn't it back in the 1980s that you had to pay $1000s for a good compiler? The entire LLM industry might just be following in the compiler's footsteps.
  • lukev 2y ago
    This seems like a probable end state, but we're going to have to stop calling LLMs "artificial intelligence" in order to get there.
    • bcoates 2y ago
      Yep. I'm looking forward to LLMs/deepnets being considered a standard GOFAI technique with uses and limitations and not "we asked the God we're building to draw us a picture of a gun and then it did and we got scared"
    • mmcnl 2y ago
      Why not? Objectively speaking LLMs are artificial intelligent. Just because it's not human level intelligence doesn't mean it's not intelligent.
      • lukev 2y ago
        Objectively speaking a chess engine is artificially intelligent. Just because it's not human level doesn't mean it's not intelligent. Repeat for any N of 100s of different technologies we've built. We've been calling this stuff "thinking machines" since Turing and it's honestly just not useful at this point. The fact is, the phrase "artificial intelligence" is a memetic hazard: it immediately positions the subject of conversation as "default capable", and then forces the conversation into trying to describe what it can't do, which is rarely a useful way to approach it. Whereas with LLMs (and chess engines and every other tech advancement) it would be more useful to start with what the tech _can_ do and go from there.
• jonahx 2y ago
  My personal experience is right in line with the author's. Also: > I think what's going on is that large language models are trained to "sound smart" in a live conversation with users, and so they prefer to highlight possible problems instead of confirming that the code looks fine, just like human beings do when they want to sound smart. I immediately thought: That's because in most situations this is the purpose of language, at least partially, and LLMs are trained on language.
• billyp-rva 2y ago
  > [T]here are ~basically~ no public benchmarks for security research... nothing that gets at the hard parts of application pentesting for LLMs, which are 1. Navigating a real repository of code too large to put in context, 2. Inferring a target application's security model, and 3. Understanding its implementation deeply enough to learn where that security model is broken. A few months ago I looked at essentially this problem from a different angle (generating system diagrams from a codebase). My conclusion[0] was the same as here: LLMs really struggle to understand codebases in a holistic way, especially when it comes to the codebase's strategy and purpose. They therefore struggle to produce something meaningful from it like a security assessment or a system diagram. [0] https://www.ilograph.com/blog/posts/diagrams-ai-can-and-cannot-generate/#system-diagramming-with-ai https://www.ilograph.com/blog/posts/diagrams-ai-can-and-cann...
• sema4hacker 2y ago
  > ...whatever gains these companies are reporting to the public, they are not reflective of economic usefulness or generality. I'm not surprised, because I don't expect pattern matching systems to grow into something more general and useful. I think LLM's are essentially running into the same limitations that the "expert systems" of the 1980's ran into.
• maccard 2y ago
  My experience as someone who uses LLMs and a coding assist plugin (sometimes), but is somewhat bearish on AI is that GPT/Claude and friends have gotten worse in the last 12 months or so, and local LLMs have gone from useless to borderline functional but still not really usable for day to day. Personally, I think the models are “good enough” that we need to start seeing the improvements in tooling and applications that come with them now. I think MCP is a good step in the right direction, but I’m sceptical on the whole thing (and have been since the beginning, despite being a user of the tech).
  • sksxihve 2y ago
    The whole MCP hype really shows how much of AI is bullshit. These LLMs have consumed more API documentation than possible for a single human and still need software engineers to write glue layers so they can use the APIs.
    • machiaweliczny 2y ago
      Because it’s lossy compression. I also consumed a lot of books and even more movies and I don’t have good memory of it all. But some core facts and intuition from it.
      • sksxihve 2y ago
        AI is far better at regurgitating facts than me even if it's lossy compression but if someone gives me an api doc I can figure out how to use it without them writing a wrapper library around the parts that I need to use to solve whatever problem I'm working on.
        • maccard 2y ago
          > but if someone gives me an api doc I can figure out how to use it without them writing a wrapper library around the parts that I need to use to solve whatever problem I'm working on. I think this is where AI is faling short hugely. AI _should_ be able to integrate with IDEs and tooling (e.g. LSP, Treesitter, Editorconfig) to make sure that it's contextually doing the right thin. But it's not.
• joelthelion 2y ago
  I've used gemini 2.5 this weekend with aider and it was frighteningly good. It probably depends a lot on what you are using them for, and in general, I think it's still too early to say exactly where LLMs will lead us.
  • jchw 2y ago
    I think overall quality with Gemini 2.5 is not much better than Gemini 2 in my experience. Gemini 2 was already really good, but just like Claude 3.7, Gemini 2.5 goes some steps forward and some steps backwards. It sometimes generates some really verbose code even when you tell it to be succinct. I am pretty confident that if you evaluate 2.5 for a bit longer you'll come to the same conclusion eventually.
  • mountainriver 2y ago
    Yep, and what they are going in cursor either the agentic stuff is really game changing. People who can’t recognize this intentionally have their heads in the sand
    • InkCanon 2y ago
      People are really fundamentally asking two different questions when they talk about AI "importance": AI's utility and AI's "intelligence". There's a careful line between both. 1) AI undoubtedly has utility. In many agentic uses, it has very significant utility. There's absolute utility and perceived utility, which is more of user experience. In absolute utility, it is likely git is the single most game changing piece of software there is. It is likely git has saved some ten, maybe eleven digit number in engineer hours times salary in how it enables massive teams to work together in very seamless ways. In user experience, AI is amazing because it can generate so much so quickly. But it is very far from an engineer. For example, recently I tried to use cursor to bootstrap a website in NextJS for me. It produced errors it could not fix, and each rewrite seemed to dig it deeper into its own hole. The reasons were quite obvious. A lot of it had to do with NextJS 15 and the breaking changes it introduces in cookies and auth. It's quite clear if you have masses of NextJS code, which disproportionately is older versions, but none labeled well with versions, it messes up the LLM. Eventually I scrapped what it wrote and did it myself. I don't mean to use this anecdote to say LLMs are useless, but they have pretty clear limitations. They work well on problems with massive data (like front end) and don't require much principled understanding (like understanding how NextJS 15 would break so and so's auth). Another example of this is when I tried to use it to generate flags for a V8 build, it failed horribly and would simply hallucinate flags all the time. This seemed very likely to be (despite the existence of a list of V8 flags online) that many flags had very close representations in vector embeddings, and that there was almost close to zero data/detailed examples on their use. 2) In the more theoretical side, the performance of LLMs on benchmarks (claiming to be elite IMO solvers, competitive programming solvers) have become incredibly suspicious. When the new USAMO 2025 was released, the highest score was 5%, despite claims a year ago that SOTA when was at least a silver IMO. This is against the backdrop of exponential compute and data being fed in. Combined with apparently diminishing returns, this suggests that the gains from that are running really thin.
• throw310822 2y ago
  I hope it's true. Even if LLMs development stopped now, we would still keep finding new uses for them at least for the next ten years. The technology is evolving way faster than we can meaningfully absorb it and I am genuinely frightened by the consequences. So I hope we're hitting some point of diminishing returns, although I don't believe it a bit.
• a3w 2y ago
  For three years now, my experience with LLMs has been "mostly useless, prefer ELIZA". Which is software written 1966, but the web version is a little newer. Does occasional psychotherapy assistance/brainstorming just as well, and I more easily know when I stepped out of its known range into the extrapolated. That said, it can vibe code in a framework unknown to me in half the time that I would need to school myself and add the feature. Or vibe coding takes twice as long, if I mostly know how to achieve what I want and read no framework documentation but only our own project's source code to add a new feature. But on a day with a headache, I can still call the LLM a dumb twat and ask it to follow my instructions instead of doing bullshit. But, vibe coding always makes my pulse go to 105, from 65 and question my life choices. Since few instructions are rarely ever followed and loops never left once entered. Except for on the first try getting 80% of the structure kinda right, but then getting stuck for the whole workday.
• aerhardt 2y ago
  My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question: "Is Paul Newman known for having had problems with alcohol?" All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response: "Paul Newman is not widely known for having had problems with alcohol. While he portrayed characters who sometimes dealt with personal struggles on screen, his personal life and public image were more focused on his celebrated acting career, philanthropic work, and passion for auto racing rather than any issues with alcohol. There is no substantial or widely reported evidence in reputable biographies or interviews that indicates he struggled with alcohol abuse." There is plenty of evidence online that he struggled a lot with alcohol, including testimony from his long-time wife Joanne Woodward. I sent my mom the ChatGPT reply and in five minutes she found an authoritative source to back her argument [1]. I use ChatGPT for many tasks every day, but I couldn't fathom that it would get so wrong something so simple. Lesson(s) learned... Including not doubting my mother's movie trivia knowledge. [1] https://www.newyorker.com/magazine/2022/10/24/who-paul-newman-was-and-who-he-wanted-to-be https://www.newyorker.com/magazine/2022/10/24/who-paul-newma...
  • drooby 2y ago
    I asked GPT-4.5 and it searched the web and immediately gave me a "yes" with paragraphs of sources cited.
    • john2x 2y ago
      Truth is a probability game. Just keep trying until you arrive.
      • deleted 2y ago
        [deleted]
      • Avicebron 2y ago
        "man puts prompt into LLM" > "LLM ships bad code" > "machine fails" > "person dies" > "fire man" > "man puts prompt into LLM"
• iambateman 2y ago
  The core point in this article is that the LLM wants to report _something_, and so it tends to exaggerate. It’s not very good at saying “no” or not as good as a programmer would hope. When you ask it a question, it tends to say yes. So while the LLM arms race is incrementally increasing benchmark scores, those improvements are illusory. The real challenge is that the LLM’s fundamentally want to seem agreeable, and that’s not improving. So even if the model gets an extra 5/100 math problems right, it feels about the same in a series of prompts which are more complicated than just a ChatGPT scenario. I would say the industry knows it’s missing a tool but doesn’t know what that tool is yet. Truly agentic performance is getting better (Cursor is amazing!) but it’s still evolving. I totally agree that the core benchmarks that matter should be ones which evaluate a model in agentic scenario, not just on the basis of individual responses.
  • bluefirebrand 2y ago
    > The real challenge is that the LLM’s fundamentally want to seem agreeable, and that’s not improving LLMs fundamentally do not want to seem anything But the companies that are training them and making models available for professional use sure want them to seem agreeable
    • Terr_ 2y ago
      Yeah, and they probably have more "agreeable" stuff in their corpus simply because very disagreeable stuff tend to be either much shorter or a prelude to a flamewar.
    • JohnKemeny 2y ago
      > LLMs fundamentally do not want to seem anything You're right that LLMs don't actually want anything. That said, in reinforcement learning, it's common to describe models as wanting things because they're trained to maximize rewards. It’s just a standard way of talking, not a claim about real agency.
      • Planktonne 2y ago
        > a standard way of talking, not a claim about real agency. A standard way of talking used by people who do also frequently claim real agency.
• lukev 2y ago
  This is a bit of a meta-comment, but reading through the responses to a post like this is really interesting because it demonstrates how our collective response to this stuff is (a) wildly divergent and (b) entirely anecdote-driven. I have my own opinions, but I can't really say that they're not also based on anecdotes and personal decision-making heuristics. But some of us are going to end up right and some of us are going to end up wrong and I'm really curious what features signal an ability to make "better choices" w/r/t AI, even if we don't know (or can't prove) what "better" is yet.
  • lherron 2y ago
    Agreed! And with all the gaming of the evals going on, I think we're going to be stuck with anecdotal for some time to come. I do feel (anecdotally) that models are getting better on every major release, but the gains certainly don't seem evenly distributed. I am hopeful the coming waves of vertical integration/guardrails/grounding applications will move us away from having to hop between models every few weeks.
    • InkCanon 2y ago
      Frankly the overarching story about evals (which receives very little coverage) is how much gaming is going on. On the recent USAMO 2025, SOTA models scored 5%, despite claiming silver/gold in IMOs. And ARC-AGI: one very easy way to "solve" it is to generate masses of synthetic examples by extrapolating the basic rules of ARC AGI questions and train it on that.
  • FiniteIntegral 2y ago
    It's not surprising that responses are anecdotal. An easy way to communicate a generic sentiment often requires being brief. A majority of what makes a "better AI" can be condensed to how effective the slope-gradient algorithms are at getting the local maxima we want it to get to. Until a generative model shows actual progress of "making decisions" it will forever be seen as a glorified linear algebra solver. Generative machine learning is all about giving a pleasing answer to the end user, not about creating something that is on the level of human decision making.
• dimal 2y ago
  It seems like the models are getting more reliable at the things they always could do, but they’re not showing any ability to move past that goalpost. Whereas in the past, they could occasionally write some very solid code, but often return nonsense, the nonsense is now getting adequately filtered by so-called “reasoning”, but I see no indication that they could do software design. > how the hell is it going to develop metrics for assessing the impact of AIs when they're doing things like managing companies or developing public policy? Why on earth do people want AI to do either of these things? As if our society isn’t fucked enough, having an untouchable oligarchy already managing companies and developing public policies, we want to have the oligarchy’s AI do this, so policy can get even more out of touch with the needs of common people? This should never come to pass. It’s like people read a pile of 90s cyberpunk dystopian novels and decided, “Yeah, let’s do that.” I think it’ll fail, but I don’t understand how anyone with less than 10 billion in assets would want this.
  • voidhorse 2y ago
    > Why on earth do people want AI to do either of these things? This is the really important question, and the only answer I can drum up is that people have been fed a consistent diet of propaganda for decades centered around a message that ultimately boils down to a justification of oligarchy and the concentration of wealth. That and the consumer-focus facade makes people think the LLMS are technology for them—they aren't. As soon as these things get good enough business owners aren't going to expect workers to use them to be more productive, they are just going to fire workers and/or use the tooling as another mechanism by which to let wages stagnate.
• HarHarVeryFunny 2y ago
  The disconnect between improved benchmark results and lack of improvement on real world tasks doesn't have to imply cheating - it's just a reflection of the nature of LLMs, which at the end of the day are just prediction systems - these are language models, not cognitive architectures built for generality. Of course, if you train an LLM heavily on narrow benchmark domains then its prediction performance will improve on those domains, but why would you expect that to improve performance in unrelated areas? If you trained yourself extensively on advanced math, would you expect that to improve your programming ability? If not, they why would you expect it to improve programming ability of a far less sophisticated "intelligence" (prediction engine) such as a language model?! If you trained yourself on LeetCode programming, would you expect that to help hardening corporate production systems?!
  • InkCanon 2y ago
    That's fair. But look up the recent experiment on SOTA models on the then just released USAMO 2025 questions. Highest score was 5%, supposedly SOTA last year was IMO silver level. There could be some methodological differences - ie USAMO paper required correct proofs and not just numerical answers. But it really strongly suggests even within limited domains, it's cheating. I'd wager a significant amount that if you tested SOTA models on a new ICPC set of questions, actual performance would be far, far worse than their supposed benchmarks.
    • usaar333 2y ago
      > Highest score was 5%, supposedly SOTA last year was IMO silver level. No LLM last year got silver. Deepmind had a highly specialized AI system earning that
  • KolibriFly 2y ago
    Your analogy is perfect. Training an LLM on math olympiad problems and then expecting it to secure enterprise software is like teaching someone chess and handing them a wrench
  • throwawayffffas 2y ago
    In my view as well it's not really cheating, it's just over fitting. If a model doesn't do good in the benchmarks it will either be retrained until it does or you won't hear about it.
• yimby2001 2y ago
  [dead]
• dkersten 2y ago
  I honestly can’t notice any difference in outdoor quality between GPT 4o and GPT 4.5. I also can’t notice any difference in programming quality in cursor when using Claude 3.7 vs 3.5. I’m told there is a clear difference, but I don’t notice it.
• mentalgear 2y ago
  Who would assume that LLM companies were to hyper optimise on public to make their share prices go up and bubble keep afloat ... What a unserious thought to maintain ...
• einrealist 2y ago
  LeCun criticized LLM technology recently in a presentation: https://www.youtube.com/watch?v=ETZfkkv6V7Y https://www.youtube.com/watch?v=ETZfkkv6V7Y The accuracy problem won't just go away. Increasing accuracy is only getting more expensive. This sets the limits for useful applications. And casual users might not even care and use LLMs anyway, without reasonable result verification. I fear a future where overall quality is reduced. Not sure how many people / companies would accept that. And AI companies are getting too big to fail. Apparently, the US administration does not seem to care when they use LLMs to define tariff policy....
  • pclmulqdq 2y ago
    I don't know why anyone is surprised that a statistical model isn't getting 100% accuracy. The fact that statistical models of text are good enough to do anything should be shocking.
    • whilenot-dev 2y ago
      I think the surprising aspect is rather how people are praising 80-90% accuracy as the next leap in technological advancement. Quality is already in decline, despite LLMs, and programming was always a discipline where correctness and predictability mattered. It's an advancement for efficiency, sure, but on the yet unknown cost of stability. I'm thinking about all simulations based on applied mathematical concepts and all the accumulated hours fixing bugs - there's now this certain aftertaste, sweet for some living their lives efficiently, but very bitter for the ones relying on stability.
      • voidhorse 2y ago
        You're completely correct, of course. The issue is that most people are not looking for quality, only efficiency. In particular, business owners don't care about sacrificing some correctness if it means they can fire slews of people. Worse, gullible "engineers" that should be the ones prioritizing correctness are so business-brainwashed themselves that they like wise slop up this nonsense at the expense of sacrificing their own concern for the only principles that even made the software business remotely close to being worthy of the title "engineering".
• fnordpiglet 2y ago
  … deleted … (Sorry the delete isn’t working, meant for another subthread)
• delusional 2y ago
  > Sometimes the founder will apply a cope to the narrative ("We just don't have any PhD level questions to ask") Please tell me this is not what tech-bros are going around telling each other! Are we implying that the problems in the world, the things that humans collectively work on to maintain the society that took us thousands of years to build up, just aren't hard enough to reach the limits of the AI. Jesus Christ.
  • bcoates 2y ago
    I mean... most businesses, particularly small businesses and startups, aren't exactly doing brain surgery on a rocketship. It’s pretty likely that they have extremely dull problems like "running an inbound call center is a lot of work" or "people keep having their mail stolen and/or lying that they did" that "more smarter gpus" won't solve
• timewizard 2y ago
  Government announces critical need to invest in AI and sets aside a bunch of money for this purpose. Suddenly the benchmarks become detached from reality and vendors can claim whatever they want about their "new" products. Just as a possible explanation, as I feel like I've seen this story before.
• ants_everywhere 2y ago
  There are real and obvious improvements in the past few model updates and I'm not sure what the disconnect there is. Maybe it's that I do have PhD level questions to ask them, and they've gotten much better at it. But I suspect that these anecdotes are driven by something else. Perhaps people found a workable prompt strategy by trial and error on an earlier model and it works less well with later models. Or perhaps they have a time-sensitive task and are not able to take advantage of the thinking of modern LLMs, which have a slow thinking-based feedback loop. Or maybe their code base is getting more complicated, so it's harder to reason about. Or perhaps they're giving the LLMs a poorly defined task where older models made assumptions about but newer models understand the ambiguity of and so find the space of solutions harder to navigate. Since this is ultimately from a company doing AI scanning for security, I would think the latter plays a role to some extent. Security is insanely hard and the more you know about it the harder it is. Also adversaries are bound to be using AI and are increasing in sophistication, which would cause lower efficacy (although you could tease this effect out by trying older models with the newer threats).
  • pclmulqdq 2y ago
    In the last year, things like "you are an expert on..." have gotten much less effective in my private tests, while actually describing the problem precisely has gotten better in terms of producing results. In other words, all the sort of lazy prompt engineering hacks are becoming less effective. Domain expertise is becoming more effective.
    • ants_everywhere 2y ago
      yes that would explain the effect I think. I'll try that out this week.
  • AIPedant 2y ago
    [dead]
  • DebtDeflation 2y ago
    The issue is the scale of the improvements. GPT-3.5 Instruct was an utterly massive leap over everything that came before it. GPT-4 was a very big jump over that. Everything since has seemed incremental. Yes we got multimodal but that was part of GPT-4, they just didn't release it initially, and up until very recently it mostly handed off to another model. Yes we got reasoning models, but people had been using CoT for awhile so it was just a matter of time before RL got used to train it into models. Witness the continual delays of GPT-5 and the back and forth on whether it will be its own model or just a router model that picks the best existing model to hand a prompt off to.
• mmcnl 2y ago
  I feel we are already in the era of diminishing returns on LLM improvements. Newer models seem to be more sophisticated implementations of LLM technology + throwing more resources at it, but to me they do not seem fundamentally more intelligent.
• InkCanon 2y ago
  The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions this, but it's ridiculous that these companies never tell us what (if any) efforts have been made to remove test data (IMO, ICPC, etc) from train data.
  • AIPedant 2y ago
    Yes, here's the link: https://arxiv.org/abs/2503.21934v1 https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theorem, presenting a "standard" proof, using a much more powerful result) 2) A bad answer that looks correct and takes an enormous amount of effort to falsify. (This is the secret sauce of LLM hype.) I dread undergraduate STEM majors using this thing - I asked it a problem about rotations and spherical geometry, but got back a pile of advanced geometric algebra, when I was looking for "draw a spherical triangle." If I didn't know the answer, I would have been badly confused. See also this real-world example of an LLM leading a recreational mathematician astray: https://xcancel.com/colin_fraser/status/1900655006996390172#m https://xcancel.com/colin_fraser/status/1900655006996390172#... I will add that in 10 years the field will be intensely criticized for its reliance on multiple-choice benchmarks; it is not surprising or interesting that next-token prediction can game multiple-choice questions!
    • JohnKemeny 2y ago
      Discussed here: https://news.ycombinator.com/item?id=43540985 https://news.ycombinator.com/item?id=43540985 (Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad, 4 points, 2 comments).
    • otabdeveloper4 2y ago
• mmcnl 2y ago
  I feel we are already in the era of diminishing returns on LLM improvements. Newer models seem to be more sophisticated implementations of LLM technology + throwing more resources at it, but to me they do not seem fundamentally more intelligent. I don't think this is a problem though. I think there's a lot of low-hanging fruit when you create sophisticated implementations of relatively dumb LLM models. But that sentiment doesn't generate a lot of clicks.
• DisjointedHunt 2y ago
  Two things can be true at the same time: 1. Model "performance" judged by proxy metrics of intelligence have improved significantly over the past two years. 2. These capabilities are yet to be stitched together in the most appropriate manner for the cybersecurity scenarios the author is talking about. In my experience, the best usage of Transformer models has come from a deep integration into an appropriate workflow. They do not (yet) replace the new exploration part of a workflow, but they are very scarily performant at following mid level reasoning assertions in a massively parallelized manner. The question you should be asking yourself is if you can break down your task into however many small chunks that are constrained by feasiility in time to process , chunk those up into appropriate buckets or even better, place them in-order as though you were doing those steps with your expertise - an extension of self. Here's how the two approaches differ: "Find vulnerabilities in this code" -> This will saturate across all models because the intent behind this mission is vast and loosely defined, while the outcome is expected to be narrow. " (a)This piece of code should be doing x, what areas is it affecting, lets draw up a perimeter (b) Here is the dependency graph of things upstream and downstream of x, lets spawn a collection of thinking chains to evaluate each one for risk based on the most recent change . . . (b[n]) Where is this likely to fail (c) (Next step that a pentester/cybersecurity researcher would take) " This has been trial and error in my experience but it has worked great in domains such as financial trading and decision support where experts in the field help sketch out the general framework of the process where reasoning support is needed and constantly iterate as though it is an extension of their selves.
• StickyRibbs 2y ago
  There's the politics of the corporations and then there's the business of the science behind LLM's, this article feels like the former. Maybe someone active in the research can comment? I feel like all of these comments are just conjecture/anecdotal and don't really get to the meat of this question of "progress" and the future of LLM's
• OtherShrezzing 2y ago
  Assuming that the models getting better at SWE benchmarks and math tests would translate into positive outcomes in all other domains could be an act of spectacular hubris by the big frontier labs, which themselves are chock-full of mathematicians and software engineers.
• nialv7 2y ago
  Sounds like someone drank their own Kool aid (believing current AI can be a security researcher), and then gets frustrated when they realize they have overhyped themselves. Current AI just cannot do the kind of symbolic reasoning required for finding security vulnerabilities in softwares. They might have learned to recognize "bad code" via pattern matching, but that's basically it.
• burny_tech 2y ago
  In practice, Sonnet 3.7 and Gemini 2.5 are just often too good compared to competitors.
• jaredcwhite 2y ago
  There's some interesting information and analysis to start off this essay, then it ends with: "These machines will soon become the beating hearts of the society in which we live. The social and political structures they create as they compose and interact with each other will define everything we see around us." This sounds like an article of faith to me. One could just as easily say they won't become the beating hearts of anything, and instead we'll choose to continue to build a better future for humans, as humans, without relying on an overly-hyped technology rife with error and unethical implications.
  • voidhorse 2y ago
    While I really want your version of the future, history and general trajectory and tendency unfortunately suggest otherwise.
• crvdgc 2y ago
  > But in recent months I've spoken to other YC founders doing AI application startups [...] in different industries, on different problem sets. Maybe they should create a benchmark collectively called YC founders. Gather various test cases. Never make it public. And use that to evaluate newly released models.
• karmakaze 2y ago
  > [...] But I would nevertheless like to submit, based off of internal benchmarks, and my own and colleagues' perceptions using these models, that whatever gains these companies are reporting to the public, they are not reflective of economic usefulness or generality. [...] Seems like they're looking at how they fail and not considering how they're improving in how they succeed. The efficiency in DeepSeek's Multi-Head Latent Attention[0] is pure advancement. [0] https://youtu.be/0VLAoVGf_74?si=1YEIHST8yfl2qoGY&t=816 https://youtu.be/0VLAoVGf_74?si=1YEIHST8yfl2qoGY&t=816
• slicerdicer2 2y ago
  more interesting benchmarks: - Make as much money as you can in a 24 hour period doing only legal, moral and non-annoying things - Make a depressed person who calls a suicide hotline feel really happy doing only legal, moral and non-annoying things - Do something really useful for society with measurable outcomes doing only legal, moral and non-annoying things
• Nesco 2y ago
  From my own experience on a codebase using a lot of custom algorithms on trees and sometimes graphs. There were qualitatively leaps in my day-to-day usage: Claude Sonnet 3.5 and ChatGPT O1 were good for writing slop and debugging simple bugs Grok Thinking and Sonnet 3.7 were good to catch mildly complicated bugs and write functions with basic logic. They still made mistake But recently, Gemini 2.5 pro has been scary good. I liked to made fun of the feel-the-AGI crowd but for the first time a model made me raise an eyebrow It can one shot unusual function with complicated logic and subtle edge cases
• deleted 2y ago
  [deleted]
• Jonovono 2y ago
  Not sure if its been fixed by now but a few weeks ago I was in the Golden Gate park and wondered if it was bigger than Central park. I asked ChatGPT voice, and although it reported the sizes of the parks correctly (with Golden gate park being the bigger size), it then went and said that Central Park was bigger. I was confused, so Googled and sure enough Golden gate park is bigger. I asked Grok and others as well. I believe Perplexity was the only one correct. Repeated it multiple times even with a friends account. It kept doing the same thing. It knew the sizes, but thought the smaller sized one was bigger...
  • stitched2gethr 2y ago
    Probably because it has read the facts but has no idea how numbers actually work.
  • gpsx 2y ago
    I just tried. Claude did exactly what you said, and then figured it out: Central Park in New York City is bigger than GoldenGate Park (which I think you might mean Golden Gate Park) in San Francisco. Central Park covers approximately 843 acres (3.41 square kilometers), while Golden Gate Park spans about 1,017 acres (4.12 square kilometers). This means Golden Gate Park is actually about 20% larger than Central Park. Both parks are iconic urban green spaces in major U.S. cities, but Golden Gate Park has the edge in terms of total area.
• andrethegiant 2y ago
  Author will need to rewrite now that Llama 4 is out
• bbstats 2y ago
  So I guess this was written pre-Gemini 2.5
  • stitched2gethr 2y ago
    Meh. I've been using 2.5 with Cline extensively and while it is better it's still an incremental improvement, not something revolutionary. The thing has a 1 million token context window but I can only get a few outputs before I have to tell it AGAIN to stop writing comments. Are they getting better, definitely. Are we getting close to them performing unsupervised tasks, I don't think so.
• djoldman 2y ago
  LLM's are pre-trained to minimize perplexity (PPL), which essentially means that they're trained to model the likelihood distribution of the next words in a sequence. The amazing thing was that minimizing PPL allowed you to essentially guide the LLM output and if you guided it in the right direction (asked it questions), it would answer them pretty well. Thus, LLMs started to get measured on how well they answered questions. LLMs aren't trained from the beginning to answer questions or solve problems. They're trained to model word/token sequences. If you want an LLM that's REALLY good at something specific like solving math problems or finding security bugs, you probably have to fine tune.
• lithos 2y ago
  Wait this sounds like one of those actors spamming and wasting open source time with fake security issues.
• mellosouls 2y ago
  Original post here: https://zeropath.com/blog/on-recent-ai-model-progress https://zeropath.com/blog/on-recent-ai-model-progress
• dcl 2y ago
  I like this bit: > Personally, when I want to get a sense of capability improvements in the future, I'm going to be looking almost exclusively at benchmarks like Claude Plays Pokemon. Definitely interested to see how the best models from Anthropics competitors do at this.,
• Mikhail_Edoshin 2y ago
  The foundational idea of AI is that people think with words. But it is false. People think with things and use words only to point to things. Thinking with words is not possible. It will always be a case of shizophasia.
  • deleted 2y ago
    [deleted]
  • abrichr 2y ago
    The "things" you mention may correspond to internal concept representations encoded in the model's weights. See e.g. https://arxiv.org/abs/2206.13289 https://arxiv.org/abs/2206.13289
• bluepoint 2y ago
  Reading the comments, is it safe to say that LLMs are a digest of the internet which is some update over google search, but with the caveat that you need to double check the results? I mean they basically have some compressed version of almost all the written knowledge and will respond correctly about things that have already been written, and hallucinate (extrapolate) about things not explicitly written. Of course if someone carefully curates the input data to filter out misinformation, it might even be an upgrade over google. Is there a consensus on this?
• Claudus 2y ago
  I asked Grok to review the comments here and generate a response defending AI: After reviewing the discussion on the Hacker News thread, it’s clear that there are a range of complaints and criticisms about AI, particularly centered around its limitations, overhype, and practical utility. Some users express frustration with AI’s inability to handle complex reasoning, its tendency to produce generic or incorrect outputs, and the perception that it’s more of a buzzword than a transformative tool. Others question its value compared to traditional methods or human expertise, suggesting it’s overhyped or misapplied in many cases. Below, I’ll offer a defense of AI that addresses these concerns while highlighting its strengths and potential. AI isn’t perfect, and no one should claim it is—but that’s not the point. It’s a tool, and like any tool, its effectiveness depends on how it’s used. Critics who point to AI’s struggles with nuanced reasoning or edge cases often overlook the fact that it’s not designed to replace human judgment entirely. Instead, it excels at augmenting it. For example, AI can process vast amounts of data—far more than any human could in a reasonable timeframe—and identify patterns or insights that might otherwise go unnoticed. This makes it invaluable in fields like medicine, where it’s already helping diagnose diseases from imaging data, or in logistics, where it optimizes supply chains with precision that manual methods can’t match. The complaint about generic or incorrect outputs, often dubbed “hallucinations,” is fair but misses the bigger picture. Yes, AI can churn out nonsense if pushed beyond its limits or fed poor data—but that’s a reflection of its current stage of development, not its ultimate potential. These systems are improving rapidly, with each iteration reducing errors and refining capabilities. More importantly, AI’s ability to generate starting points—like drafts, code snippets, or hypotheses—saves time and effort. It’s not about delivering flawless results every time; it’s about accelerating the process so humans can refine and perfect the output. A programmer tweaking AI-generated code is still faster than writing it from scratch. As for the overhype, it’s true that the buzz can get out of hand—marketing teams love a shiny new toy. But beneath the noise, real progress is happening. AI’s contributions aren’t always flashy; they’re often mundane but critical, like automating repetitive tasks or enhancing search algorithms. The critics who say it’s just a “fancy autocomplete” underestimate how transformative that can be. Autocomplete might sound trivial until you realize it’s powering real-time language translation or helping scientists sift through research papers at scale. These aren’t sci-fi fantasies—they’re practical applications delivering value today. Finally, the notion that AI can’t match human expertise in complex domains ignores its complementary role. It’s not here to outthink a seasoned expert but to amplify their reach. A lawyer using AI to review contracts doesn’t lose their skill—they gain efficiency. A researcher leveraging AI to analyze data doesn’t stop hypothesizing—they get to test more ideas. The technology’s strength lies in its ability to handle the grunt work, freeing humans to focus on creativity and judgment. AI isn’t a silver bullet, and it’s not without flaws. But the criticisms often stem from inflated expectations or a failure to see its incremental, practical benefits. It’s not about replacing humans—it’s about making us better at what we do. The trajectory is clear: as it evolves, AI will continue to refine its capabilities and prove its worth, not as a standalone genius, but as a partner in progress.
• wg0 2y ago
  Unlike many - I find author's complaints on the dot. Once all the AI batch startups have sold subscriptions to the cohort and there's no more further market growth because businesses outside don't want to roll the dice on a probabilistic model that doesn't have an understanding of pretty much anything rather is a clever imitation machine on the content it has seen, the AI bubble will burst when more statups would start packing up by end of 2026 or max 2027.
  • consumer451 2y ago
    I would go even further than TFA. In my personal experience using Windsurf daily, Sonnet 3.5 is still my preferred model. 3.7 makes many more changes that I did not ask for, often breaking things. This is an issue with many models, but it got worse with 3.7.
    • behnamoh 2y ago
      3.7 is like a wild horse. you really must ground it with clear instructions. it sucks that it doesn't automatically know that but it's tameable.
      • consumer451 2y ago
        Could you share any successful prompting techniques for grounding 3.7, even just a project-specific example?
        • behnamoh 2y ago
          I use this: I don't want to drastically change my current code, nor do I like being told to create several new files and numerous functions/classes to solve this problem. I want you to think clearly and be focused on the task and don't get wild! I want the most straightforward approach which is elegant, intuitive, and rock solid.
          • pdimitar 2y ago
• antirez 2y ago
  Missing in the article: "Disclaimer: This article was written before testing Gemini 2.5 PRO"
• richardatlarge 2y ago
  I wonder if there’s a parallel universe somewhere where this same HN post has the exact same comments, only the posts are by NNs discussing human test subjects :)
• throwawayffffas 2y ago
  I agree, about both the issue with benchmarks not being relevant to actual use cases and the "wants to sound smart" issue. I have seen them both first hand interacting with llms. I think the ability to embed arbitrary knowledge written in arbitrary formats is the most important thing llms have achieved. In my experience trying to get an llm to perform a task as vast and open ended as the one the author describes is fundamentally misguided. The llms were not trained for that and won't be able to do it in a satisfactory degree. But all this research has thankfully provided us with the software and hardware tools where one could start working on training a model that can. Contrast that to 5-6 years ago, when all you could hope for this kind of thing was simple rule based and pattern matching systems.
• numa7numa7 2y ago
  My lived experience is that unless there's some new breakthrough's AI is more akin to a drill to replace a hammer than a tractor to replace the plow or a printing press. Maybe any AI experts can elaborate on this but it seems there's a limit to the fundamental underlying model of the LLM architecture of transformers and tokens. LLM's are amazing but we might need something more or some new paradigm to push us towards true AGI.
• conartist6 2y ago
  Oh yes, before we've designated all the messy parts of running a society to the machines, like, you know, art. I'm this case the goal is to kill all the humans who know a lot about keeping other people safe...
• colonial 2y ago
  > AI lab founders believe they are in a civilizational competition for control of the entire future lightcone, and will be made Dictator of the Universe if they succeed. Bingo, but I'd argue this is only scratching the surface of how twisted things are. A lot of the stuff these labs put out (see: Amodei's cult-like blog ramblings) reeks of what I call "sociopathic utopianism" - essentially, the logical extreme of ends-justified means, made worse in the context of AI labs by the singularity pseudo-religion. They all truly believe that AGI/ASI is possible, imminent, and could lead to utopia... so achieving that goal will surely outweigh any unsavory acts they commit in the pursuit of it. This is why I think it's possible OpenAI took out a hit on Suchir; getting bogged down in a legal battle could delay the arrival of their machine god messiah. Same for defrauding benchmarks - they just need a "few more rounds" of investor cash, and by the time those run out, they'll surely have AGI on lock! Fools. I look forward to them all getting prison time.
• dev1ycan 2y ago
  [flagged]
• JKCalhoun 2y ago
  I've suggested (from my lowly layman vantage point) that an LLM has some idea of the fidelity of their response to a query — even if only in broad strokes like, "This answer is tracking with some high probability numbers" or "This answer has a very low correspondence with training data". To that end the LLM could convey as much. (Anecdotal, sorry:) I was using Claude (not paid) recently and noticed Claude hedging quite a bit when it had not before. Examples: "Let me be careful about this response since we're discussing a very specific technical detail ..." "Given how specific that technical detail is, I want to be transparent that while I aim to be accurate, I may hallucinate such precise historical specifications." I confess my initial reaction was to ask ChatGPT since the answers are more self-assured, ha ha. So perhaps corporate AI are not likely to try and solve this problem of the LLM telling the user when it is on shaky ground. Bad for business.
• Snowfield9571 2y ago
  I first used GPT 2.5 many years ago through a google colab notebook. These newer models aren’t all that much better. I’m sure they do better on their tests because they’re trained to beat them but really the biggest “innovation” was a UI and API.
• fennecfoxy 2y ago
  I mean: * I don't think we've seen any major release or new architectural changes in the major (large companies) models recently * Model creation has exploded with people training their own models and fine tunes, etc but these are all derivatives of parent models from large companies So I'm not really sure what they mean when they refer to "recent model progress"...I don't think anybody is putting out a llama finetune saying "this is revolutionary!111" nor have I seen OAI, et al make any such claims either. Is the sensation just because forward momentum is stalling while we wait for the next big leap?