6 ms·

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will

• codingwagie 1y ago
  I'm seeing big advances that arent shown in the benchmarks, I can simply build software now that I couldnt build before. The level of complexity that I can manage and deliver is higher.
  • shmoogy 1y ago
    Yeah I kind of feel like I'm not moving as fast as I did, because the complexity and features grow - constant scope creep due to moving faster.
  • alightsoul 1y ago
    mind telling examples?
    • motorest 1y ago
      Not OP, but a couple of days ago I managed to vibecode my way through a small app that pulled data from a few services and did a few validation checks. By itself its not very impressive, but my input was literally "this is how the responses from endpoint A,B and C look like. This field included somewhere in A must be somewhere in the response from B, and the response from C must feature this and that from response A and B. If the responses include links, check that they exist". To my surprise, it generated everything in one go. No retry nor Agent mode churn needed. In the not so distant past this would require progressing through smaller steps, and I had to fill in tests to nudge Agent mode to not mess up. Not today.
      • alightsoul 1y ago
        what tools did you use?
        • motorest 1y ago
          > what tools did you use? Nothing fancy. Visual Studio Code + Copilot, agent mode, a couple prompt files, and that's it.
          • munksbeer 1y ago
            Do you mind me asking which language and if you have any esoteric constraints in the apps you build? We use a java in a monorepo, and have a full custom rolled framework on top of which we build our apps. Do you find vibe coding works ok with those sort of constraints, or do you just end up with a generic app?
      • corysama 1y ago
        I’m wrapping up doing literally the same thing. I did it step-by-step. But, for me there was also a process of figuring out how it should work.
  • IanCal 1y ago
    A really important thing is the distinction between performance and utility. Performance can improve linearly and utility can be massively jumpy. For some people/tasks performance can have improved but it'll have been "interesting but pointless" until it hits some threshold and then suddenly you can do things with it.
  • protocolture 1y ago
    I am finding that my ability to use it to code, aligns almost perfectly with increasing token memory.
  • kevinqi 1y ago
    yeah, the benchmarks are just a proxy. o3 was a step change where I started to really be able to build stuff I couldn't before
  • iLoveOncall 1y ago
    Okay but this has all to do with the tooling and nothing to do with the models.
    • mofeien 1y ago
      Can you explain why?
      • iLoveOncall 1y ago
        You can write projects with LLMs thanks to tools that can analyze your local project's context, which didn't exist a year ago. You could use Cursor, Windsurf, Q CLI, Claude Code, whatever else with Claude 3 or even an older model and you'd still get usable results. It's not the models which have enabled "vibe coding", it's the tools. An additional proof of that is that the new models focus more and more on coding in their releases, and other fields have not benefited at all from the supposed model improvements. That wouldn't be the case if improvements were really due to the models and not the tooling.
        • eru 1y ago
          You need a certain quality of model to make 'vibe coding' work. For example, I think even with the best tooling in the world, you'd be hard pressed to make GPT 2 useful for vibe coding.
          • iLoveOncall 1y ago
            I'm not claiming otherwise. I'm just saying that people say "look what we can do with the new models" when they're completely ignoring the fact that the tooling has improved a hundred fold (or rather, there was no tooling at all and now there is).
            • eru 1y ago
              OK, no objections from me there.
• energy123 1y ago
  That would require AIME 2024 going above 100%. There was always going to be diminishing returns in these benchmarks. It's by construction. It's mathematically impossible for that not to happen. But it doesn't mean the models are getting better at a slower pace. Benchmark space is just a proxy for what we care about, but don't confuse it for the actual destination. If you want, you can choose to look at a different set of benchmarks like ARC-AGI-2 or Epoch and observe greater than linear improvements, and forget that these easier benchmarks exist.
  • croddin 1y ago
    There is still plenty of room for growth on the ARC-AGI benchmarks. ARC-AGI 2 is still <5% for o3-pro and ARC-AGI 1 is only at 59% for o3-pro-high: "ARC-AGI-1: * Low: 44%, $1.64/task * Medium: 57%, $3.18/task * High: 59%, $4.16/task ARC-AGI-2: * All reasoning efforts: <5%, $4-7/task Takeaways: * o3-pro in line with o3 performance * o3's new price sets the ARC-AGI-1 Frontier" - https://x.com/arcprize/status/1932535378080395332 https://x.com/arcprize/status/1932535378080395332
    • saberience 1y ago
      I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.
      • HDThoreaun 1y ago
        arc agi is the closest any widely used benchmark is coming to an IQ test, its straight logic/reasoning. Looking at the problem set its hard for me to choose a better benchmark for "when this is better than humans we have agi"
        • saberience 1y ago
• jstummbillig 1y ago
  It's hard to be 100% certain, but I am 90% certain that the benchmarks leveling off, at this point, should tell us that we are really quite dumb and simply not good very good at either using or evaluating the technology (yet?).
  • alightsoul 1y ago
    either that or the improvements aren't as large as before.
  • motorest 1y ago
    > (...) at this point, should tell us that we are really quite dumb and simply not good very good at either using or evaluating the technology (yet?). I don't know about that. I think it's mainly because nowadays LLMs can output very inconsistent results. In some applications they can generate surprisingly good code, but during the same session they can also do missteps and shit the bed while following a prompt to small changes. For example, sometimes I still get prompt responses that outright delete critical code. I'm talking about things like asking "extract this section of your helper method into a new methid" and in response the LLM deletes the app's main function. This doesn't happen all the time, or even in the same session for the same command. How does one verify these things?
• XCSme 1y ago
  I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy. Even though it's a large10% increase first then only a 0.999% increase.
  • jsjohnst 1y ago
    The saying goes: From 90% to 99% is a 10x reduction in error rate, but 99% to 99.999% is a 1000x decrease in error rates.
  • zmgsabst 1y ago
    Sometimes it’s nice to frame it the other way, eg: 90% -> 1 error per 10 99% -> 1 error per 100 99.99% -> 1 error per 10,000 That can help to see the growth in accuracy, when the numbers start getting small (and why clocks are framed as 1 second lost per…).
    • XCSme 1y ago
      Still, for the human mind it doesn't make intuitive sense. I guess it's the same problem with the mind not intuitively grasping the concept of exponential growth and how fast it grows.
      • XCSme 1y ago
        ChatGPT quick explanation: Humans struggle with understanding exponential growth due to a cognitive bias known as *Exponential Growth Bias (EGB)*—the tendency to underestimate how quickly quantities grow over time. Studies like Wagenaar & Timmers (1979) and Stango & Zinman (2009) show that even educated individuals often misjudge scenarios involving doubling, such as compound interest or viral spread. This is because our brains are wired to think linearly, not exponentially, a mismatch rooted in evolutionary pressures where linear approximations were sufficient for survival. Further research by Tversky & Kahneman (1974) explains that people rely on mental shortcuts (heuristics) when dealing with complex concepts. These heuristics simplify thinking but often lead to systematic errors, especially with probabilistic or nonlinear processes. As a result, exponential trends—such as pandemics, technological growth, or financial compounding—often catch people by surprise, even when the math is straightforward.
• littlestymaar 1y ago
  > I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. This kind of expectations explains why there hasn't been a GPT-5 so far, and why we get a dumb numbering scheme instead for no reason. At least Claude eventually decided not to care anymore and release Claude 4 even if the jump from 3.7 isn't particularly spectacular. We're well into the diminishing returns at this point, so it doesn't really make sense to postpone the major version bump, it's not like they're going to make a big leap again anytime soon.
  • sailingparrot 1y ago
    > We're well into the diminishing returns at this point Scaling laws, by definition have always had diminishing returns because it's a power law relationship with compute/params/data, but I am assuming you mean diminishing beyond what the scaling laws predict. Unless you know the scale of e.g. o3-pro vs GPT-4, you can't definitively say that. Because of that power law relationship, it requires adding a lot of compute/params/data to see a big jump, rule of thumb is you have to 10x your model size to see a jump in capabilities. I think OpenAI has stuck with the trend of using major numbers to denote when they more than 10x the training scale of the previous model. * GPT-1 was 117M parameters. * GPT-2 was 1.5B params (~10x). * GPT-3 was 175B params (~100x GPT-2 and exactly 10x Turing-NLG, the biggest previous model). After that it becomes more blurry as we switched to MoEs (and stopped publishing), scaling laws for parameters applies to a monolithic models, not really to MoEs. But looking at compute we know GPT-3 was trained on ~10k V100, while GPT-4 was trained on a ~25k A100 cluster, I don't know about training time, but we are looking at close to 10x compute. So to train a GPT-5-like model, we would expect ~250k A100, or ~150k B200 chips, assuming same training time. No one has a cluster of that size yet, but all the big players are currently building it. So OpenAI might just be reserving GPT-5 name for this 10x-GPT-4 model.
    • littlestymaar 1y ago
      > but I am assuming you mean diminishing beyond what the scaling laws predict. You're assuming wrong, in fact focusing on scaling law underestimate the rate of progress as there is also a steady stream algorithmic improvements. But still, even though hardware and software progress, we are facing diminishing returns and that means that there's no reason to believe that we will see another leap as big as GPT-3.5 to GPT-4 in a single release. At least until we stumble upon radically new algorithms that reset the game. I don't think it make any economic sense to wait until you have your “10x model” when you can release 2 or 3 incremental models in the meantime, at which point your “x10” becomes an incremental improvement in itself.
• avereveard 1y ago
  There's a new set of metrics that capture advances better than MMLU or it's pro version but nothing yet as standardized and specifically very few have a hidden set of tests to keep advancements from been from directional fine tuning.