9 ms·
What's going on with their SWE bench graph?[0] GPT-5 non-thinking is labeled 52.8% accuracy, but o3 is shown as a much shorter bar, yet it's labeled 69.1%. And
by mtlynch 1y ago
What's going on with their SWE bench graph?[0]
GPT-5 non-thinking is labeled 52.8% accuracy, but o3 is shown as a much shorter bar, yet it's labeled 69.1%. And 4o is an identical bar to o3, but it's labeled 30.8%...
[0] https://i.postimg.cc/DzkZZLry/y-axis.png https://i.postimg.cc/DzkZZLry/y-axis.png
- nonhaver 1y agoalso wondering this. had to pause the livestream to make sure i wasnt crazy. definitely eyebrow raising
- bwestergard 1y ago"GPT-5, please generate a slideshow for your launch presentation."
- Bluestein 1y ago"Dang it! Claude!, please ..."
- croemer 1y agoThe barplot is wrong, the numbers are correct. Looks like they had a dummy plot and never updated it, only the numbers to prevent leaking? Screenshot of the blog plot: https://imgur.com/a/HAxIIdC https://imgur.com/a/HAxIIdC
- hnuser123456 1y agoHaha, even with that, it says 4o does worse with 2 passes than with 1. Edit: Nevermind, just now the first one is SWE-bench and 2nd is aider.
- croemer 1y agoThose are different benchmarks
- hnuser123456 1y agoI see now on the website, the screenshot cut off the header for the first benchmark, looked like it was just comparing 1-pass and 2-pass.
- croemer 1y agoYes, sorry didn't fit everything on the screenshot.
- bufferoverflow 1y ago[dead]
- drmidnight 1y agoGPT-5 generated the chart
- lacoolj 1y agoBest answer on this page. Thanks for the laugh. I needed it.
- Upvoter33 1y agoTufte used to call this creating a "visual lie" - you just don't start the y-axis at 0, you start it wherever, in order to maximize the difference. it's dishonest.
- amarcheschi 1y ago52 above 60 seems wrong whatever way you put it
- arjie 1y agoMust be some sort of typo type thing in the presentation since the launch site has it correct here https://openai.com/index/introducing-gpt-5/#:~:text=Accuracy%2C%20pass%401-,Coding,-74.9%25 https://openai.com/index/introducing-gpt-5/#:~:text=Accuracy... Look at the image just above "Instruction following and agentic tool use"
- mcs5280 1y agoThey vibecharted
- Bluestein 1y agoNew term of art :)
- netule 1y agoThis reminds me of the agent demo's MLB stadium map from a few weeks ago: https://youtu.be/1jn_RpbPbEc?t=1435 https://youtu.be/1jn_RpbPbEc?t=1435 (at timestamp) Completely bonkers stuff.
- datadrivenangel 1y agostable diffusion is great for this!
- datadrivenangel 1y agohttps://news.ycombinator.com/item?id=44830684 https://news.ycombinator.com/item?id=44830684
- Aurornis 1y agoAs someone who spent years quadruple checking every figure in every slide for years to avoid a mistake like this, it’s very confusing to see this out of the big launch announcement of one of the most high profile startups around. Even the small presentations we gave to execs or the board were checked for errors so many times that nothing could possibly slip through.
- nicce 1y agoIt is not mistake. It is common tactic to make illusion of improvement.
- dvfjsdhgfv 1y agoWould they risk such an obvious blunder and being ridiculed for being "AI-sloppy"? I don't believe it.
- nicce 1y agoI don’t believe for mistake either. As others have said, these graphs are worth of billions. Everything is calculated. They take the risk that some will notice but most will not. They say that it is mistake for those who notice.
- crmi 1y agoPerhaps they're taking a leaf from nvidias book - influencers dunking on their bar charts gives a lot of free press coverage/mindshare
- MrNeon 1y agoI've seen that sentiment on reddit as well and I can't phantom how you think it being on purpose is more likely than a mistake when 1 - The error is so blatantly large 2 - There is a graph without error right next to it 3 - The errors are not there in the system card and the presentation page
- nicce 1y ago
- bhouston 1y agoSounds like a graph that was generated via AI. :)
- tacker2000 1y agoWow imgur has gone to shit. I open the image on mobile and then try to zoom it and bam some other “related content” is opened…!
- yz-exodao 1y agoAlso, what's this??? https://imgur.com/a/5CF34M6 https://imgur.com/a/5CF34M6
- jasonjmcghee 1y agoDeception - guessing it's % of responses that deceived the user / gave misleading information
- yz-exodao 1y agoSure, but 50.0 > 47.4...
- jasonjmcghee 1y agoOh man... didn't even notice. I've been deceived. That's bad.
- deleted 1y ago[deleted]
- godelski 1y agoIn everything except the first set of bars, bigger bar == bigger number. But also scale is really off... I don't think anything here is proportionally correct even within the same grouping.
- croemer 1y agoImgur is down, hug of death from screenshot links on HN. {"data":{"error":"Imgur is temporarily over capacity. Please try again later."},"success":false,"status":403} Or rate limited.
- koolala 1y agostats say this image got 500 views. imgur is much much more populated than HN
- mikert89 1y agoAGI is launching, lets complain about the charts
- amarcheschi 1y agoAny time now
- Mawr 1y agoDon't ask questions, just consume product.
- seydor 1y agosomeone copy pasted the 3rd bar to the 2nd
- mbowcut2 1y agoit looks like the 2nd and 3rd bar never got updated from the dummy data placeholders lol.
- edwinarbus 1y agocross-posting: https://x.com/sama/status/1953513280594751495 https://x.com/sama/status/1953513280594751495 "wow a mega chart screwup from us earlier--wen GPT-6?! correct on the blog though." blog: https://openai.com/index/introducing-gpt-5/ https://openai.com/index/introducing-gpt-5/
- 18172828286177 1y agoProbably generated by an LLM
- anigbrowl 1y ago(whispers) they're bullshit artists It's like those idiotic ads at the end of news articles. They're not going after you, the smart discerning logician, they're going after the kind of people that don't see a problem. There are a lot of not-smart people and their money is just as good as yours but easier to get.
- hansmayer 1y agoExactly this, but it will still be a net negative for all of us. Why? Increasingly I have to argue with non-technical geniuses who have "checked" some complex technical issue with ChatGPT, they themselves lacking even the basic foundations in computer science. So you have an ever increasing number of smartasses who think that this technology finally empowers them. Finally they get "level up" with that arrogant techie. And this will ultimately doom us, because as we know, idiots are in majority and they often overrule the few sane voices.