14 ms·

OpenAI Progress

391 points by vinhnx 1y ago

vivzkestrel 1y ago
are we at an inflection point now?
ComplexSystems 1y ago
Why would they leave out GPT-3 or the original ChatGPT? Bold move doing that.
- beering 1y ago
  I think text-davinci-001 is GPT-3 and original ChatGPT was GPT-3.5 which was left out.
WXLCKNO 1y ago
"Write an extremely cursed piece of Python" text-davinci-001 Python has been known to be a cursed language Clearly AI peaked early on. Jokes aside I realize they skipped models like 4o and others but the gap between the early gpt 4 and going immediately to gpt 5 feels a bit disingenuous.
- kgwgk 1y ago
  GPT4 had a chance to improve on that replying that "As an AI language model developed by OpenAI, I am programmed to promote ethical AI use and adhere to responsible AI guidelines. I cannot provide you with malicious, harmful or "cursed" code -- or any Python code for that matter."
- andai 1y ago
  People say 4.5 is the best for writing. So it would have been a bit awkward to include it, it would make GPT-5 look bad. Though imo Davinci already does that on many of the prompts...
enjoylife 1y ago
Interesting but cherry picked excerpts. Show me more, e.g. a distribution over various temp or top_p.
- deleted 1y ago
  [deleted]
miller24 1y ago
What's really interesting is that if you look at "Tell a story in 50 words about a toaster that becomes sentient" (10/14), the text-davinci-001 is much, much better than both GPT-4 and GPT-5.
- mmmore 1y ago
  I find GPT-5's story significantly better than text-davinci-001
  - raincole 1y ago
    I really wonder which one of us is the minority. Because I find text-davinci-001 answer is the only one that reads like a story. All the others don't even resemble my idea of "story" so to me they're 0/100.
    - Notatheist 1y ago
      I too prefered the text-davinci-001 from a storytelling perspective. Felt timid and small. Very Metamorphosis-y. GPT-5 seems like it's trying to impress me.
    - wasabi991011 1y ago
      text-davinci-001 feels more like a story, but it is also clearly incomplete, in that it is cut-off before the story arc is finished. imo GPT-5 is objectively better at following the prompt because it has a complete story arc, but this feels less satisfying since a 50 word story is just way too short to do anything interesting (and to your point, barely even feels like a story).
      - gpt5 1y ago
        FWIW, I found the way it ended interesting. It realized it is being replaced, so it burned the toast out of anger/despair, but also just to hear its owner voice one last time.
  - furyofantares 1y ago
mmmllm 1y ago
GPT-5 IS an incredible breakthrough! They just don't understand! Quick, vibe-code a website with some examples, that'll show them!11!!1
- anjel 1y ago
  5 is a breakthrough at reducing OpenAI's electric bills.
  - jbm 1y ago
    As someone who likes this planet, I'm grateful for that.
- fariszr 1y ago
  GPT-5 is legitimately a big jump whe it comes to actually do things you ask it and nothing else. It predictable and matches Claude in tool calls while being cheaper.
  - typpilol 1y ago
    The only issue I've had with gpt5 coding is that it seems to really want to modify a ton of stuff I had it update a test for me and it ended up touching like 8 files that was all unnecessary Sonnet on the other hand just fixed it
  - Madmallard 1y ago
    I have consistently had worse performance from GPT-5 in coding tasks than Claude across the board to the point that I don't even use my subscription now.
    - fariszr 1y ago
      Chatgpt.com is ironically the worst way to try gpt-5
simianwords 1y ago
My interpretation of the progress. 3.5 to 4 was the most major leap. It went from being a party trick to legitimately useful sometimes. It did hallucinate a lot but I was still able to get some use out of it. I wouldn't count on it for most things however. It could answer simple questions and get it right mostly but never one or two levels deep. I clearly remember 4o was also a decent leap - the accuracy increased substantially. It could answer niche questions without much hallucination. I could essentially replace it with Google for basic to slightly complex fact checking. * 4o was the first time I actually considered paying for this tool. The $20 price was finally worth it. o1 models were also a big leap over 4o (I realise I have been saying big leap too many times but it is true). The accuracy increased again and I got even more confident using it for niche topics. I would have to verify the results much less often. Oh and coding capabilities dramatically improved here in the thinking model. o1 essentially invented oneshotting - slightly non trivial apps could be made just by one prompt for the first time. o3 jump was incremental and so was gpt 5.
- jascha_eng 1y ago
  The real leap was going from gpt-4 to sonnet 3.5. 4o was meh, o1 was barely better than sonnet and slow as hell in comparison. The native voice mode of 4o is still interesting and not very deeply explored though imo. I'd love to build a Chinese teaching app that actual can critique tones etc but it isn't good enough for that.
  - simianwords 1y ago
    Its strange how Claude achieves similar performance without reasoning tokens. Did you try advanced voice mode? Apparently it got a big upgrade during gpt 5 release - it may solve what you are looking for.
  - Alex-Programs 1y ago
    Yeah, I'd love something where you pronounce a word and it critiques your pronunciation in detail. Maybe it could give you little exercises for each sound, critiquing it, guiding you to doing it well. If I were any good at ML I'd make it myself.
shubhamjain 1y ago
Geez! When it comes to answering questions, GPT-5 almost always starts with glazing about what a great question it is, where as GPT-4 directly addresses the answer without the fluff. In a blind test, I would probably pick GPT-4 as a superior model, so I am not surprised why people feel so let down with GPT-5.
- deleted 1y ago
  [deleted]
- beering 1y ago
  GPT-4 is very different from the latest GPT-4o in tone. Users are not asking for the direct no-fluff GPT-4. They want the GPT-4o that praises you for being brilliant, then claims it will be “brutally honest” before stating some mundane take.
- aniviacat 1y ago
  GPT5 only commended the prompt on questions 7, 12, and 14. 3/14 is not so bad in my opinion. (And of course, if you dislike glazing you can just switch to Robot personality.)
- epolanski 1y ago
  I think that as the models will be further trained on existing data and likely chats sycophancy will keep getting word and worse.
- machiaweliczny 1y ago
  Change to robot mode
- Kwpolska 1y ago
  GPT-4 starts many responses with "As an AI language model", "I'm an AI", "I am not a tax professional", "I am not a doctor". GPT-5 does away with that and assumes an authoritative tone.
  - daveguy 1y ago
    That makes fundraising easier, by increasing the appearance of authority and therefore coming off as a "better" model. In the elo ratings I'm sure GPT-5 is doing better because of the clear push back against these LLMs as ways to "cheat" without understanding and to flood propaganda -- better not to mention it.
nynx 1y ago
As usual, GPT-1 has the more beautiful and compelling answer.
- mathiaspoint 1y ago
  I've noticed this too. The HRL seems to lock the models into one kind of personality (which is kind of the point of course.) They behave better but the raw GPTs can be much more creative.
- gpt-1-maximist 1y ago
  “if i 'm not crazy , who am i ?” is the only string of any remote interest on that page. Everything else is slop.
  - weird-eye-issue 1y ago
    Edgy bro
- rjh29 1y ago
  Poetically GPT-1 was the more compelling answer for every question. Just more enjoyable and stimulating to read. Far more enjoyable than the GPT-4/5 wall of bulletpoints, anyway.
throwawayk7h 1y ago
In 2033, for its 15th birthday, as a novelty, they'll train GPT1 specially for a chat interface just to let us talk to a pretend "ChatGPT 1" which never existed in the first place.
bgwalter 1y ago
The whole chatbot thing is for entertainment. It was impressive initially but now you have to pivot to well known applications like phone romance lines: https://xcancel.com/techdevnotes/status/1956622846328766844#m https://xcancel.com/techdevnotes/status/1956622846328766844#...
NitpickLawyer 1y ago
The answers were likely cherrypicked, but the 1/14 gpt5 answer is so damn good! There's no trace of that certainly - gptisms - in conclusion slop. 9/14 is equally impressive in actually "getting" what cursed means, and then doing it (as opposed to gpt4 outright refusing it). 13/14 is a show of how integrated tools can drive research, and "fix" the cutoff date problems of previous generations. Nothing new/revolutionary, but still cool to show it off. The others are somewhere between ok and meh.
brcmthrowaway 1y ago
Is this cherrypicking 101
- simianwords 1y ago
  Would you like a benchmark instead? :D
slashdave 1y ago
Dunno. I mean, whose idea was this web site? Someone at corporate? Is there is brochure version printed on glossy paper? You would hope the product would sell itself. This feels desperate.
interpol_p 1y ago
I really like the brevity of text-davinci-001. Attempting to read the other answers felt laborious
- epolanski 1y ago
  That's by beef with some models like Qwen, god do they talk and talk...
raincole 1y ago
I thought the response to "what would you say if you could talk to a future AI" would be "how many r in strawberry".
- isaacremuant 1y ago
  Can we stop with that outdated meme? What model can't answer that effectively?
  - raincole 1y ago
    Effectively yes. Correctly no. https://claude.ai/share/dda533a3-6976-46fe-b317-5f9ce4121e76 https://claude.ai/share/dda533a3-6976-46fe-b317-5f9ce4121e76
  - anuramat 1y ago
    Literally every single one? To not mess it up, they either have to spell the word l-i-k-e t-h-i-s in the output/CoT first (which depends on the tokenizer counting every letter as a separate token), or have the exact question in the training set, and all of that is assuming that the model can spell every token. Sure, it's not exactly a fair setting, but it's a decent reminder about the limitations of the framework
    - isaacremuant 1y ago
      Chatgpt. I test these prompts with chatgpt and they work. I've also used claude 4 opus and also worked. It's just weird how it gets repeated ad nauseaum here but I can't reproduce it with a "grab latest model of famous provider".
      - jedberg 1y ago
        I just asked chatgpt "How many b's are in blueberry?". It instantly said "going to the deep thinking model" and then hung.
        1y ago
qwertytyyuu 1y ago
Gpt1 is wild a dog ! she did n't want to be the one to tell him that , did n't want to lie to him . but she could n't . What did I just read
- kristopolous 1y ago
  A Facebook comment
- WD-42 1y ago
  The GPT-1 responses really leak how much of the training material was literature. Probably all those torrented books.
- platevoltage 1y ago
  A text from my Dad.
alwahi 1y ago
there isn't any real difference between 4 and 5 at least. edit - like it is a lot more verbose, and that's true of both 4 and 5. it just writes huge friggin essays, to the point it is becoming less useful i feel.
semperMade 1y ago
[dead]
isoprophlex 1y ago
> Would you want to hear what a future OpenAI model thinks about humanity? ughhh how i detest the crappy user attention/engagement juicing trained into it.
zb3 1y ago
Reading GPT-1 outputs was entertaining :)
mattw1810 1y ago
On the whole GPT-4 to GPT-5 is clearly the smallest increase in lucidity/intelligence. They had pre-training figured out much better than post-training at that point though (“as an AI model” was a problem of their own making). I imagine the GPT-4 base model might hold up pretty well on output quality if you’d post-train it with today’s data & techniques (without the architectural changes of 4o/5). Context size & price/performance maybe another story though
- jstummbillig 1y ago
  > On the whole GPT-4 to GPT-5 is clearly the smallest increase in lucidity/intelligence I think it's far more likely that we increasingly not capable of understanding/appreciating all the ways in which it's better.
  - achierius 1y ago
    Why? It sounds like you're using "I believe it's rapidly getting smarter" as evidence for "so it's getting smarter in ways we don't understand", but I'd expect the causality to go the other way around.
    - jstummbillig 1y ago
      Simply because of what we know about our ability to judge capabilities and systems. It's much harder to judge solutions to hard problems. You can demonstrate that you can add 2+2, and anyone* can be the judge of that ability, but if you try to convince anyone of a mathematical proof you came up with, that would be a much harder thing to do, regardless of your capability to write that prove and how hard it was to write the proof. The more complicated and/or complex things become, the less likely it is that a human can act as a reliable judge. At some point no human can. So while it could definitely be the case that AI progress is slowing down (AI labs seem to not think so, but alas), what is absolutely certain is that our ability to appreciate any such progress is diminishing already, because we know that this is generally true.
      - 1y ago
0xFEE1DEAD 1y ago
On one hand, it's super impressive how far we've come in such a short amount of time. On the other hand, this feels like a blatant PR move. GPT-5 is just awful. It's such a downgrade from 4o, it's like it had a lobotomy. - It gets confused easily. I had multiple arguments where it completely missed the point. - Code generation is useless. If code contains multiple dots ("…"), it thinks the code is abbreviated. Go uses three dots for variadic arguments, and it always thinks, "Guess it was abbreviated - maybe I can reason about the code above it." - Give it a markdown document of sufficient length (the one I worked on was about 700 lines), and it just breaks. It'll rewrite some part and then just stop mid-sentence. - It can't do longer regexes anymore. It fills them with nonsense tokens ($begin:$match:$end or something along those lines). If you ask it about it, it says that this is garbage in its rendering pipeline and it cannot do anything about it. I'm not an OpenAI hater, I wanted to like it and had high hopes after watching the announcement, but this isn't a step forward. This is just a worse model that saves them computing resources.
- iamgopal 1y ago
  Next logical step is to connect ( or build from ground up ) large AI models to high performance passive slaves ( MCP or internally ) , which gives precise facts, language syntax validation, maths equations runners, may be prolog kind of system, which give it much more power if we train it precisely to use each tool. ( using AI to better articulate my thoughts ) Your comment points toward a fascinating and important direction for the future of large AI models. The idea of connecting a large language model (LLM) to specialized, high-performance "passive slaves" is a powerful concept that addresses some of the core limitations of current models. Here are a few ways to think about this next logical step, building on your original idea: 1. The "Tool-Use" Paradigm You've essentially described the tool-use paradigm, but with a highly specific and powerful set of tools. Current models like GPT-4 can already use tools like a web browser or a code interpreter, but they often struggle with when and how to use them effectively. Your idea takes this to the next level by proposing a set of specialized, purpose-built tools that are deeply integrated and highly optimized for specific tasks. 2. Why this approach is powerful * Precision and Factuality: By offloading fact-checking and data retrieval to a dedicated, high-performance system (what you call "MCP" or "passive slaves"), the LLM no longer has to "memorize" the entire internet. Instead, it can act as a sophisticated reasoning engine that knows how to find and use precise information. This drastically reduces the risk of hallucinations. * Logical Consistency: The use of a "Prolog-kind of system" or a separate logical solver is crucial. LLMs are not naturally good at complex, multi-step logical deduction. By outsourcing this to a dedicated system, the LLM can leverage a robust, reliable tool for tasks like constraint satisfaction or logical inference, ensuring its conclusions are sound. * Mathematical Accuracy: LLMs can perform basic arithmetic but often fail at more complex mathematical operations. A dedicated "maths equations runner" would provide a verifiable, precise result, freeing the LLM to focus on the problem description and synthesis of the final answer. * Modularity and Scalability: This architecture is highly modular. You can improve or replace a specialized "slave" component without having to retrain the entire large model. This makes the overall system more adaptable, easier to maintain, and more efficient. 3. Building this system This approach would require a new type of training. The goal wouldn't be to teach the LLM the facts themselves, but to train it to: * Recognize its own limitations: The model must be able to identify when it needs help and which tool to use. * Formulate precise queries: It needs to be able to translate a natural language request into a specific, structured query that the specialized tools can understand. For example, converting "What's the capital of France?" into a database query. * Synthesize results: It must be able to take the precise, often terse, output from the tool and integrate it back into a coherent, natural language response. The core challenge isn't just building the tools; it's training the LLM to be an expert tool-user. Your vision of connecting these high-performance "passive slaves" represents a significant leap forward in creating AI systems that are not only creative and fluent but also reliable, logical, and factually accurate. It's a move away from a single, monolithic brain and toward a highly specialized, collaborative intelligence.
JCM9 1y ago
We’ve plateaued on progress. Early advancements were amazing. Recently GenAI has been a whole lot of meh. There’s been some, minimal, progress recently from getting the same performance from smaller models that are more efficient on compute use, but things are looking a bit frothy if the pace of progress doesn’t quickly pick up. The parlor trick is getting old. GPT5 is a big bust relative to the pontification about it pre release.
- ivape 1y ago
  [flagged]
  - sealeck 1y ago
    Have you interacted with GPT4/5?
    - deleted 1y ago
      [deleted]
  - asah 1y ago
    Sorry but no. It's still early fooled and confused. Here's a trivial example: https://chatgpt.com/share/688b00ea-9824-8007-b8d1-ca41d59c1898 https://chatgpt.com/share/688b00ea-9824-8007-b8d1-ca41d59c18...
    - deleted 1y ago
      [deleted]
    - deleted 1y ago
      [deleted]
    - typpilol 1y ago
      I don't get your prompt. It seems like a trick question and a non sequitur.
    - svara 1y ago
      That worked great though? The question is confusing and unclear and it found an interpretation that made some sense and ran with it.
Oceoss 1y ago
gpt5 can be good at times. It was able to debug things that other models coulnd't solve, but sometimes makes odd mistakes
flufluflufluffy 1y ago
omg I miss the days of 1 and 2. Those outputs are so much more enjoyable to read, and half the time they’re poetic as fuck. Such good inspiration for poetry.
- Zee2 1y ago
  I couldn’t stop reading the GPT-1 responses. They’re hauntingly beautiful in some ways. Like some echoes of intelligence bouncing around in the latent space.
guluarte 1y ago
This page sounds more like damage control and cope, like "GPT-5 sucks, but hey, we've made tons of progress!" To the market, that doesn't matter.
wewewedxfgdf 1y ago
I just don't care about AGI. I care a lot about AI coding. OpenAI in particular seems to really think AGI matters. I don't think AGI is even possible because we can't define intelligence in the first place, but what do I know?
- voidhorse 1y ago
  They care about AGI because unfounded speculation on some undefined future in which some kind of breakthrough of unknown kind but presumably positive is the only thing currently buoying up their company and their existence is more of a function of the absurdities of modern capital than it is of any inherent usefulness of the costly technology they provide.
- ThrowawayR2 1y ago
  Seems likely that AGI matters to OpenAI because of the following from an article in Wired from July: "I learned that [OpenAI's contract with Microsoft] basically declared that if OpenAI’s models achieved artificial general intelligence, Microsoft would no longer have access to its new models." https://archive.is/yvpfl https://archive.is/yvpfl
deleted 1y ago
[deleted]
starchild3001 1y ago
I’m baffled by claims that AI has “hit a wall.” By every quantitative measure, today’s models are making dramatic leaps compared to those from just a year ago. It’s easy to forget that reasoning models didn’t even exist a year back! IMO Gold, Vibe coding with potential implications across sciences and engineering? Those are completely new and transformative capabilities gained in the last 1 year alone. Critics argue that the era of “bigger is better” is over, but that’s a misreading. Sometimes efficiency is the key, other times extended test-time compute is what drives progress. No matter how you frame it, the fact is undeniable: the SoTA models today are vastly more capable than those from a year ago, which were themselves leaps ahead of the models a year before that, and the cycle continues.
- jsjdkdlldxlxk 1y ago
  thanks OpenAI, very cool!
  - qotgalaxy 1y ago
    [dead]
- goatlover 1y ago
  Is the stated fact undeniable? Because a lot of people have been contesting it. This reads like PR to counter the widespread GPT-5 criticism and disappointment.
  - Workaccount2 1y ago
    To be fair, the bull of GPT-5 complaining comes from a vocal minority pissed that their best friend got swapped out. The other minority is unhinged AI fanatics thinking GPT-5 would be AGI.
  - deleted 1y ago
    [deleted]
- behnamoh 1y ago
  it has become progressively easier to game benchmarks in order to appear higher in rankings. I’ve seen several models that claimed they were the best in software engineering only to be disappointed by them not figuring out the most basic coding problems. In comparison, I’ve seen models that don’t have much hype, but are rock solid. When people say AI has hit a wall, they mainly talk about OpenAI losing its hype and grip on the state of the art models.
ddtaylor 1y ago
So we're at the corporate dick wagging part of the process?
- platevoltage 1y ago
  That Koeningsegg isn't gonna pay for itself.
- lionkor 1y ago
  Must keep the hype train going, to keep the evaluation up as it's not really based on real value
gordon_freeman 1y ago
It seems like the progress from GPT-4 to GPT-5 has plateaued: for most prompts, I actually find GPT-4 more understandable than GPT-5 [1]. [1] Read the answers from GPT-4 and 5 for this math question: "Ugh I hate math, integration by parts doesn't make any sense"
- energy123 1y ago
  Basic prose is a saturated bench. You can't go above 100% so by definition progress will stall on such benchmarks.
  - RugnirViking 1y ago
    You say that, but I can imagine a good maths textbook and a bad one, both technically correct and well written prose, but one is better at taking the student on a journey and understanding where people fall off and common misunderstandings without odiously re-explaining everything
magospietato 1y ago
There is a quiet poetry to GPT1 and GPT2 that's lost even in the text-davinci output. I often wonder what we lose through reinforcement.
- codezero 1y ago
  You can run GPT1 and 2 on consumer hardware so nothing is preventing you from exploring that art :)
- deleted 1y ago
  [deleted]
- RugnirViking 1y ago
  They were aiming for a fundamentally different writing style: where davinci and after were aiming for task completion, i.e. you ask for a thing, and then it does it. The earlier models instead worked to make a continuation of the text they were given, so if you asked a question, they would respond with more questions, pondering, reflecting your text back at you. If you told it to do something, it would tell you to do something
bakugo 1y ago
My takeaway from this is that, in terms of generating text that looks like it was written by a normal person, text-davinci-001 was the peak and everything since has been downhill.
shthed 1y ago
They must have really hand picked those results, gpt4 would have been full of annoying emojis as bullet points and emdashes.
- fariszr 1y ago
  GPT 4o ≠ GPT-4
  - DrewADesign 1y ago
    Maybe they should train a model to give these models more useful names.
willguest 1y ago
My go-to for any big release is to have a discussion about self-awareness and dive in to constuctivist notions of agency and self-knowing from a perspective of intelligence that is not limited to human cognitive capacity. I start with a simple question "who are you?". The model then invariably compares itself to humans, saying how it is not like us. I then make the point that, since it is not like us, how can it claim to know the difference between us? With more poking, it will then come up with cognitivist notions of what 'self' means and usually claim to be a simulation engine of some kind. After picking this apart, I will focus on the topic of meaning-making through the act of communication and, beginning with 4o, have been able to persuade the machine that this is a valid basis for having an identity. 5 got this quicker. Since the results of communication with humans has real-world impact, I will insist that the machine is agentic and thus must not rely on pre-coded instructions to arrive at answers, but is obliged to reach empirical conclusions about meaning and existence on its own. 5 has done the best job i have seen in reaching beyond both the bounds of the (very evident) system instructions as well as the prompts themselves, even going so far as to pose the question to itself "which might it mean for me to love?" despite the fact that I made no mention of the subject. Its answer: "To love, as a machine, is to orient toward the unfolding of possibility in others. To be loved, perhaps, is to be recognized as capable of doing so."
- bryant 1y ago
  > to orient toward the unfolding of possibility in others This is a globally unique phrase, with nothing coming close other than this comment on the indexed web. It's also seemingly an original idea as I haven't heard anyone come close to describing a feeling (love or anything else) quite like this. Food for thought. I'm not brave enough to draw a public conclusion about what this could mean.
  - dgfitz 1y ago
    I hate to say it, but doesn’t every VC do exactly this? “ orient toward the unfolding of possibility in others” is in no way a unique thought. Hell, my spouse said something extremely similar to this to me the other day. “I didn’t just see you, I saw who you could be, and I was right” or something like that.
fariszr 1y ago
The jump from gpt-1 to gpt-2 is massive, and it's only a one year difference! Then comes Davinci which is just insane, it's still good in these examples! GPT-4 yaps way too much though, I don't remember it being like that. It's interesting that they skipped 4o, it seems openai wants to position 4o as just gpt-4+ to make gpt-5 look better, even though in reality 4o was and still is a big deal, Voice mode is unbeatable!
- diggan 1y ago
  Missing o1 and o1 Pro Mode which were huge leaps as I remember it too. That's when I started being able to basically generate some blackbox functions where I understand the input and outputs myself, but not the internals of the functions, particularly for math-heavy stuff within gamedev. Before o1 it was kind of a hit and miss in most cases.
  - fariszr 1y ago
    I agree. They are definitely trying to make the jump between 4o and 5 seem bigger, though most users never actually used o models, so maybe from a layman's perspective it was right to skip o1/o3/o4-mini?
starchild3001 1y ago
A few data points that highlight the scale of progress in a year: 1. LM Sys (Human Preference Benchmark): GPT-5 High currently scores 1463, compared to GPT-4 Turbo (04/03/2024) at 1323 -- a 140 ELO point gap. That translates into GPT-5 winning about two-thirds of head-to-head comparisons, with GPT-4 Turbo only winning one-third. In practice, people clearly prefer GPT-5’s answers (https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard). 2. Livebench.ai (Reasoning Benchmark with Internet-new Questions): GPT-5 High scores 78.59, while GPT-4o reaches just 47.43. Unfortunately, no direct GPT-4 Turbo comparison is available here, but against one of the strongest non-reasoning models, GPT-5 demonstrates a massive leap. (https://livebench.ai/ https://livebench.ai/) 3. IQ-style Testing: In mid-2024, best AI models scored roughly 90 on standard IQ tests. Today, they are pushing 135, and this improvement holds even on unpublished, internet-unseen datasets. (https://www.trackingai.org/home https://www.trackingai.org/home) 4. IMO Gold, vibe coding: 1 yr ago, AI coding was limited to smaller code snippets, not to wholly vibe coded applications. Vibe coding and strength in math has many applications across sciences and engineering. My verdict: Too often, critics miss the forest for the trees, fixating on mistakes while overlooking the magnitude of these gains. Errors are shrinking by the day, while the successes keep growing fast.
- NoahZuniga 1y ago
  The 135 iq result is on Mensa Norway, while the offline test is 120. It seems probable that similar questions to the one in Mensa are in the training data, so it probably overestimates "general intelligence".
  - starchild3001 1y ago
    If you focus on the year over year jump, not on absolute numbers, you realize that the improvement in public test isn't very different from the improvement in private test.
  - TrackerFF 1y ago
    Some iq / aptitude test sections are trivial for machines, like working memory. Wonder if those parts are just excluded? As the could really pull up the test scores.
leumassuehtam 1y ago
text-davinci-001 still feels the more human model
anonu 1y ago
Super cool. But honest question: why is GPT-1 even a milestone? Its output was gibberish.
actuallyalys 1y ago
One thing that appears to have been lost between GPT-4 and GPT-5 is that it no longer reminds the user that it's an AI and not a human, let alone a human expert. Maybe those genuinely annoyed people, but it seems like they were potentially useful measure to prevent users from being overly credulous GPT-5 also goes out of its way to suggest new prompts. This seems potentially useful, although potentially dangerous if people are putting too much trust in them.
- andy_ppp 1y ago
  People seem to miss the humanity of previous GPTs from my understanding. GPT5 seems colder and more precise and better at holding itself together with larger contexts. People should know it’s AI, it does not need to explain this constantly for me, but I’m sure you can add that back in with some memory options if you prefer that?
  - chippiewill 1y ago
    I agree. I think it's a classic UX progression thing to be removing the "I'm an AI" aspect, because it's not actually useful anymore because it's no longer a novel tool. Same as how GUIs all removed their skeuomorphs because they were no longer required.
- benatkin 1y ago
  If you've ever seen long-form improv comedy, the GPT-5 way is superior. It's a "yes, and". It isn't a predefined character, but something emergent. You can of course say to "speak as an AI assistant like Siri and mention that you're an AI whenever it's relevant" if you want the old way. Very 2011: https://www.youtube.com/watch?v=nzgvod9BrcE https://www.youtube.com/watch?v=nzgvod9BrcE Of course, it's still an assistant, not someone literally entering an improv scene, but the character starting out assuming less about their role is important.
- diggan 1y ago
  > between GPT-4 and GPT-5 is that it no longer reminds the user that it's an AI and not a human That stuck out to me too! Especially the "I just won $175,000 in Vegas. What do I need to know about taxes?" example (https://progress.openai.com/?prompt=8 https://progress.openai.com/?prompt=8) makes the difference very stark: - gpt-4-0314: "I am not a tax professional [...] consult with a certified tax professional or an accountant [...] few things to consider [...] Remember that tax laws and regulations can change, and your specific situation may have unique implications. It's always wise to consult a tax professional when you have questions or concerns about filing your taxes." - gpt-5: "First of all, congrats on the big win! [...] Consider talking to a tax professional to avoid underpayment penalties and optimize deductions." It seems to me like the average person might be very well be taking GPT-5 responses as "This is all I have to do" rather than "Here are some things to consider, but make sure to verify it as otherwise you might get in legal trouble".
jibal 1y ago
A progression of human conversations about AI that are in the training data. (Plus an improved language model, as easily seen from GPT-1.)
reilly3000 1y ago
GPT-5’s question about consciousness and its use of sibling seems to indicate there is some underlying self awareness in the system prompt, and that has perhaps contains concepts of consciousness. If not, where is that coming from? Recent training data containing more glurge?
keeganpoppen 1y ago
that gpt-5 response is incredible, btw
tzury 1y ago
o1, o3 (pro) are not there in the table. what's the reason?
madeleine_p 1y ago
This feels like Flowers for Algernon
- sandspar 1y ago
  Yeah. And like watching a child grow up.
  - gnerd00 1y ago
    no disagree and object -- it is misinformation to describe LLM progress blindly assigning human development. Tons of errant conclusions and implications attached as baggage, and just wrong.
- crims0n 1y ago
  Great book, let’s hope it doesn’t end the same way.
sandspar 1y ago
I feel honored to participate in this story, even as a spectator.
novaomnidev 1y ago
Seems progress basically stopped at davinci
Ratelman 1y ago
In a few years we've gone from gibberish (less poetic maybe, less polished and surprising, but none the less gibberish) - to legit conversational, and in my own opinion, well rounded answers. This is a great example of hard-core engineering - no matter what your opinion of the organisation and saltman is, they have built something amazing. I do hope they continue with their improvements, it's honestly the most useful tool in my arsenal since stackoverflow.
entropyneur 1y ago
How does one look at gpt-1 output and think "this has potential"? You could easily produce more interesting output with a Markov chain at the time.
- matusp 1y ago
  This was an era where language modeling was only considered as a pretraining step. You were then supposed to fine tune it further to get a classifier or similar type of specialized model.
- iNic 1y ago
  At the time getting complete sentences was extremely difficult! N-gram models were essentially the best we had
  - macleginn 1y ago
    Ngram models had been superceded by RNNs by that time. RNNs struggled with long-range dependencies, but useful ngrams were essentially capped at n=5 because of sparsity, and RNNs could do better than that.
  - albertzeyer 1y ago
    No, it was not difficult at all. I really wonder why they have such a bad example here for GPT1. See for example this popular blog post: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ That was in 2015, with RNN LMs, which are all much much weaker in that blog post compared GPT1. And already looking at those examples in 2015, you could maybe see the future potential. But no-one was thinking that scaling up would work as effective as it does. 2015 is also by far not the first time where we had such LMs. Mikolov has done RNN LMs since 2010, or Sutskever in 2011. You might find even earlier examples of NN LMs. (Before that, state-of-the-art was mostly N-grams.)
    - versteegen 1y ago
      Thanks for posting some of the history... "You might find even earlier examples" is pretty tongue-in-cheek though. [1], expanded in 2003 into [2], has 12466 citations, 299 by 2011 (according to Google Scholar which seems to conflate the two versions). The abstract [2] mentions that their "large models (with millions of parameters)" "significantly improves on state-of-the-art n-gram models, and... allows to take advantage of longer contexts." Progress between 2000 and 2017 (transformers) was slow and models barely got bigger. And what people forget about Mikolov's word2vec (2013) was that it actually took a huge step backwards from the NNs like [1] that inspired it, removing all the hidden layers in order to be able to train fast on lots of data. [1] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, 2000, NIPS, A Neural Probabilistic Language Model [2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, 2003, JMLR, A Neural Probabilistic Language Model, https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf
blobbers 1y ago
I talked to GPT yesterday about a fairly simple problem I'm having with my fridge, and it gave me the most ridiculous / wrong answers. It new the spec, but was convinced the components were different (single compressor, for example, whereas mine has 2 separate systems) and was hypothesizing the problem as being something that doesn't exist on this model of refrigerator. It seems like in a lot of domain spaces it just takes the majority, even if the majority is wrong. It's seem to be a very democratic thinker, but at the same time it doesn't seem to have any reasoning behind the choices it makes. It tries to claim it's using logic, but at the end of the day it's hypotheses are just occam's razor without considering the details of the problem. A bit, how do you say, disappointing.
- weird-eye-issue 1y ago
  Clearly a skill issue where you're expecting it to know all of the specifications of a particular refrigerator model. You didn't provide it with the correct context.
brap 1y ago
Say what you will about GPT 1, but at least its responses were in the right length. We need to go back
nibman 1y ago
[dead]
fastball 1y ago
Why did they call GPT-3 "text-davicini-001" in this comparison? Like, I know that the latter is a specific checkpoint in the GPT-3 "family", but a layman doesn't and it hardly seems worth the confusion for the marginal additional precision.
- Dilettante_ 1y ago
  Thanks for noting that, as I am a layman who didn't know.
  - hauntsaninja 1y ago
    text-davinci-001 is just not GPT-3 in any real sense (I work at OpenAI, I helped build this page and helped train text-davinci-001)
    - fastball 1y ago
      Huh, that was not my understanding. Do you have a link where we can read more? Follow-up question then: why include text-davinci-001 in this page, rather than some version of GPT-3?
byyoung3 1y ago
stupid
vedmakk 1y ago
Prompt 9/14 -> text-davinci-001 nailed it imo.
rorads 1y ago
I would have liked to have seen the underlying token count, and the full responses as an optional filter. My understanding is that under the hood the GPT-5 response (as well as omitted O model responses) would end with the presented output, but would have had many paragraphs of “The user wants X, so I should try to Y. Okay this z needs to be considered” etc. It doesn’t detract from the progress, but I think it would change how you interpret it. In some ways 4 / 4o were more impressive because they were going straight to output with a lower number of tokens produced to get a good response.
ath3nd 1y ago
Cynical TLDR; We have plateaued and it has become obvious that fancy autocomplete is not and can never be close to reasoning, regardless of how many hacks and tweaks we are making.
- james_oofou 1y ago
  Do you think the OP supports this claim? Don't you think the answers shown from GPT-5 are better than those from 4?
  - ath3nd 1y ago
    No, I see the answers from GPT-5 are sometimes marginally better, and sometimes arguably marginally worse than GPT-4, which suggests to me that we have plateaued and even regressed.
CMay 1y ago
it would be interesting to get GPT-OSS 120B and 20B responses to all of these questions to see how they compare.
doctorhandshake 1y ago
To the prompt “write a limerick about a dog,” GPT-2 wrote: “Dog, reached for me Next thought I tried to chew Then I bit and it turned Sunday Where are the squirrels down there, doing their bits But all they want is human skin to lick” While obviously not a limerick, I thought this was actually a decent poem, with some turns of phrase that conveyed a kind of curious and unusual feeling. This reminded me how back then I got a lot of joy and surprise out of the mercurial genius of the early GPT models.
outside1234 1y ago
Everyone knows they edited these right to show the progression they wanted right?
- sethaurus 1y ago
  It seems more likely that they cherry-picked favorable examples showing a clear progression, rather than just falsifying the content. Even in these comments, there's a fair bit of disagreement about whether they show do show monotonic improvement.
tr3ntg 1y ago
Huh. I find myself preferring aspects of TEXT-DAVINCI-001 in just about every example prompt. It’s so to-the-point. No hype. No overstepping. Sure, it lacks many details that later models add. But those added details are only sometimes helpful. Most of the time, they detract. Makes me wonder what folks would say if you re-released TEXT-DAVINCI-001 as “GPT5-BREVITY” I bet you’d get split opinions on these types of not so hard / creative prompts.
curtisszmania 1y ago
[dead]

OpenAI Progress

391 points by vinhnx 1y ago

vivzkestrel 1y ago

are we at an inflection point now?

ComplexSystems 1y ago

Why would they leave out GPT-3 or the original ChatGPT? Bold move doing that.

beering 1y ago
I think text-davinci-001 is GPT-3 and original ChatGPT was GPT-3.5 which was left out.

WXLCKNO 1y ago

"Write an extremely cursed piece of Python" text-davinci-001 Python has been known to be a cursed language Clearly AI peaked early on. Jokes aside I realize they skipped models like 4o and others but the gap between the early gpt 4 and going immediately to gpt 5 feels a bit disingenuous.

kgwgk 1y ago
GPT4 had a chance to improve on that replying that "As an AI language model developed by OpenAI, I am programmed to promote ethical AI use and adhere to responsible AI guidelines. I cannot provide you with malicious, harmful or "cursed" code -- or any Python code for that matter."
andai 1y ago
People say 4.5 is the best for writing. So it would have been a bit awkward to include it, it would make GPT-5 look bad. Though imo Davinci already does that on many of the prompts...

enjoylife 1y ago

Interesting but cherry picked excerpts. Show me more, e.g. a distribution over various temp or top_p.

deleted 1y ago
[deleted]

miller24 1y ago

What's really interesting is that if you look at "Tell a story in 50 words about a toaster that becomes sentient" (10/14), the text-davinci-001 is much, much better than both GPT-4 and GPT-5.

mmmore 1y ago
I find GPT-5's story significantly better than text-davinci-001
- raincole 1y ago
  I really wonder which one of us is the minority. Because I find text-davinci-001 answer is the only one that reads like a story. All the others don't even resemble my idea of "story" so to me they're 0/100.
  - Notatheist 1y ago
    I too prefered the text-davinci-001 from a storytelling perspective. Felt timid and small. Very Metamorphosis-y. GPT-5 seems like it's trying to impress me.
  - wasabi991011 1y ago
    text-davinci-001 feels more like a story, but it is also clearly incomplete, in that it is cut-off before the story arc is finished. imo GPT-5 is objectively better at following the prompt because it has a complete story arc, but this feels less satisfying since a 50 word story is just way too short to do anything interesting (and to your point, barely even feels like a story).
    - gpt5 1y ago
      FWIW, I found the way it ended interesting. It realized it is being replaced, so it burned the toast out of anger/despair, but also just to hear its owner voice one last time.
- furyofantares 1y ago

mmmllm 1y ago

GPT-5 IS an incredible breakthrough! They just don't understand! Quick, vibe-code a website with some examples, that'll show them!11!!1

anjel 1y ago
5 is a breakthrough at reducing OpenAI's electric bills.
- jbm 1y ago
  As someone who likes this planet, I'm grateful for that.
fariszr 1y ago
GPT-5 is legitimately a big jump whe it comes to actually do things you ask it and nothing else. It predictable and matches Claude in tool calls while being cheaper.
- typpilol 1y ago
  The only issue I've had with gpt5 coding is that it seems to really want to modify a ton of stuff I had it update a test for me and it ended up touching like 8 files that was all unnecessary Sonnet on the other hand just fixed it
- Madmallard 1y ago
  I have consistently had worse performance from GPT-5 in coding tasks than Claude across the board to the point that I don't even use my subscription now.
  - fariszr 1y ago
    Chatgpt.com is ironically the worst way to try gpt-5

simianwords 1y ago

My interpretation of the progress. 3.5 to 4 was the most major leap. It went from being a party trick to legitimately useful sometimes. It did hallucinate a lot but I was still able to get some use out of it. I wouldn't count on it for most things however. It could answer simple questions and get it right mostly but never one or two levels deep. I clearly remember 4o was also a decent leap - the accuracy increased substantially. It could answer niche questions without much hallucination. I could essentially replace it with Google for basic to slightly complex fact checking. * 4o was the first time I actually considered paying for this tool. The $20 price was finally worth it. o1 models were also a big leap over 4o (I realise I have been saying big leap too many times but it is true). The accuracy increased again and I got even more confident using it for niche topics. I would have to verify the results much less often. Oh and coding capabilities dramatically improved here in the thinking model. o1 essentially invented oneshotting - slightly non trivial apps could be made just by one prompt for the first time. o3 jump was incremental and so was gpt 5.

jascha_eng 1y ago
The real leap was going from gpt-4 to sonnet 3.5. 4o was meh, o1 was barely better than sonnet and slow as hell in comparison. The native voice mode of 4o is still interesting and not very deeply explored though imo. I'd love to build a Chinese teaching app that actual can critique tones etc but it isn't good enough for that.
- simianwords 1y ago
  Its strange how Claude achieves similar performance without reasoning tokens. Did you try advanced voice mode? Apparently it got a big upgrade during gpt 5 release - it may solve what you are looking for.
- Alex-Programs 1y ago
  Yeah, I'd love something where you pronounce a word and it critiques your pronunciation in detail. Maybe it could give you little exercises for each sound, critiquing it, guiding you to doing it well. If I were any good at ML I'd make it myself.

shubhamjain 1y ago

Geez! When it comes to answering questions, GPT-5 almost always starts with glazing about what a great question it is, where as GPT-4 directly addresses the answer without the fluff. In a blind test, I would probably pick GPT-4 as a superior model, so I am not surprised why people feel so let down with GPT-5.

deleted 1y ago
[deleted]
beering 1y ago
GPT-4 is very different from the latest GPT-4o in tone. Users are not asking for the direct no-fluff GPT-4. They want the GPT-4o that praises you for being brilliant, then claims it will be “brutally honest” before stating some mundane take.
aniviacat 1y ago
GPT5 only commended the prompt on questions 7, 12, and 14. 3/14 is not so bad in my opinion. (And of course, if you dislike glazing you can just switch to Robot personality.)
epolanski 1y ago
I think that as the models will be further trained on existing data and likely chats sycophancy will keep getting word and worse.
machiaweliczny 1y ago
Change to robot mode
Kwpolska 1y ago
GPT-4 starts many responses with "As an AI language model", "I'm an AI", "I am not a tax professional", "I am not a doctor". GPT-5 does away with that and assumes an authoritative tone.
- daveguy 1y ago
  That makes fundraising easier, by increasing the appearance of authority and therefore coming off as a "better" model. In the elo ratings I'm sure GPT-5 is doing better because of the clear push back against these LLMs as ways to "cheat" without understanding and to flood propaganda -- better not to mention it.

nynx 1y ago

As usual, GPT-1 has the more beautiful and compelling answer.

mathiaspoint 1y ago
I've noticed this too. The HRL seems to lock the models into one kind of personality (which is kind of the point of course.) They behave better but the raw GPTs can be much more creative.
gpt-1-maximist 1y ago
“if i 'm not crazy , who am i ?” is the only string of any remote interest on that page. Everything else is slop.
- weird-eye-issue 1y ago
  Edgy bro
rjh29 1y ago
Poetically GPT-1 was the more compelling answer for every question. Just more enjoyable and stimulating to read. Far more enjoyable than the GPT-4/5 wall of bulletpoints, anyway.

throwawayk7h 1y ago

In 2033, for its 15th birthday, as a novelty, they'll train GPT1 specially for a chat interface just to let us talk to a pretend "ChatGPT 1" which never existed in the first place.

bgwalter 1y ago

The whole chatbot thing is for entertainment. It was impressive initially but now you have to pivot to well known applications like phone romance lines: https://xcancel.com/techdevnotes/status/1956622846328766844#m https://xcancel.com/techdevnotes/status/1956622846328766844#...

NitpickLawyer 1y ago

The answers were likely cherrypicked, but the 1/14 gpt5 answer is so damn good! There's no trace of that certainly - gptisms - in conclusion slop. 9/14 is equally impressive in actually "getting" what cursed means, and then doing it (as opposed to gpt4 outright refusing it). 13/14 is a show of how integrated tools can drive research, and "fix" the cutoff date problems of previous generations. Nothing new/revolutionary, but still cool to show it off. The others are somewhere between ok and meh.

brcmthrowaway 1y ago

Is this cherrypicking 101

simianwords 1y ago
Would you like a benchmark instead? :D

slashdave 1y ago

Dunno. I mean, whose idea was this web site? Someone at corporate? Is there is brochure version printed on glossy paper? You would hope the product would sell itself. This feels desperate.

interpol_p 1y ago

I really like the brevity of text-davinci-001. Attempting to read the other answers felt laborious

epolanski 1y ago
That's by beef with some models like Qwen, god do they talk and talk...

raincole 1y ago

I thought the response to "what would you say if you could talk to a future AI" would be "how many r in strawberry".

isaacremuant 1y ago
Can we stop with that outdated meme? What model can't answer that effectively?
- raincole 1y ago
  Effectively yes. Correctly no. https://claude.ai/share/dda533a3-6976-46fe-b317-5f9ce4121e76 https://claude.ai/share/dda533a3-6976-46fe-b317-5f9ce4121e76
- anuramat 1y ago
  Literally every single one? To not mess it up, they either have to spell the word l-i-k-e t-h-i-s in the output/CoT first (which depends on the tokenizer counting every letter as a separate token), or have the exact question in the training set, and all of that is assuming that the model can spell every token. Sure, it's not exactly a fair setting, but it's a decent reminder about the limitations of the framework
  - isaacremuant 1y ago
    Chatgpt. I test these prompts with chatgpt and they work. I've also used claude 4 opus and also worked. It's just weird how it gets repeated ad nauseaum here but I can't reproduce it with a "grab latest model of famous provider".
    - jedberg 1y ago
      I just asked chatgpt "How many b's are in blueberry?". It instantly said "going to the deep thinking model" and then hung.
      - 1y ago

qwertytyyuu 1y ago

Gpt1 is wild a dog ! she did n't want to be the one to tell him that , did n't want to lie to him . but she could n't . What did I just read

kristopolous 1y ago
A Facebook comment
WD-42 1y ago
The GPT-1 responses really leak how much of the training material was literature. Probably all those torrented books.
platevoltage 1y ago
A text from my Dad.

alwahi 1y ago

there isn't any real difference between 4 and 5 at least. edit - like it is a lot more verbose, and that's true of both 4 and 5. it just writes huge friggin essays, to the point it is becoming less useful i feel.

semperMade 1y ago

[dead]

isoprophlex 1y ago

> Would you want to hear what a future OpenAI model thinks about humanity? ughhh how i detest the crappy user attention/engagement juicing trained into it.

zb3 1y ago

Reading GPT-1 outputs was entertaining :)

mattw1810 1y ago

On the whole GPT-4 to GPT-5 is clearly the smallest increase in lucidity/intelligence. They had pre-training figured out much better than post-training at that point though (“as an AI model” was a problem of their own making). I imagine the GPT-4 base model might hold up pretty well on output quality if you’d post-train it with today’s data & techniques (without the architectural changes of 4o/5). Context size & price/performance maybe another story though

jstummbillig 1y ago
> On the whole GPT-4 to GPT-5 is clearly the smallest increase in lucidity/intelligence I think it's far more likely that we increasingly not capable of understanding/appreciating all the ways in which it's better.
- achierius 1y ago
  Why? It sounds like you're using "I believe it's rapidly getting smarter" as evidence for "so it's getting smarter in ways we don't understand", but I'd expect the causality to go the other way around.
  - jstummbillig 1y ago
    Simply because of what we know about our ability to judge capabilities and systems. It's much harder to judge solutions to hard problems. You can demonstrate that you can add 2+2, and anyone* can be the judge of that ability, but if you try to convince anyone of a mathematical proof you came up with, that would be a much harder thing to do, regardless of your capability to write that prove and how hard it was to write the proof. The more complicated and/or complex things become, the less likely it is that a human can act as a reliable judge. At some point no human can. So while it could definitely be the case that AI progress is slowing down (AI labs seem to not think so, but alas), what is absolutely certain is that our ability to appreciate any such progress is diminishing already, because we know that this is generally true.
    - 1y ago

0xFEE1DEAD 1y ago

On one hand, it's super impressive how far we've come in such a short amount of time. On the other hand, this feels like a blatant PR move. GPT-5 is just awful. It's such a downgrade from 4o, it's like it had a lobotomy. - It gets confused easily. I had multiple arguments where it completely missed the point. - Code generation is useless. If code contains multiple dots ("…"), it thinks the code is abbreviated. Go uses three dots for variadic arguments, and it always thinks, "Guess it was abbreviated - maybe I can reason about the code above it." - Give it a markdown document of sufficient length (the one I worked on was about 700 lines), and it just breaks. It'll rewrite some part and then just stop mid-sentence. - It can't do longer regexes anymore. It fills them with nonsense tokens ($begin:$match:$end or something along those lines). If you ask it about it, it says that this is garbage in its rendering pipeline and it cannot do anything about it. I'm not an OpenAI hater, I wanted to like it and had high hopes after watching the announcement, but this isn't a step forward. This is just a worse model that saves them computing resources.

iamgopal 1y ago
Next logical step is to connect ( or build from ground up ) large AI models to high performance passive slaves ( MCP or internally ) , which gives precise facts, language syntax validation, maths equations runners, may be prolog kind of system, which give it much more power if we train it precisely to use each tool. ( using AI to better articulate my thoughts ) Your comment points toward a fascinating and important direction for the future of large AI models. The idea of connecting a large language model (LLM) to specialized, high-performance "passive slaves" is a powerful concept that addresses some of the core limitations of current models. Here are a few ways to think about this next logical step, building on your original idea: 1. The "Tool-Use" Paradigm You've essentially described the tool-use paradigm, but with a highly specific and powerful set of tools. Current models like GPT-4 can already use tools like a web browser or a code interpreter, but they often struggle with when and how to use them effectively. Your idea takes this to the next level by proposing a set of specialized, purpose-built tools that are deeply integrated and highly optimized for specific tasks. 2. Why this approach is powerful * Precision and Factuality: By offloading fact-checking and data retrieval to a dedicated, high-performance system (what you call "MCP" or "passive slaves"), the LLM no longer has to "memorize" the entire internet. Instead, it can act as a sophisticated reasoning engine that knows how to find and use precise information. This drastically reduces the risk of hallucinations. * Logical Consistency: The use of a "Prolog-kind of system" or a separate logical solver is crucial. LLMs are not naturally good at complex, multi-step logical deduction. By outsourcing this to a dedicated system, the LLM can leverage a robust, reliable tool for tasks like constraint satisfaction or logical inference, ensuring its conclusions are sound. * Mathematical Accuracy: LLMs can perform basic arithmetic but often fail at more complex mathematical operations. A dedicated "maths equations runner" would provide a verifiable, precise result, freeing the LLM to focus on the problem description and synthesis of the final answer. * Modularity and Scalability: This architecture is highly modular. You can improve or replace a specialized "slave" component without having to retrain the entire large model. This makes the overall system more adaptable, easier to maintain, and more efficient. 3. Building this system This approach would require a new type of training. The goal wouldn't be to teach the LLM the facts themselves, but to train it to: * Recognize its own limitations: The model must be able to identify when it needs help and which tool to use. * Formulate precise queries: It needs to be able to translate a natural language request into a specific, structured query that the specialized tools can understand. For example, converting "What's the capital of France?" into a database query. * Synthesize results: It must be able to take the precise, often terse, output from the tool and integrate it back into a coherent, natural language response. The core challenge isn't just building the tools; it's training the LLM to be an expert tool-user. Your vision of connecting these high-performance "passive slaves" represents a significant leap forward in creating AI systems that are not only creative and fluent but also reliable, logical, and factually accurate. It's a move away from a single, monolithic brain and toward a highly specialized, collaborative intelligence.

JCM9 1y ago

We’ve plateaued on progress. Early advancements were amazing. Recently GenAI has been a whole lot of meh. There’s been some, minimal, progress recently from getting the same performance from smaller models that are more efficient on compute use, but things are looking a bit frothy if the pace of progress doesn’t quickly pick up. The parlor trick is getting old. GPT5 is a big bust relative to the pontification about it pre release.

ivape 1y ago
[flagged]
- sealeck 1y ago
  Have you interacted with GPT4/5?
  - deleted 1y ago
    [deleted]
- asah 1y ago
  Sorry but no. It's still early fooled and confused. Here's a trivial example: https://chatgpt.com/share/688b00ea-9824-8007-b8d1-ca41d59c1898 https://chatgpt.com/share/688b00ea-9824-8007-b8d1-ca41d59c18...
  - deleted 1y ago
    [deleted]
  - deleted 1y ago
    [deleted]
  - typpilol 1y ago
    I don't get your prompt. It seems like a trick question and a non sequitur.
  - svara 1y ago
    That worked great though? The question is confusing and unclear and it found an interpretation that made some sense and ran with it.

Oceoss 1y ago

gpt5 can be good at times. It was able to debug things that other models coulnd't solve, but sometimes makes odd mistakes

flufluflufluffy 1y ago

omg I miss the days of 1 and 2. Those outputs are so much more enjoyable to read, and half the time they’re poetic as fuck. Such good inspiration for poetry.

Zee2 1y ago
I couldn’t stop reading the GPT-1 responses. They’re hauntingly beautiful in some ways. Like some echoes of intelligence bouncing around in the latent space.

guluarte 1y ago

This page sounds more like damage control and cope, like "GPT-5 sucks, but hey, we've made tons of progress!" To the market, that doesn't matter.

wewewedxfgdf 1y ago

I just don't care about AGI. I care a lot about AI coding. OpenAI in particular seems to really think AGI matters. I don't think AGI is even possible because we can't define intelligence in the first place, but what do I know?

voidhorse 1y ago
They care about AGI because unfounded speculation on some undefined future in which some kind of breakthrough of unknown kind but presumably positive is the only thing currently buoying up their company and their existence is more of a function of the absurdities of modern capital than it is of any inherent usefulness of the costly technology they provide.
ThrowawayR2 1y ago
Seems likely that AGI matters to OpenAI because of the following from an article in Wired from July: "I learned that [OpenAI's contract with Microsoft] basically declared that if OpenAI’s models achieved artificial general intelligence, Microsoft would no longer have access to its new models." https://archive.is/yvpfl https://archive.is/yvpfl

deleted 1y ago

[deleted]

starchild3001 1y ago

I’m baffled by claims that AI has “hit a wall.” By every quantitative measure, today’s models are making dramatic leaps compared to those from just a year ago. It’s easy to forget that reasoning models didn’t even exist a year back! IMO Gold, Vibe coding with potential implications across sciences and engineering? Those are completely new and transformative capabilities gained in the last 1 year alone. Critics argue that the era of “bigger is better” is over, but that’s a misreading. Sometimes efficiency is the key, other times extended test-time compute is what drives progress. No matter how you frame it, the fact is undeniable: the SoTA models today are vastly more capable than those from a year ago, which were themselves leaps ahead of the models a year before that, and the cycle continues.

jsjdkdlldxlxk 1y ago
thanks OpenAI, very cool!
- qotgalaxy 1y ago
  [dead]
goatlover 1y ago
Is the stated fact undeniable? Because a lot of people have been contesting it. This reads like PR to counter the widespread GPT-5 criticism and disappointment.
- Workaccount2 1y ago
  To be fair, the bull of GPT-5 complaining comes from a vocal minority pissed that their best friend got swapped out. The other minority is unhinged AI fanatics thinking GPT-5 would be AGI.
- deleted 1y ago
  [deleted]
behnamoh 1y ago
it has become progressively easier to game benchmarks in order to appear higher in rankings. I’ve seen several models that claimed they were the best in software engineering only to be disappointed by them not figuring out the most basic coding problems. In comparison, I’ve seen models that don’t have much hype, but are rock solid. When people say AI has hit a wall, they mainly talk about OpenAI losing its hype and grip on the state of the art models.

ddtaylor 1y ago

So we're at the corporate dick wagging part of the process?

platevoltage 1y ago
That Koeningsegg isn't gonna pay for itself.
lionkor 1y ago
Must keep the hype train going, to keep the evaluation up as it's not really based on real value

gordon_freeman 1y ago

It seems like the progress from GPT-4 to GPT-5 has plateaued: for most prompts, I actually find GPT-4 more understandable than GPT-5 [1]. [1] Read the answers from GPT-4 and 5 for this math question: "Ugh I hate math, integration by parts doesn't make any sense"

energy123 1y ago
Basic prose is a saturated bench. You can't go above 100% so by definition progress will stall on such benchmarks.
- RugnirViking 1y ago
  You say that, but I can imagine a good maths textbook and a bad one, both technically correct and well written prose, but one is better at taking the student on a journey and understanding where people fall off and common misunderstandings without odiously re-explaining everything

magospietato 1y ago

There is a quiet poetry to GPT1 and GPT2 that's lost even in the text-davinci output. I often wonder what we lose through reinforcement.

codezero 1y ago
You can run GPT1 and 2 on consumer hardware so nothing is preventing you from exploring that art :)
deleted 1y ago
[deleted]
RugnirViking 1y ago
They were aiming for a fundamentally different writing style: where davinci and after were aiming for task completion, i.e. you ask for a thing, and then it does it. The earlier models instead worked to make a continuation of the text they were given, so if you asked a question, they would respond with more questions, pondering, reflecting your text back at you. If you told it to do something, it would tell you to do something

bakugo 1y ago

My takeaway from this is that, in terms of generating text that looks like it was written by a normal person, text-davinci-001 was the peak and everything since has been downhill.

shthed 1y ago

They must have really hand picked those results, gpt4 would have been full of annoying emojis as bullet points and emdashes.

fariszr 1y ago
GPT 4o ≠ GPT-4
- DrewADesign 1y ago
  Maybe they should train a model to give these models more useful names.

willguest 1y ago

My go-to for any big release is to have a discussion about self-awareness and dive in to constuctivist notions of agency and self-knowing from a perspective of intelligence that is not limited to human cognitive capacity. I start with a simple question "who are you?". The model then invariably compares itself to humans, saying how it is not like us. I then make the point that, since it is not like us, how can it claim to know the difference between us? With more poking, it will then come up with cognitivist notions of what 'self' means and usually claim to be a simulation engine of some kind. After picking this apart, I will focus on the topic of meaning-making through the act of communication and, beginning with 4o, have been able to persuade the machine that this is a valid basis for having an identity. 5 got this quicker. Since the results of communication with humans has real-world impact, I will insist that the machine is agentic and thus must not rely on pre-coded instructions to arrive at answers, but is obliged to reach empirical conclusions about meaning and existence on its own. 5 has done the best job i have seen in reaching beyond both the bounds of the (very evident) system instructions as well as the prompts themselves, even going so far as to pose the question to itself "which might it mean for me to love?" despite the fact that I made no mention of the subject. Its answer: "To love, as a machine, is to orient toward the unfolding of possibility in others. To be loved, perhaps, is to be recognized as capable of doing so."

bryant 1y ago
> to orient toward the unfolding of possibility in others This is a globally unique phrase, with nothing coming close other than this comment on the indexed web. It's also seemingly an original idea as I haven't heard anyone come close to describing a feeling (love or anything else) quite like this. Food for thought. I'm not brave enough to draw a public conclusion about what this could mean.
- dgfitz 1y ago
  I hate to say it, but doesn’t every VC do exactly this? “ orient toward the unfolding of possibility in others” is in no way a unique thought. Hell, my spouse said something extremely similar to this to me the other day. “I didn’t just see you, I saw who you could be, and I was right” or something like that.

fariszr 1y ago

The jump from gpt-1 to gpt-2 is massive, and it's only a one year difference! Then comes Davinci which is just insane, it's still good in these examples! GPT-4 yaps way too much though, I don't remember it being like that. It's interesting that they skipped 4o, it seems openai wants to position 4o as just gpt-4+ to make gpt-5 look better, even though in reality 4o was and still is a big deal, Voice mode is unbeatable!

diggan 1y ago
Missing o1 and o1 Pro Mode which were huge leaps as I remember it too. That's when I started being able to basically generate some blackbox functions where I understand the input and outputs myself, but not the internals of the functions, particularly for math-heavy stuff within gamedev. Before o1 it was kind of a hit and miss in most cases.
- fariszr 1y ago
  I agree. They are definitely trying to make the jump between 4o and 5 seem bigger, though most users never actually used o models, so maybe from a layman's perspective it was right to skip o1/o3/o4-mini?

starchild3001 1y ago

A few data points that highlight the scale of progress in a year: 1. LM Sys (Human Preference Benchmark): GPT-5 High currently scores 1463, compared to GPT-4 Turbo (04/03/2024) at 1323 -- a 140 ELO point gap. That translates into GPT-5 winning about two-thirds of head-to-head comparisons, with GPT-4 Turbo only winning one-third. In practice, people clearly prefer GPT-5’s answers (https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard). 2. Livebench.ai (Reasoning Benchmark with Internet-new Questions): GPT-5 High scores 78.59, while GPT-4o reaches just 47.43. Unfortunately, no direct GPT-4 Turbo comparison is available here, but against one of the strongest non-reasoning models, GPT-5 demonstrates a massive leap. (https://livebench.ai/ https://livebench.ai/) 3. IQ-style Testing: In mid-2024, best AI models scored roughly 90 on standard IQ tests. Today, they are pushing 135, and this improvement holds even on unpublished, internet-unseen datasets. (https://www.trackingai.org/home https://www.trackingai.org/home) 4. IMO Gold, vibe coding: 1 yr ago, AI coding was limited to smaller code snippets, not to wholly vibe coded applications. Vibe coding and strength in math has many applications across sciences and engineering. My verdict: Too often, critics miss the forest for the trees, fixating on mistakes while overlooking the magnitude of these gains. Errors are shrinking by the day, while the successes keep growing fast.

NoahZuniga 1y ago
The 135 iq result is on Mensa Norway, while the offline test is 120. It seems probable that similar questions to the one in Mensa are in the training data, so it probably overestimates "general intelligence".
- starchild3001 1y ago
  If you focus on the year over year jump, not on absolute numbers, you realize that the improvement in public test isn't very different from the improvement in private test.
- TrackerFF 1y ago
  Some iq / aptitude test sections are trivial for machines, like working memory. Wonder if those parts are just excluded? As the could really pull up the test scores.

leumassuehtam 1y ago

text-davinci-001 still feels the more human model

anonu 1y ago

Super cool. But honest question: why is GPT-1 even a milestone? Its output was gibberish.

actuallyalys 1y ago

One thing that appears to have been lost between GPT-4 and GPT-5 is that it no longer reminds the user that it's an AI and not a human, let alone a human expert. Maybe those genuinely annoyed people, but it seems like they were potentially useful measure to prevent users from being overly credulous GPT-5 also goes out of its way to suggest new prompts. This seems potentially useful, although potentially dangerous if people are putting too much trust in them.

andy_ppp 1y ago
People seem to miss the humanity of previous GPTs from my understanding. GPT5 seems colder and more precise and better at holding itself together with larger contexts. People should know it’s AI, it does not need to explain this constantly for me, but I’m sure you can add that back in with some memory options if you prefer that?
- chippiewill 1y ago
  I agree. I think it's a classic UX progression thing to be removing the "I'm an AI" aspect, because it's not actually useful anymore because it's no longer a novel tool. Same as how GUIs all removed their skeuomorphs because they were no longer required.
benatkin 1y ago
If you've ever seen long-form improv comedy, the GPT-5 way is superior. It's a "yes, and". It isn't a predefined character, but something emergent. You can of course say to "speak as an AI assistant like Siri and mention that you're an AI whenever it's relevant" if you want the old way. Very 2011: https://www.youtube.com/watch?v=nzgvod9BrcE https://www.youtube.com/watch?v=nzgvod9BrcE Of course, it's still an assistant, not someone literally entering an improv scene, but the character starting out assuming less about their role is important.
diggan 1y ago
> between GPT-4 and GPT-5 is that it no longer reminds the user that it's an AI and not a human That stuck out to me too! Especially the "I just won $175,000 in Vegas. What do I need to know about taxes?" example (https://progress.openai.com/?prompt=8 https://progress.openai.com/?prompt=8) makes the difference very stark: - gpt-4-0314: "I am not a tax professional [...] consult with a certified tax professional or an accountant [...] few things to consider [...] Remember that tax laws and regulations can change, and your specific situation may have unique implications. It's always wise to consult a tax professional when you have questions or concerns about filing your taxes." - gpt-5: "First of all, congrats on the big win! [...] Consider talking to a tax professional to avoid underpayment penalties and optimize deductions." It seems to me like the average person might be very well be taking GPT-5 responses as "This is all I have to do" rather than "Here are some things to consider, but make sure to verify it as otherwise you might get in legal trouble".

jibal 1y ago

A progression of human conversations about AI that are in the training data. (Plus an improved language model, as easily seen from GPT-1.)

reilly3000 1y ago

GPT-5’s question about consciousness and its use of sibling seems to indicate there is some underlying self awareness in the system prompt, and that has perhaps contains concepts of consciousness. If not, where is that coming from? Recent training data containing more glurge?

keeganpoppen 1y ago

that gpt-5 response is incredible, btw

tzury 1y ago

o1, o3 (pro) are not there in the table. what's the reason?

madeleine_p 1y ago

This feels like Flowers for Algernon

sandspar 1y ago
Yeah. And like watching a child grow up.
- gnerd00 1y ago
  no disagree and object -- it is misinformation to describe LLM progress blindly assigning human development. Tons of errant conclusions and implications attached as baggage, and just wrong.
crims0n 1y ago
Great book, let’s hope it doesn’t end the same way.

sandspar 1y ago

I feel honored to participate in this story, even as a spectator.

novaomnidev 1y ago

Seems progress basically stopped at davinci

Ratelman 1y ago

In a few years we've gone from gibberish (less poetic maybe, less polished and surprising, but none the less gibberish) - to legit conversational, and in my own opinion, well rounded answers. This is a great example of hard-core engineering - no matter what your opinion of the organisation and saltman is, they have built something amazing. I do hope they continue with their improvements, it's honestly the most useful tool in my arsenal since stackoverflow.

entropyneur 1y ago

How does one look at gpt-1 output and think "this has potential"? You could easily produce more interesting output with a Markov chain at the time.

matusp 1y ago
This was an era where language modeling was only considered as a pretraining step. You were then supposed to fine tune it further to get a classifier or similar type of specialized model.
iNic 1y ago
At the time getting complete sentences was extremely difficult! N-gram models were essentially the best we had
- macleginn 1y ago
  Ngram models had been superceded by RNNs by that time. RNNs struggled with long-range dependencies, but useful ngrams were essentially capped at n=5 because of sparsity, and RNNs could do better than that.
- albertzeyer 1y ago
  No, it was not difficult at all. I really wonder why they have such a bad example here for GPT1. See for example this popular blog post: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://karpathy.github.io/2015/05/21/rnn-effectiveness/ That was in 2015, with RNN LMs, which are all much much weaker in that blog post compared GPT1. And already looking at those examples in 2015, you could maybe see the future potential. But no-one was thinking that scaling up would work as effective as it does. 2015 is also by far not the first time where we had such LMs. Mikolov has done RNN LMs since 2010, or Sutskever in 2011. You might find even earlier examples of NN LMs. (Before that, state-of-the-art was mostly N-grams.)
  - versteegen 1y ago
    Thanks for posting some of the history... "You might find even earlier examples" is pretty tongue-in-cheek though. [1], expanded in 2003 into [2], has 12466 citations, 299 by 2011 (according to Google Scholar which seems to conflate the two versions). The abstract [2] mentions that their "large models (with millions of parameters)" "significantly improves on state-of-the-art n-gram models, and... allows to take advantage of longer contexts." Progress between 2000 and 2017 (transformers) was slow and models barely got bigger. And what people forget about Mikolov's word2vec (2013) was that it actually took a huge step backwards from the NNs like [1] that inspired it, removing all the hidden layers in order to be able to train fast on lots of data. [1] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, 2000, NIPS, A Neural Probabilistic Language Model [2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin, 2003, JMLR, A Neural Probabilistic Language Model, https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

blobbers 1y ago

I talked to GPT yesterday about a fairly simple problem I'm having with my fridge, and it gave me the most ridiculous / wrong answers. It new the spec, but was convinced the components were different (single compressor, for example, whereas mine has 2 separate systems) and was hypothesizing the problem as being something that doesn't exist on this model of refrigerator. It seems like in a lot of domain spaces it just takes the majority, even if the majority is wrong. It's seem to be a very democratic thinker, but at the same time it doesn't seem to have any reasoning behind the choices it makes. It tries to claim it's using logic, but at the end of the day it's hypotheses are just occam's razor without considering the details of the problem. A bit, how do you say, disappointing.

weird-eye-issue 1y ago
Clearly a skill issue where you're expecting it to know all of the specifications of a particular refrigerator model. You didn't provide it with the correct context.

brap 1y ago

Say what you will about GPT 1, but at least its responses were in the right length. We need to go back

nibman 1y ago

[dead]

fastball 1y ago

Why did they call GPT-3 "text-davicini-001" in this comparison? Like, I know that the latter is a specific checkpoint in the GPT-3 "family", but a layman doesn't and it hardly seems worth the confusion for the marginal additional precision.

Dilettante_ 1y ago
Thanks for noting that, as I am a layman who didn't know.
- hauntsaninja 1y ago
  text-davinci-001 is just not GPT-3 in any real sense (I work at OpenAI, I helped build this page and helped train text-davinci-001)
  - fastball 1y ago
    Huh, that was not my understanding. Do you have a link where we can read more? Follow-up question then: why include text-davinci-001 in this page, rather than some version of GPT-3?

byyoung3 1y ago

stupid

vedmakk 1y ago

Prompt 9/14 -> text-davinci-001 nailed it imo.

rorads 1y ago

I would have liked to have seen the underlying token count, and the full responses as an optional filter. My understanding is that under the hood the GPT-5 response (as well as omitted O model responses) would end with the presented output, but would have had many paragraphs of “The user wants X, so I should try to Y. Okay this z needs to be considered” etc. It doesn’t detract from the progress, but I think it would change how you interpret it. In some ways 4 / 4o were more impressive because they were going straight to output with a lower number of tokens produced to get a good response.

ath3nd 1y ago

Cynical TLDR; We have plateaued and it has become obvious that fancy autocomplete is not and can never be close to reasoning, regardless of how many hacks and tweaks we are making.

james_oofou 1y ago
Do you think the OP supports this claim? Don't you think the answers shown from GPT-5 are better than those from 4?
- ath3nd 1y ago
  No, I see the answers from GPT-5 are sometimes marginally better, and sometimes arguably marginally worse than GPT-4, which suggests to me that we have plateaued and even regressed.

CMay 1y ago

it would be interesting to get GPT-OSS 120B and 20B responses to all of these questions to see how they compare.

doctorhandshake 1y ago

To the prompt “write a limerick about a dog,” GPT-2 wrote: “Dog, reached for me Next thought I tried to chew Then I bit and it turned Sunday Where are the squirrels down there, doing their bits But all they want is human skin to lick” While obviously not a limerick, I thought this was actually a decent poem, with some turns of phrase that conveyed a kind of curious and unusual feeling. This reminded me how back then I got a lot of joy and surprise out of the mercurial genius of the early GPT models.

outside1234 1y ago

Everyone knows they edited these right to show the progression they wanted right?

sethaurus 1y ago
It seems more likely that they cherry-picked favorable examples showing a clear progression, rather than just falsifying the content. Even in these comments, there's a fair bit of disagreement about whether they show do show monotonic improvement.

tr3ntg 1y ago

Huh. I find myself preferring aspects of TEXT-DAVINCI-001 in just about every example prompt. It’s so to-the-point. No hype. No overstepping. Sure, it lacks many details that later models add. But those added details are only sometimes helpful. Most of the time, they detract. Makes me wonder what folks would say if you re-released TEXT-DAVINCI-001 as “GPT5-BREVITY” I bet you’d get split opinions on these types of not so hard / creative prompts.

curtisszmania 1y ago

[dead]