8 ms·

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an

• phillipcarter 2y ago
  As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks. But, still, this is incredibly impressive.
  • qt31415926 2y ago
    Which parts of reasoning do you think is missing? I do feel like it covers a lot of 'reasoning' ground despite its on the surface simplicity
    • mistermann 2y ago
      Optimal phenomenological reasoning is going to be a tough nut to crack. Luckily we don't know the problem exists, so in a cultural/phenomenological sense it is already cracked.
    • phillipcarter 2y ago
      I think it's hard to enumerate the unknown, but I'd personally love to see how models like this perform on things like word problems where you introduce red herrings. Right now, LLMs at large tend to struggle mightily to understand when some of the given information is not only irrelevant, but may explicitly serve to distract from the real problem.
      • KaoruAoiShiho 2y ago
        o1 already fixed the red herrings...
      • zmgsabst 2y ago
        That’s not inability to reason though, that’s having a social context. Humans also don’t tend to operate in a rigorously logical mode and understand that math word problems are an exception where the language may be adversarial: they’re trained for that special context in school. If you tell the LLM that social context, eg that language may be deceptive, their “mistakes” disappear. What you’re actually measuring is the LLM defaults to assuming you misspoke trying to include relevant information rather than that you were trying to trick it — which is the social context you’d expect when trained on general chat interactions. Establishing context in psychology is hard.
    • Agentus 2y ago
      kinda interesting, every single CS person (especially phds) when talking about reasoning are unable to concisely quantify, enumerate, qualify, or define reasoning. people with (high) intelligence talking and building (artificial) intelligence but never able to convincingly explain aspects of intelligence. just often talk ambiguously and circularly around it. what are we humans getting ourselves into inventing skynet :wink. its been an ongoing pet project to tackle reasoning, but i cant answer your question with regards to llms.
      • YeGoblynQueenne 2y ago
        >> Kinda interesting, every single CS person (especially phds) when talking about reasoning are unable to concisely quantify, enumerate, qualify, or define reasoning. Kinda interesting that mathematicians also can't do the same for mathematics. And yet.
        • Agentus 2y ago
          well lets just say i think i can explain reasoning better than anyone ive encountered. i have my own hypothesized theory on what it is and how it manifests in neural networks. i doubt your mathmatician example is equivalent. examples that are fresh on the mind that further my point. ive heard yann lecun baffled by llms instantiation/emergence of reasoning, along with other ai researchers. eric Schmidt thinks the agentic reasoning is the current frontier and people should be focusing on that. was listening to the start of an ai machine learning interview a week ago with some cs phd asked to explain reasoning and the best he could muster up is you know it when you see it…. not to mention the guy responding to the grandparent that gave a cop out answer ( all the most respect to him).
          • necovek 2y ago
            Care to enlighten us with your explanation of what "reasoning" is?
            • Agentus 2y ago
    • john_minsk 2y ago
      My personal 5 cents is that reasoning will be there when LLM gives you some kind of outcome and then when questioned about it can explain every bit of result it produced. For example, if we asked an LLM to produce an image of a "human woman photorealistic" it produces result. After that you should be able to ask it "tell me about its background" and it should be able to explain "Since user didn't specify background in the query I randomly decided to draw her standing in front of a fantasy background of Amsterdam iconic houses. Usually Amsterdam houses are 3 stories tall, attached to each other and 10 meters wide. Amsterdam houses usually have cranes on the top floor, which help to bring goods to the top floor since doors are too narrow for any object wider than 1m. The woman stands in front of the houses approximately 25 meters in front of them. She is 1,59m tall, which gives us correct perspective. It is 11:16am of August 22nd which I used to calculate correct position of the sun and align all shadows according to projected lighting conditions. The color of her skin is set at RGB:xxxxxx randomly" etc. And it is not too much to ask LLMs for it. LLMs have access to all the information above as they read all the internet. So there is definitely a description of Amsterdam architecture, what a human body looks like or how to correctly estimate time of day based on shadows (and vise versa). The only thing missing is logic that connects all this information and which is applied correctly to generate final image. I like to think about LLMs as a fancy genius compressing engines. They took all the information in the internet, compressed it and are able to cleverly query this information for end user. It is a tremendously valuable thing, but if intelligence emerges out of it - not sure. Digital information doesn't necessarily contain everything needed to understand how it was generated and why.
      • concordDance 2y ago
        > if we asked an LLM to produce an image of a "human woman photorealistic" it produces result Large language models don't do that. You'd want an image model. Or did you mean "multi-model AI system" rather than "LLM"?
        • owenpalmer 2y ago
          It might be possible for a language model to paint a photorealistic picture though.
    • Xmd5a 2y ago
      LLMs are still bound to a prompting session. They can't form long term memories, can't ponder on it and can't develop experience. They have no cognitive architecture. 'Agents' (i.e. workflows intermingling code and calls to LLMs) are still a thing (as shown by the fact there is a post by anthropic on this subject on the front page right now) and they are very hard to build. Consequence of that for instance: it's not possible to have a LLM explore exhaustively a topic.
      • mjhagen 2y ago
        LLMs don’t, but who said AGI should come from LLMs alone. When I ask ChatGPT about something “we” worked on months ago, it “remembers” and can continue on the conversation with that history in mind. I’d say, humans are also bound to promoting sessions in that way.
        • Xmd5a 2y ago
          Last time I used ChatGPT 'memory' feature it got full very quickly. It remembered my name, my dog's name and a couple tobacco casing recipes he came up with. OpenAI doesn't seem to be using embeddings and a vector database, just text snippets it injects in every conversation. Because RAG is too brittle ? The same problem arises when composing LLM calls. Efficient and robust workflows are those whose prompts and/or DAG were obtained via optimization techniques. Hence DSPy. Consider the following use case: keeping a swimming pool water clean. I can have a long running conversation with a LLM to guide me in getting it right. However I can't have a LLM handle the problem autonomously. I'd like to have it notify me on its own "hey, it's been 2 days, any improvement? Do you mind sharing a few pictures of the pool as well as the ph/chlorine test results ?". Nothing mind-boggingly complex. Nothing that couldn't be achieved using current LLMs. But still something I'd have to implement myself and which turns out to be more complex to achieve than expected. This is the kind of improvement I'd like to see big AI companies going after rather than research-grade ultra smart AIs.
    • deleted 2y ago
      [deleted]
    • amelius 2y ago
      Does it include the use of tools to accomplish a task? Does it include the invention of tools?
    • tim333 2y ago
      Current AI is good at text but not very good at 3d physical stuff like fixing your plumbing.
  • azeirah 2y ago
    I'd like to see this o3 thing play 5d chess with multiverse time travel or baba is you. The only effect smarter models will have is that intelligent people will have to use less of their brain to do their work. As has always been the case, the medium is the message, and climate change is one of the most difficult and worst problems of our time. If this gets software people to quit en-masse and start working in energy, biology, ecology and preservation? Then it has succeeded.
    • concordDance 2y ago
      > climate change is one of the most difficult and worst problems of our time. Slightly surprised to see this view here. I can think of half a dozen more serious problems off hand (e.g. population aging, institutional scar tissue, dysgenics, nuclear proliferation, pandemic risks, AI itself) along most axes I can think of (raw $ cost, QALYs, even X-risk).
      • TranquilMarmot 2y ago
        None of those problems really matter if we don't have a planet to live on
        • concordDance 2y ago
          You've been greviously mislead if you think climate change could plausibly make the world uninhabbitable in the next couple of centuries given current trajectories. I advise going to the primary sources and emailing a climate scientist at your local university for some references.
          • TranquilMarmot 2y ago
            > going to the primary sources and emailing a climate scientist at your local university for some references I assume you've done this, otherwise you wouldn't be telling me to? Bold of you to assume my ignorance on this subject. You sound like you've fallen for corporate grifters who care more about short-term profit and gains over long-term sustainability (or you are one of said grifters, in which case why are you wasting your time on HN, shouldn't you be out there grinding?!) Severe weather events are going to get more common and more devastating over the next couple of decades. They'll come for you and people you care about, just as they come for me and people I care about. It doesn't matter what you think you know about it.
• ALittleLight 2y ago
  It's not saturated. 85% is average human performance, not "best human" performance. There is still room for the model to go up to 100% on this eval.
• scotty79 2y ago
  Still it's comparing average human level performance with best AI performance. Examples of things o3 failed at are insanely easy for humans.
  • FrustratedMonky 2y ago
    There are things Chimps do easily that humans fail at, and vice/versa of course. There are blind spots, doesn't take away from 'general'.
    • deleted 2y ago
      [deleted]
    • noobermin 2y ago
      The downvotes should tell you, this is a decided "hype" result. Don't poo poo it, that's not allowed on AI slop posts on HN.
      • FrustratedMonky 2y ago
        Yeah, I didn't realize Chimp studies, or neuroscience were out of vogue. Even in tech, people form strong 'beliefs' around what they think is happening.
    • Matumio 2y ago
      We can't agree whether Portia spiders are intelligent or just have very advanced instincts. How will we ever agree about what human intelligence is, or how to separate it from cultural knowledge? If that even makes sense.
      • FrustratedMonky 2y ago
        I guess my point is more, if we can't decide about Portia Spiders or Chimps, then how can we be so certain about AI. So offering up Portia and Chimps as counter examples.
  • 2y ago
• cryptoegorophy 2y ago
  What’s interesting is it might be very close to human intelligence than some “alien” intelligence, because after all it is a LLM and trained on human made text, which kind of represents human intelligence.
  • hammock 2y ago
    In that vein, perhaps the delta between o3 @ 87.5% and Human @ 85% represents a deficit in the ability of text to communicate human reasoning. In other words, it's possible humans can reason better than o3, but cannot articulate that reasoning as well through text - only in our heads, or through some alternative medium.
    • 85392_school 2y ago
      I wonder how much of an effect amount of time to answer has on human performance.
      • yunwal 2y ago
        Yeah, this is sort of meaningless without some idea of cost or consequences of a wrong answer. One of the nice things about working with a competent human is being able to tell them "all of our jobs are on the line" and knowing with certainty that they'll come to a good answer.
    • unsupp0rted 2y ago
      It's possible humans reason better through text than not through text, so these models, having been trained on text, should be able to out-reason any person who's not currently sitting down to write.
  • hamburga 2y ago
    Agreed. I think what really makes them alien is everything else about them besides intelligence. Namely, no emotional/physiological grounding in empathy, shame, pride, and love (on the positive side) or hatred (negative side).
• antirez 2y ago
  NNs are not algorithms.
  • notfish 2y ago
    An algorithm is “a process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer” How does a giant pile of linear algebra not meet that definition?
    • antirez 2y ago
      It's not made of "steps", it's an almost continuous function to its inputs. And a function is not an algorithm: it is not an object made of conditions, jumps, terminations, ... Obviously it has computation capabilities and is Turing-complete, but is the opposite of an algorithm.
      • raegis 2y ago
        > It's not made of "steps", it's an almost continuous function to its inputs. Can you define "almost continuous function"? Or explain what you mean by this, and how it is used in the A.I. stuff?
        • taneq 2y ago
          Well, it's a bunch of steps, but they're smaller. /s
      • janalsncm 2y ago
        If it wasn’t made of steps then Turing machines wouldn’t be able to execute them. Further, this is probably running an algorithm on top of an NN. Some kind of tree search. I get what you’re saying though. You’re trying to draw a distinction between statistical methods and symbolic methods. Someday we will have an algorithm which uses statistical methods that can match human performance on most cognitive tasks, and it won’t look or act like a brain. In some sense that’s disappointing. We can build supersonic jets without fully understanding how birds fly.
• 6gvONxR4sf7o 2y ago
  Human performance is much closer to 100% on this, depending on your human. It's easy to miss the dot in the corner of the headline graph in TFA that says "STEM grad."
  • tim333 2y ago
    A fair comparison might be average human. The average human isn't a STEM grad. It seems STEM grad approximately equals an IQ of 130. https://www.accommodationforstudents.com/student-blog/the-subjects-with-the-highest-iqs https://www.accommodationforstudents.com/student-blog/the-su... From a post elsewhere the scores on ARC-AGI-PUB are approx average human 64%, o3 87%. https://news.ycombinator.com/item?id=42474659 https://news.ycombinator.com/item?id=42474659 Though also elsewhere, o3 seems very expensive to operate. You could probably hire a PhD researcher for cheaper.
    • jeremyjh 2y ago
      Why would an average human be more fair than a trained human? The model is trained.
• hypoxia 2y ago
  It actually beats the human average by a wide margin: - 64.2% for humans vs. 82.8%+ for o3. ... Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3 (limited compute) [2] Public Eval: - 91.5%: o3 (unlimited compute) [2] - 82.8%: o3 (limited compute) [2] - 64.2%: human average (Mechanical Turk) [1] [3] Public Training: - 76.2%: human average (Mechanical Turk) [1] [3] ... References: [1] https://arcprize.org/guide https://arcprize.org/guide [2] https://arcprize.org/blog/oai-o3-pub-breakthrough https://arcprize.org/blog/oai-o3-pub-breakthrough [3] https://arxiv.org/abs/2409.01374 https://arxiv.org/abs/2409.01374
  • usaar333 2y ago
    Super human isn't beating rando mech turk. Their post has stem grad at nearly 100%
    • tripletao 2y ago
      This is correct. It's easy to get arbitrarily bad results on Mechanical Turk, since without any quality control people will just click as fast as they can to get paid (or bot it and get paid even faster). So in practice, there's always some kind of quality control. Stricter quality control will improve your results, and the right amount of quality control is subjective. This makes any assessment of human quality meaningless without explanation of how those humans were selected and incentivized. Chollet is careful to provide that, but many posters here are not. In any case, the ensemble of task-specific, low-compute Kaggle solutions is reportedly also super-Turk, at 81%. I don't think anyone would call that AGI, since it's not general; but if the "(tuned)" in the figure means o3 was tuned specifically for these tasks, that's not obviously general either.
• dyauspitr 2y ago
  I’ll believe it when the AI can earn money on its own. I obviously don’t mean someone paying a subscription to use the AI I mean, letting the AI lose on the Internet with only the goal of making money and putting it into a bank account.
  • hamburga 2y ago
    Do trading bots count?
    • 1659447091 2y ago
      No, the AI would have to start from zero and reason it's way to making itself money online, such as the humans who were first in their online field of interest (e-commerce, scams, ads etc from the 80's and 90's) when there was no guidance, only general human intelligence that could reason their way into money making opportunities and reason their way into making it work.
      • concordDance 2y ago
        I don't think humans ever do that. They research/read and ask other humans.
        • 1659447091 2y ago
          Which AI already has stored in spades, even more so since people in the 80's 90's weren't working with the information available today. The AI is free to research and read all the information stored from other humans as well, just like the humans who reasoned their way into money making opportunities--only with vastly more information now, talk about an advantage. But is it intelligent enough do so without a human giving direct/step-by-step instructions; the way humans figure it out?
  • creer 2y ago
    You don't think there are already plenty of attempts out there? When someone is "disinterested enough" to publish though, note the obvious way to launch a new fund or advisor with a good track record: crank out a pile of them, run them one or two years, discard the many losers and publish the one or two top winners. I.E. first you should be suspicious of why it's being published, then of how selected that result is.
• lastdong 2y ago
  Curious about how many tests were performed. Did it consistently manage to successfully solve many of these types of problems?
• dmead 2y ago
  This is so strange. people think that an llm trained on programming questions and docs can do mundane tasks like this means intelligent? Come on. It really calls into question two things. 1. You don't know what you're talking about about. 2. You have a perverse incentive to believe this such that you will preach it to others and elevate some job salary range or stock. Either way, not a good look.
  • javaunsafe2019 2y ago
    This