15 ms·
First thoughts on o3 pro
- treetalker 1y ago> We’re in the era of task-specific models. On one hand, we have “normal” models like 3.5 Sonnet and 4o—the ones we talk to like friends, who help us with our writing … … > [M]odels today are so good … … > o3 pro (left) clearly understanding the confines of it’s environment way better. Miracle models that are so good at helping us with our writing, yet we still use it's as a possessive form.
- landl0rd 1y ago[flagged]
- DylanSp 1y agoNot gender-related; the correct form for the possessive is "its" without an apostrophe, not "it's".
- jurgenaut23 1y agolol, what an amazing meta exchange right there
- Groxx 1y agoI'm on the fence with its/it's... but mostly I think I'd be fine with "its" disappearing. It's almost never ambiguous in writing or speaking given context, and it's a weird possessive-form inconsistency.
- david-gpu 1y agoWhich one is the most inconsistent? He -> His It -> Its She -> Her He -> He is -> He's It -> It is -> It's She -> She is -> She's The only one breaking the pattern is Her, which instead should be Shes if we wanted to keep things consistent.
- coffeecantcode 1y agoActually if we’re keeping with the pattern it would be Shis
- spiderfarmer 1y agoOr shits
- myk9001 1y agoDoes this happen to be the one billionth HN comment?
- david-gpu 1y agoGood catch! Thank you.
- tsimionescu 1y agoAll of the pronoun contractions are somewhat inconsistent with noun possessives. So "its", but "the dog's". There's no reason why you couldn't write "it's" for both "it is" and "its". You already write both "the dog's ball" and "the dog's tired", and both "it's" and "it's" are pronounced identically, which doesn't pose any difficulties in speech. His, her, and my are a different matter, since they are all pronounced differently from he's*, she's, and I'm. * at least in most accents, "his" uses the kit vowel, while "he's" uses the fleece vowel
- wetpaws 1y agoDog's/dogs are ambiguous. It's/its is not.
- tsimionescu 1y ago"Dog's" is ambiguous in itself (dog is / that belongs to the dog), but this doesn't cause problems in practice. It's exactly the same ambiguity as spelling "it's" for the possessive would give. Also, it's / its is only unambiguous in writing. In speech, they are identical, in every accent of English - and yet people understand each other perfectly fine in spoken English, so the ambiguity is not a problem in practice.
- Groxx 1y agoIt's honestly pretty funny to me that apparently this is one of the worst things I've ever said here. I mean. I don't disagree. But lol
- layer8 1y agoMaybe use of spell-checkers will finally catch on when they are AI-based. ;)
- simonw 1y agoSomething I like about this piece is how much in reinforces the idea that models like o3 Pro are really hard to get good results out of. I don't have an intuition at all for when I would turn to o3 Pro yet. What kind of problems do I have where outsourcing to a huge model that crunches for several minutes are worthwhile? I'm enjoying regular o3 a lot right now, especially with the huge price drop from the other day. o3 Pro is a lot harder to get my head around.
- JSR_FDED 1y agoSame here, I’d be very interested to learn how others decide what model to use for which tasks. I find these minutes-long iterations so painful that in practice I always go for the fast non-reasoning models.
- SOLAR_FIELDS 1y agoImagine a tricky distributed systems problem where you feed all of the context of your setup to the LLM and it uses the advanced reasoning to diagnose possible avenues. I did that recently with a frontier model to unwrap some very tricky istio related connection pooling issues causing syn/ack floods. For coding I usually use the fast frontier model like o4minihigh, but I bust out the fancy research models when I want things like general architecture and design feedbacks that require broader advanced reasoning
- CamperBob2 1y agoI don't often have LLMs write a lot of code for me, but when I do, I don't mind waiting a couple more minutes for a result that will waste less of my time in debugging when I try to use it. Also it's useful to have models review code that I wrote -- in some cases years ago -- to uncover old bugs. Current models are generally far too eager to say "Yup! Looks good! You da man!" when there are actually serious flaws in the code they are reviewing. So again, this is a task that justifies use of the most powerful models currently available, and that doesn't have to run in real time.
- dist-epoch 1y agoRandom thought: dump your knowledge base into it (Obsidian, ...) and ask it to reorganize it, delete duplication, obsolete stuff, optimize it. Or tell it what you know about non-programming subject X, and ask it to explain it to you such that you understand it better. Or for coding: ask it to look at your code, and suggest large scale architecture changes. For these kinds of tasks, the models are still lacking.
- gleb 1y agoo3 pro seems to be good with meta-prompting. Meaning, when you ask it to create a prompt for you. In particular it seems to be more concise than o3 when doing this. Has anybody else noticed this?
- vendiddy 1y agoDo you have an example? I'm trying to understand what you mean by meta promoting
- selfhoster11 1y agoLiterally just asking it to write a prompt to ask itself (or maybe another LLM) to do something. Example: You want to improve a piece of code that interprets bytecode and translates it into native 8051 assembly. You have the code, but you don't really have any ideas on how to improve it. You want the LLM to come up with ideas. Without meta prompting, your prompt would be like this: "Here's a piece of code that interprets bytecode and translates it into native 8051 assembly. Think of ways to improve it." With meta prompting, your initial prompt would be like this: "Here's a piece of code that interprets bytecode and translates it into native 8051 assembly. My goal is to improve it. Don't think about the improvements, though. Instead, write a large language model prompt that will guide the LLM on ways of thinking about possible improvements. Incorporate as many methods as you can think of." The o3 would then come back with a prompt like this: ---------- Here's a piece of code that interprets bytecode and translates it into native 8051 assembly. I'd like you to think of some ways to improve it. Here are some guidelines about what you should and shouldn't do: - Describe different models of optimization (e.g., peephole, mid-level IR transformation, architecture-aware instruction packing) that might apply to this code. - Provide brief explanations behind each improvement idea, optionally citing known compiler or assembler optimization strategies. - Use critical code analysis strategies such as abstraction refactoring, loop unrolling evaluation, and inline expansion assessment. - Suggest external tools, static analysis techniques, or benchmarks that would support an empirical review of the code’s performance. - Draft a structured list of investigative questions I should ask to uncover hidden inefficiencies or architecture mismatches. - Consider ways to minimize register swaps and optimize instruction selection for the 8051 architecture. - Try breaking your ideas into perspectives—a performance-focused one, a maintenance-focused one, and a correctness-focused one. - Emphasize reasoning paths: “Consider factors like X, contrast options A and B…” - Do not directly suggest code changes—instead, construct a high-quality LLM prompt I can run that will guide another model into doing this with rich and varied insights. - Step into the shoes of different roles: a compiler designer, a security auditor, and a systems programmer. Think from each of their perspectives. - Don’t just list ideas—group them by impact level and feasibility, and reflect on trade-offs. - Don’t rewrite the entire architecture. Focus on micro-optimizations and structural improvements that retain the current functional design. - Evaluate whether parts of the bytecode can be cached or memoized to avoid reprocessing. - Analyze for common performance bottlenecks typical in interpreters, such as inefficient branching or redundant memory access. - Avoid superficial ideas like "add comments" or "modernize syntax" unless critically justified. ---------- As you can see, the meta-prompt is essentially an instruction to "compile" a shorter prompt into something much larger and more detailed - which may be wrong, but if the meta-prompt basically encodes well-known methodology (like above), it's hard for it to go wrong. (Yes, I did use an LLM to generate the list above - I checked each bullet point by hand just to make sure they weren't nonsensical.)
- bananapub 1y ago> On the other, we have gigantic, slow, expensive, IQ-maxxing reasoning models that we go to for deep analysis (they’re great at criticism), one-shotting complex problems, and pushing the edge of pure intelligence. I quite enjoy having an LLM write much of my tedious code these days, but comments like this are just bizarre to me. Can someone share a text question that I can ask an expensive slow LLM that will demonstrate “deep analysis” or “iq-maxxing” on any topic? Whenever I ask them factual or discussion questions I usually get something riddled with factual errors or just tedious, like reading an essay someone wrote for school.
- semi-extrinsic 1y agoI don't have any good idea of what are "good" prompts for demonstrating such models. But what I would ask such a model is the following. I have no idea if it would fall on it's face or not. Can you write a version of Chorin's projection method for the Navier-Stokes equations that is both explicit and second order in time? Ideally the model should not need a more detailed prompt than this. A first-year grad student in numerical analysis certainly would not.
- varjag 1y agoThe one I asked o3-pro yesterday was "Research the annual smoking tobacco production in Soviet Union 1939-1958 and plot it in a graph versus male population size"
- riku_iki 1y agoAnd how was result? Did you verify that it found reliable source of data?
- nerdsniper 1y agoThis is the kind of thing I absolutely don’t trust it for. It generates a very convincing-sounding report but for a lot of tasks I’ve found the numbers won’t reasonably match up to my own.
- serjester 1y agoI found o3 pro to need a paradigm shift, where the latency makes it impossible to use in anything but in async manner. You have a broad question, likely somewhat vague, and you pass it off to o3 with a ton of context. Then maybe 20 minutes later, you're going to have a decently good answer. Definitely stronger than any other models - it genuinely has taste. Yet, the scary thing here is that increasingly I'm starting to feel like the bottleneck. A human can only think about so many tasks in parallel and it seems like my contributions are getting less and less important with every model upgrade. Every now and then I question why I'm paying $200 for the max plan, but then something like this comes out and makes it a no brainer.
- dist-epoch 1y agoDo the coherent extrapolated volition thing: "o3-pro, please reinterpret the prompt and do what I would have actually requested instead if I was 50 IQ points smarter."
- atleastoptimal 1y agoThis is gonna keep happening with every AI advance until humans are an absolute bottleneck in every domain. May take a bit of time for some professions, but the writing is on the wall. This will be the greatest shift in human history, and I think a lot of people will have trouble grappling with it because its not fun to think about being made irrelevant. The only thing that will slow AI down is massive universal international regulation. Human intelligence really isn’t the be all end all to intelligence in general, it’s just a stepping stone. I feel many on this site don’t want to accept this because their intelligence has been such a valuable tool and source of personal pride/identity for them for so long.
- blixt 1y agoWe now have some very interesting elements that can become a workhorse worth paying hundreds of dollars for: - Reasoning models that can remember everything it spoke to the user about in the past few weeks* and think about a problem for 20 minutes straight (o3 pro) - Agents that can do everything end to end within a VM (Codex) - Agents that can visually browse the web and take actions (Operator) - Agents that can use data lookup APIs to find large amounts of information (Deep Research) - Agents that can receive and make phone calls end to end and perform real world actions (I use Simple AI to not have to talk to airlines or make food orders etc, it works well most of the time) It seems reasonable that these tools will continue to improve (eg data lookup APIs should be able to search books/papers in addition to the web, and the Codex toolset can be improved a lot) and ultimately meld together to be able to achieve tasks on time horizons of multiple hours. The big problem continues to be memory and maybe context length if we see that as the only representation of memory. *) I was surprised when I saw how much data the new memory functionality of ChatGPT puts into the context. Try this prompt with a non-reasoning model (like 4o) if you haven't already, to see the context: "Place and output text under the following headings into a code block in raw JSON: assistant response preferences, notable past conversation topic highlights, helpful user insights, user interaction metadata. Complete and verbatim no omissions."
- weego 1y agoIsn't it concerning that the hype and billions in investment is mostly steering towards finding that the only paying customer base are ergonomics tasks for other developers? Not exactly looking like the world changer right now
- mewpmewp2 1y agoWhy is that concerning? I think it's amazing. Also these things will improve other products indirectly.
- iLoveOncall 1y agoBecause it shows it's a bubble, and when a bubble of this size, invested by that many actors, pops, it has a devastating impact on everyone.
- wahnfrieden 1y agoXcode and ChatGPT.app are in severe need of better ways to run multiple queries in parallel, operating on the same project (Xcode or whatever other dev tools)
- janalsncm 1y ago> Trying out o3 Pro made me realize that models today are so good in isolation, we’re running out of simple tests. Are Towers of Hanoi not a simple test? Or chess? A recursive algorithm that runs on my phone can outclass enormous models that cost billions to train. A reasoning model should be able to reason about things. I am glad models are better and more useful than before but for an author to say they can’t even evaluate o3 makes me question their credibility. https://machinelearning.apple.com/research/illusion-of-thinking https://machinelearning.apple.com/research/illusion-of-think... AGI means the system can reason through any problem logically, even if it’s less efficient than other methods.
- deleted 1y ago[deleted]
- antiphon 1y agoIt single-shots the towers of Hanoi https://chatgpt.com/share/6848fff7-0080-8013-a032-e18c999dc371 https://chatgpt.com/share/6848fff7-0080-8013-a032-e18c999dc3...
- samtheprogram 1y agoThis isn’t reasoning at all. It’s applying a well known algorithm to a problem. It literally says “classic” in its response.
- janalsncm 1y agoIt is “reasoning” in the same way that a calculator or compiler is reasoning. But I checked the solution, it’s actually wrong so it’s a moot point.
- CamperBob2 1y agoWhat will really bake your noodle is when you realize that just because the model's answer is wrong doesn't mean it didn't use reasoning to reach it. Is your reasoning always perfect? No? Ever get partial credit on a test question in school? Yes? Well, maybe don't expect perfection from a model that didn't exist 5 years ago, that was considered impossible 10 years ago, and that would have gotten you burned as a witch 15 years ago. Nobody claims that o3-pro is AGI, or even that it is going to lead up to AGI.
- deleted 1y ago[deleted]
- MagicMoonlight 1y ago>The plan o3 gave us was plausible, reasonable; but the plan o3 Pro gave us was specific and rooted enough that it actually changed how we are thinking about our future. >This is hard to capture in an eval. ChatGPT wrote this article
- guffins 1y agoPart of me hates how certain ways of writing have been designated as hints that something was written by AI… I love using em dashes, for example. But yeah, ChatGPT models tend to overuse weird abbreviations of common words in a way that humans never do. I’d say “in an eval” when talking on Slack with very close coworkers, but _never_ on a public post. Still, I don’t want to make accusations based on writing style, because like I said, I have my own idiosyncrasies that common models coincidentally share.
- guffins 1y agoCorrection: I wasn’t aware of OpenAI’s “evals” tool. If that’s the kind of thing the author was referring to, I may have misjudged.
- zone411 1y agoI benchmarked it on four of my benchmarks so far. Got first place in two of them: https://github.com/lechmazur/confabulations https://github.com/lechmazur/confabulations https://github.com/lechmazur/nyt-connections https://github.com/lechmazur/nyt-connections https://github.com/lechmazur/generalization https://github.com/lechmazur/generalization https://github.com/lechmazur/writing https://github.com/lechmazur/writing
- kenjackson 1y agoIt seems like you often have LLMs grading each other. Aren’t you concerned that some models may not be “smart” enough to grade a smarter model appropriately?
- ACCount36 1y agoUsing LLMs for evaluating LLMs is incredibly common. The point isn't in having a "perfect" evaluator, but in having a cheap and somewhat consistent evaluator. This approach holds up well enough... as long as you don't try to use it for RL. If you do, chances are, you'll end up with an adversarial LLM that aims solely for breaking and saturating the evaluator.
- agos 1y agoI’m not sure I would use “consistent” to characterize LLMs
- kenjackson 1y agoBut I feel like the evaluator should generally be stronger/better than what its evaluating. Otherwise you risk it evaluating at a lower level, while the better LLM is writing with more nuance that the lower LLM doesn't pick up on. I've seen some places, e.g., NY Times, use expert panels to review the results from LLMs. For example, getting the author of a book/essay to evaluate how well the LLM summarizes and answers questions about the book/essay. While it's not scalable, it does seem like it will better evaluate cutting edge models.
- nxobject 1y agoRe context and overthinking: > One thing I noticed from early access: if you don’t give it enough context, it does tend to overthink. I agree with this – that being said, I find that simply asking at the end of a prompt "Do you need any clarifications before you continue?" does a pretty good job at helping AI pin down details as well.
- joshvm 1y agoThe first thing I asked was along the lines of "Find me an implementation of X that does not depend on framework Y, because it's not maintained". o3 thought for a minute and then returns a repository that depends on Y. In its defense, 4.1 also found the same repo and commented that it "has a light dependency", which was cute, if unhelpful. Adding a "think about whether your answer actually depends on Y" fixed it, and the response was pretty well thought out. My only gripe is the current persona seems to be tech-bro-informal. It opened with "TL;DR" and included a lot of developer slang that reads like it's trying to be cool in a way that masks overconfidence.
- tonyhart7 1y agoseems like the AI model is plateau isn't??? its just undercutting Gemini by a close margin in terms of capabilities
- bobjordan 1y agoI got frustrated with the new o3-pro mode today. I just wasted a few hours of my day waiting 15-20 minutes for answers that were totally out of line with the workflow I've had since the first o1-pro model came out. It's a completely different beast to work with. It feels like it hits output limits way easier, and you have to work around it. Today after I finally gave up, I just told the model I was disappointed and asked it to explain its limitations. It was actually helpful, and told me I could ask for a download link to get a file that wasn't cut off. But why should I have to do that? It's definitely not more user-friendly and totally the opposite experience as working with Google Gemini 2.5 pro. Honestly, this experience made it obvious how much harder OpenAI's models are to work with now compared to Google's. I've been using Gemini 2.5 Pro and it's super hard to find its limits. For the $20 I spend, it's not even a competition anymore. My new workflow is clear: throw everything at Gemini 2.5 Pro to get the real work done, then maybe spot-check it with the OpenAI models. I'll probably just migrate to the top Gemini ultra tier when the “deep thinking” mode is available. I'm just not happy with the openai experience on any of their models after getting used to the huge context window in Gemini. OpenAI used to at least keep me happy with o1-pro but now that they removed it and o3-pro kind of sucks to work with taking 20 minutes to output and have lower confidence in the time spent, I don’t think I have a reason to default to them anymore. Gemini is definitely more user friendly and my default option now.
- megaloblasto 1y agoMococoa drink, all natural cocoa beans from the upper slopes of Mount Nicaragua. No artificial sweeteners
- kenjackson 1y agoWhat seems clear is there is no consensus. Gemini 2.5 Pro just seems consistently worse to me, but I’ve seen others sing its praises. This might be more like iPhone vs Android than a true stack ranking of models.
- mcny 1y agoGemini 2.5 pro has been consistently excellent for me, when it works. It sometimes just spins and spins with no results but when it comes with something, it has been pretty good.
- b0a04gl 1y agoi gave it a 4 step research task with branching subtasks. told it upfront what the goal was. halfway through it forgot why it was doing step 2. asked it to summarise progress so far and it hallucinated a step i never mentioned. restarted from scratch with memory enabled. same thing. no state carryover. no grounding. if you don’t constantly babysit the thread and refeed everything, it breaks. persistent memory is surface-level. no real continuity. just isolated task runner. autonomy without continuity is not autonomy
- replwoacause 1y agoSounds pretty useless
- b0a04gl 1y ago[dead]
- Omarbev 1y agoThe directing is great.
- rthnbgrredf 1y agoI think o3-pro is just o3-very-high. And for my taste it is a bit too high.
- jdthedisciple 1y agoWe learn that good reasoning models lack social skills. So kinda like autists (in a good way).
- ralfd 1y ago>I wrote up all my thoughts, got ratio’ed by @sama I have no idea what this verb means.
- Fuzzwah 1y agoIf only there was a way to learn the meaning of words.... Let me chatgpt that for you: When someone “gets ratioed,” it means their post has more negative feedback than positive engagement. The number of comments or replies outweighs the number of likes or retweets, indicating the content is controversial. When someone says they "got ratioed by <someone>", it usually means that the reply or response from that person received significantly more engagement (likes, retweets, etc.) than the original post. It very often happens when a popular account disagrees with your take. When someone with a big following replies to your post, their audience quickly sees it, and if their response is witty, harsh, or widely agreed with, it can snowball into a ratio very fast. Their followers will usually like and boost their reply, sometimes swarming your original post with disagreement or mockery.
- defrost 1y agoWhen a reply to a tweet gets more likes than the tweet it was replying to, usually indicating the unpopularity or stupidity of the original tweet Damn you got 12 likes and his reply got 271 likes? He totally ratioed you bro https://www.urbandictionary.com/define.php?term=Ratioed https://www.urbandictionary.com/define.php?term=Ratioed
- ForgedLabsJames 1y agoits fast AF bro!
- buremba 1y agoIn a world where LLMs can write code fairly well and make use of browsers, I'm not sure if MCP is truly the "USB-C port of AI applications." The more MCP tools I expose to the LLM, the harder it becomes for the LLM to get the job done. Instead, a single run_python tool works much better and faster. This is especially true for the reasoning models where context matters more.