9 ms·
So far, it seems like this is the hierarchy o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf
by sss111 2y ago
So far, it seems like this is the hierarchy
o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini
o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf https://cdn.openai.com/o3-mini-system-card.pdf
- losvedir 2y agoWhat about "o1 Pro mode". Is that just o1 but with more reasoning time, like this new o3-mini's different amount of reasoning options?
- MichaelBurge 2y agoo1-pro is a different model than o1.
- JohnPrine 2y agoI don't think this is true
- losvedir 2y agoAre you sure? Do you have any source for that? In this article[0] that was discussed here on HN this week, they say (claim): > In fact, the O1 model used in OpenAI's ChatGPT Plus subscription for $20/month is basically the same model as the one used in the O1-Pro model featured in their new ChatGPT Pro subscription for 10x the price ($200/month, which raised plenty of eyebrows in the developer community); the main difference is that O1-Pro thinks for a lot longer before responding, generating vastly more COT logic tokens, and consuming a far larger amount of inference compute for every response. Granted "basically" is pulling a lot of weight there, but that was the first time I'd seen anyone speculate either way. [0] https://youtubetranscriptoptimizer.com/blog/05_the_short_case_for_nvda https://youtubetranscriptoptimizer.com/blog/05_the_short_cas...
- ukuina 2y agoIs o1-pro not the o1 equivalent of o3-mini-high?
- bobjordan 2y agoI have been paying $200 per month for 01-pro mode and I am very disappointed right now because they have completely replaced the model today. It used to think for 1-5 minutes and deliver an unbelievably useful one-shot answer. Now, it only thinks for 7 seconds just like the 03-mini model and I can't tell the difference in the answers. I hope this is just a day 1 implementation bug but I suspect they have just decided to throw the $200 per month customers under the bus so that they'd have more capacity to launch the 03 model for everybody. I can't tell the difference between the models now and it is definitely not because the free 03 model delivers the quality that 01-pro-mode had! I'm so disappointed!
- unification_fan 2y agoThey have been doing this since day 1. I was a subscriber in the first few months (then occasionally every few months to check out the progress) and so many people complained that OpenAI fiddled with the models daily without saying anything to anyone. It was always attributed to variability but we all know it's not.
- rcarmo 2y agoThis is why I use the Azure-hosted versions (disclosure: I’m an MS FTE, but I use all sorts of 3rd party models for my own projects) - I _know_ which version is behind each endpoint and when they will be replaced (you can also pin versions within a support window that varies according to model), so I don’t have to rework all my prompts and throw work away at the drop of a hat.
- forrestthewoods 2y agoOpenAI needs a new branding scheme.
- airstrike 2y agoChatGPT Series X O one
- nejsjsjsbsb 2y agoThe Llama folk know how. Good old 90s version scheme.
- sho_hn 2y agoI think OpenAI really needs to rethink its product naming, especially now that they have a portfolio where there's no such clear hierarchy, but they have a place along different axis (speed, cost, reasoning, capabilities, etc). Your summary attempt e.g. also misses o3-mini vs o3-mini-high. Lots of trade-ofs.
- echelon 2y agoIt's like AWS SKU naming (`c5d.metal`, `p5.48xlarge`, etc.), except non-technical consumers are expected to understand it.
- maeil 2y agoHave you seen Azure VM SKU naming? It's.. impressive.
- buildbot 2y agoAnd it doesn’t even line up with the actual instances you’ll be offered. At one point I was using some random Nvidia A10 node that was supposed to be similar to Standard_NV36adms_A10_v5, but was an NC series for some reason with slightly different letters…
- nejsjsjsbsb 2y agoThose are not names but hashes used to look up the specs.
- echelon 2y agoI was thinking we might treat model names analogously, but their specs can be moving targets.
- sss111 2y agoYeah I tried my best :( I think they could've borrowed a page out of Apple's book, even mountain names would be better. Plus Sonoma, Ventura, and Yosemite are cool names.
- deleted 2y ago[deleted]
- LoveMortuus 2y agoHow would the DeepSeek fit into this? Or can it not compare? I don't know much about this stuff, but I've heard recently many people talk about DeepSeek and how unexpected it was.
- sss111 2y agoDeepseek V3 is equivalent to 4o. Deepseek R1 is equivalent to o1 (if not better) I think someone should just build an AI model comparing website at this point. Include all benchmarks and pricing
- jsk2600 2y agoThis one is good: https://artificialanalysis.ai/ https://artificialanalysis.ai/
- withinboredom 2y agoLooks like this only compares commercial models, and not the ones I can download and actually run locally.
- TuxSH 2y agohttps://livebench.ai/#/ https://livebench.ai/#/ My experience is as follows: - "Reason" toggle just got enabled for me as a free tier user of ChatGPT's webchat. Apparently this is o3-mini - I have Copilot Pro (offered to me for free), which apparently has o1 too (as well as Sonnet, etc.) From my experience DeepSeek R1 (webchat) is more expressive, more creative and its writing style is leagues better than OpenAI's models, however it under-performs Sonnet when changing code ("code completion"). Comparison screenshots for prompt "In C++, is a reference to "const C" a "const reference to C"?": https://imgur.com/a/c-is-reference-to-const-c-const-reference-to-c-llm-comparison-km8cwZd https://imgur.com/a/c-is-reference-to-const-c-const-referenc... tl;dr keep using Claude for code and DeepSeek webchat for technical questions
- dutchbookmaker 2y ago
- gundmc 2y agoIf this is the hierarchy, why does 4o score so much higher than o1 on LLM Arena? Worrisome for OpenAI that Gemini's mini/flash reasoning model outscores both o1 and 4o handily.
- crazysim 2y agoIs it possible people are voting for speed of responsiveness too?
- kgeist 2y agoI suspect people on LLM Arena don't ask complex questions too often, and reasoning models seem to perform worse than simple models when the goal is just casual conversation or retrieving embedded knowledge. Reasoning models probably 'overthink' in such cases. And slower, too.
- imtringued 2y agoThe LLM Arena deletes your prompt when you restart so what's the point in trying to write a complicated prompt and testing an exhaustive number of pairs? It's easy to pin this on the users, but that website is hostile to putting in any effort. This is something I've noticed a lot actually. A lot of AI projects just give you an input field and call it a day. Expecting the user to do the heavy lifting.
- energy123 2y agoo1 on LLM Arena often times out (network error) while thinking. But they still allow you to vote and they make it seem as if your vote is registered.
- ActVen 2y agoI really wish they would open up the reasoning effort toggle on o1 API. o1 Pro Mode is still the best overall model I have used for many complex tasks.
- bobjordan 2y agoHave you tried the o1-pro mode model today, because now it sucks!
- ALittleLight 2y agoThat seems very bad. What's the point of a new model that's worse than 4o? I guess it's cheaper in the API and a bit better at coding - but, this doesn't seem compelling. With DeepSeek I heard OpenAI saying the plan was to move releases on models that were meaningfully better than the competition. Seems like what we're getting is the scheduled releases that are worse than the current versions.
- thegeomaster 2y agoIt's quite a bit better than coding --- they hint that it can tie o1's performance for coding, which already benchmarks higher than 4o. And it's significantly cheaper, and presumably faster. I believe API costs account for the vast majority of COGS at most today's AI startups, so they would be very motivated to switch to a cheaper model that has similar performance.
- mgens 2y agoRight. For large-volume requests that use reasoning this will be quite useful. I have a task that requires the LLM to convert thousands of free-text statements into SQL select statements, and o3-mini-high is able to get many of the more complicated ones that GPT-4o and Sonnet 3.5 failed at. So I will be switching this task to either o3-mini or DeepSeek-R1.
- usaar333 2y agoFor non-stem perhaps. For math/coding problems, o3 mini is tied if not better than o1.
- jes5199 2y agoI spent some time yesterday struggling with o3-mini-high trying to do a trigonometry problem, we went round and round and could not solve it. o1 solved it in one shot.
- koakuma-chan 2y agoYou cannot compare GPT-4o and o*(-mini) because GPT-4o is not a reasoning model.
- lxgr 2y agoSure you can. "Reasoning" is ultimately an implementation detail, and the only thing that matters for capabilities is results, not process.
- koakuma-chan 2y agoBy "reasoning" I meant the fact that o*(-mini) does "chain-of-thought", in other words, it prompts itself to "reason" before responding to you, whereas GPT-4o(-mini) just directly responds to your prompt. Thus, it is not appropriate to compare o*(-mini) and GPT-4o(-mini) unless you implement "chain-of-thought" for GPT-4o(-mini) and compare that with o*(-mini). See also: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/chain-of-thought https://docs.anthropic.com/en/docs/build-with-claude/prompt-...
- wordpad25 2y agoThat's like saying you can't compare a sedan to a truck. Sure you can. Even though one is more appropriate for certain tasks than the other.
- dutchbookmaker 2y agoIt is a nuanced point but what is better, a sedan or a truck? I think we are still at that stage of the conversation so it doesn't make much sense. I do think it is a good metaphor for how all this shakes out though in time.
- freehorse 2y agoYes you use the models for the same things, and one is better than the other for said thing. The reasoning process is an implementation detail that does not concern anybody when evaluating the models, esp since "open"ai does not expose it. I just want llms to to task X which is usually "write a function in Y language that does W, taking these Z stuff into account", and for that i have found no reason to switch away from sonnet yet.
- thot_experiment 2y agoat least if i ran the company you'd know that ChatGPTasdhjf-final-final-use_this_one.pt > ChatGPTasdhjf-final.pt > ChatGPTasdhjf.pt > ChatGPTasd.pt> ChatGPT.pt
- idonotknowwhy 2y agoDid you just ls my /workspace dir? Lol
- withinboredom 2y agoyeah, you can def tell they are partnered with Microsoft.
- lumost 2y agoI actually switched back from o1-preview to GPT-4o due to tooling integration and web search. I find that more often than not, the ability of GPT-4o to use these tools outweighs o1's improved accuracy.
- singularity2001 2y agono the reasoning models should not directly be compared with the normal models: they often take 10 times as long to answer which only makes sense for difficult questions