24 ms·
OpenAI O3-Mini
- siliconc0w 2y agoThe real heated contest here amongst the top AI labs is to see who can come up with the most confusing product names.
- not_a_bot_4sho 2y agoSomeone dropped the ball with Phi models. There is clearly an opportunity for XP and Ultimate and X/S editions.
- tdb7893 2y agoIt's nice to see Google finally having competition in a space it used to really dominate (though they definitely still are holding their own with all the Gemini naming). I feel like it takes real effort to have product names be this confusing and capricious
- gundmc 2y agoGemini naming seems pretty straightforward at this point. 2.0 is the full model, flash is a smaller/faster/cheaper model, and flash thinking is a smaller/faster/cheaper reasoning model with Cost.
- coder543 2y ago> 2.0 is the full model Not quite. "2.0 Flash" is also called 2.0. The "Pro" models are the full models. But, I love how they have both "gemini-exp-1206" and "gemini-2.0-flash-thinking-exp-01-21". The first one doesn't even say what type of model it is, presumably it should have been "gemini-2.0-pro-exp-1206", but they didn't want to label it that for some reason, and now they're putting a hyphen in the date string where they weren't before. Not to mention they have both "Flash" and "Flash-8B"... which I think will confuse people. IMO, it should be "Flash-${Parameters}B" for both of them if they're going to mention it for one. But, I generally think Google's Gemini naming structure has been pretty decent.
- TheOtherHobbes 2y agoSurprised Apple hasn't gone with iI Pro Max.
- dilap 2y agoHaven't used openai in a bit -- whyyy did they change "system" role (now basically an industry-wide standard) to "developer"? That seems pointlessly disruptive.
- logicchains 2y agoThey mention in the model card, it's so that they can have a separate "system" role that the user can't change, and they trained the model to prioritise it over the "developer" role, to combat "jailbreaks". Thank God for DeepSeek.
- BoorishBears 2y ago2 years ago I'd say it's an oversight, because there's 0 chance a top down directive would ask for this. But given how OpenAI employees act online these days I wouldn't be surprised if someone on the ground proposed it as a way to screw with all the 3rd parties who are using OpenAI compatible endpoints or even use OpenAI's SDK in their official docs in some cases.
- danielovichdk 2y agoI read this as a full on marketing note targeted towards software developers.
- mise_en_place 2y agoToo little too late IMO. This is not impressive at all, what am I missing here?
- ben_w 2y agoThere's only two kinds of software, prototype and obsolete. I was taught that last millennium.
- esafak 2y agoThat's not true. Is Google Maps a prototype or obsolete?
- ben_w 2y agoThe website or the database? I'd say the former is obsolete and the latter is still a prototype.
- esafak 2y agoI understand that saas products are constantly evolving but this is an unusual definition of obsolescence and prototypes. Google Maps has been running like a tank for two decades, and it is pretty feature complete.
- jstummbillig 2y agoIdk, everything: The price point + performance?
- sumedh 2y ago> This is not impressive at all, what am I missing here? Compared to?
- brcmthrowaway 2y agoGamechanger?
- sss111 2y agoThis time, it is a save-face release, especially because Azure, AWS, and OpenRouter have started offering DeepSeek
- aspect0545 2y agoIt’s always a game changer, isn’t it?
- colonelspace 2y agoYep. And groundbreaking.
- maeil 2y agoIt changes the landscape with its multifaceted approach.
- BoredPositron 2y agoIs it AGI yet?
- sss111 2y agoSo far, it seems like this is the hierarchy o1 > GPT-4o > o3-mini > o1-mini > GPT-4o-mini o3 mini system card: https://cdn.openai.com/o3-mini-system-card.pdf https://cdn.openai.com/o3-mini-system-card.pdf
- losvedir 2y agoWhat about "o1 Pro mode". Is that just o1 but with more reasoning time, like this new o3-mini's different amount of reasoning options?
- MichaelBurge 2y agoo1-pro is a different model than o1.
- JohnPrine 2y agoI don't think this is true
- losvedir 2y agoAre you sure? Do you have any source for that? In this article[0] that was discussed here on HN this week, they say (claim): > In fact, the O1 model used in OpenAI's ChatGPT Plus subscription for $20/month is basically the same model as the one used in the O1-Pro model featured in their new ChatGPT Pro subscription for 10x the price ($200/month, which raised plenty of eyebrows in the developer community); the main difference is that O1-Pro thinks for a lot longer before responding, generating vastly more COT logic tokens, and consuming a far larger amount of inference compute for every response. Granted "basically" is pulling a lot of weight there, but that was the first time I'd seen anyone speculate either way. [0] https://youtubetranscriptoptimizer.com/blog/05_the_short_case_for_nvda https://youtubetranscriptoptimizer.com/blog/05_the_short_cas...
- ukuina 2y agoIs o1-pro not the o1 equivalent of o3-mini-high?
- 2y ago
- vincentpants 2y agoWow, it got to the top of the front page so fast! Weird!
- dang 2y agoI took a quick look at the data and FWIW the votes look legit to me, if that's what you were wondering.
- vincentpants 2y agoIt actually was what I was wondering. Thank you @Dang!
- throwaway314155 2y agoI'm fairly certain it was sarcasm.
- deleted 2y ago[deleted]
- Qwuke 2y agoIt did get 29 points in 3 minutes, which seems like a lot even for a fan favorite, but is also consistent with previous OpenAI announcements here.
- johnneville 2y agoI posted a verge article first but then checked and saw the openai blog and posted that. I'd guess it's the officialness / domain that makes ppl click on this so easily.
- zurfer 2y agoto be fair, I was waiting for this release the whole day
- s-macke 2y agoo3-mini was announced for today, and OpenAI typically publishes in the morning hours (PT). Many people were eagerly waiting. The publication was imminent. I kept checking both Twitter and Hacker News for updates. Just add ten more people like me and the news will become top news within a few minutes. That is legit.
- pookieinc 2y agoCan't wait to try this. What's amazing to me is that when this was revealed just one short month ago, the AI landscape looked very different than it does today with more AI companies jumping into the fray with very compelling models. I wonder how the AI shift has affected this release internally, future releases and their mindset moving forward... How does the efficiency change, the scope of their models, etc.
- echelon 2y agoThere's no moat, and they have to work even harder. Competition is good.
- wahnfrieden 2y agoCollaboration is even better, per open source results. It is the closed competition model that’s being left in the dust.
- lesuorac 2y agoI really don't think this is true. OpenAI has no moat because they have nothing unique; they're using mostly other people's (like Transformers) architectures and other companies hardware. Their value-prop (moat) is that they've burnt more money than everybody else. That moat is trivially circumvented by lighting a larger pile of money and less trivially by lighting the pile more efficently. OpenAI isn't the only company. The Tech companies being beaten massively by Microsoft in #of H100s purchases are the ones with a moat. Google / Amazon with their custom AI chips are going to have a better performance per cost than others and that will be a moat. If you want to get the same performance per cost then you need to spend the time making your own chips which is years of effort (=moat).
- sangnoir 2y ago> That moat is trivially circumvented by lighting a larger pile of money and less trivially by lighting the pile more efficently. DeepSeek has proven that the latter is possible, which drops a couple of River crossing rocks into the moat.
- iamjackg 2y agoI'm very interested in their Jailbreak evaluations: they're new to me. I might have missed previous mentions.
- Ninjinka 2y ago50 messages a day -> 150 messages a day for Plus and Team users
- cjbarber 2y agoThe interesting question to me is how far these reasoning models can be scaled. With another 12 months of compute scaling (for synthetic data generation and RL) how good will these models be at coding? I talked with Finbarr Timbers (ex-DeepMind) yesterday about this and his take is that we'll hit diminishing returns – not because we can't make models more powerful, but because we're approaching diminishing returns in areas that matter to users and that AI models may be nearing a plateau where capability gains matter less than UX.
- futureshock 2y agoI think in a lot of ways we are already there. Users are clearly already having difficulty seeing which model is better or if new models are improving over old models. People go back to the same gotcha questions and get different answers based on the random seed. Even the benchmarks are getting very saturated. These models already do an excellent job with your homework, your corporate PowerPoints and your idle questions. At some point only experts would be able to decide if one response was really better than another. Our biggest challenge is going to be finding problem domains with low performance that we can still scale up to human performance. And those will be so niche that no one will care. Agents on the other hand still have a lot of potential. If you can get a model to stay on task with long context and remain grounded then you can start firing your staff.
- dagelf 2y agoDon't underestimate how much the long tail means to the general public.
- xyzzy9563 2y agoWhat is the comparison of this versus DeepSeek in terms of good results and cost?
- reissbaker 2y agoProbably a good idea to wait for external benchmarks like Aider, but my guess is it'll be somewhere between DeepSeek V3 and R1 in terms of benchmarks — R1 trades blows with o1-high, and V3 is somewhat lower — but I'd expect o3-mini to be considerably faster. Despite the blog post saying paid users can access o3-mini today, I don't see it as an option yet in their UI... But IIRC when they announced o3-mini in December they claimed it would be similar to 4o in terms of overall latency, and 4o is much faster than V3/R1 currently.
- Synaesthesia 2y agoDeepseek is the state of the art right now in terms of performance and output. It's really fast. The way it "explains" how it's thinking is remarkable.
- fpgaminer 2y agoDeepSeek is great because: 1) you can run the model locally, 2) the research was openly shared, and 3) the reasoning tokens are open. It is not, in my experience, state of the art. In all of my side by side comparisons thus far in real world applications between DeepSeek V3 and R1 vs 4o and o1, the latter has always performed better. OpenAI's models are also more consistent, glitching out maybe one in 10,000, whereas DeepSeek's models will glitch out 1 in 20. OpenAI models also handle edge cases better and have a better overall grasp of user intentions. I've had DeepSeek's models consistently misinterpret prompts, or confuse data in the prompts with instructions. Those are both very important things that make DeepSeek useless for real world applications. At least without finetuning them, which then requires using those huge 600B parameter models locally. So it is by no means state of the art. Gemini Flash 2.0 also performs better than DeepSeek V3 in all my comparisons thus far. But Gemini Flash 2.0 isn't robust and reliable either. But as a piece of research, and a cool toy to play with, I think DeepSeek is great.
- evertedsphere 2y ago>developer messages looks like finally their threat model has been updated to take into account that the user might be too "unaligned" to be trusted with the ability to provide a system message of their own
- logicchains 2y agoIf their models ever fail to keep ahead of the competition in terms of smarts, users are going to ditch them in mass for a competitor that doesn't treat their users like their enemy.
- reissbaker 2y ago...I'm pretty sure they just renamed the key...
- buyucu 2y agowhy should anyone use this when deepseek is free/cheaper? openai is no longer relevant.
- ilaksh 2y agoI don't think OpenAI is training on your data. At least they say they don't, and I believe that. I wouldn't be surprised if the NSA or something has access to data if they request it or something though. But DeepSeek clearly states in their terms of service that they can train on your API data or use it for other purposes. Which one might assume their government can access as well. We need direct eval comparisons between o3-mini and DeepSeek.. Or, well they are numbers so we can look them up on leaderboards.
- csomar 2y agoYou can pay for the compute and be certain that no one in recording your data with deepseek.
- seinecle 2y agoYes but DeepSeek models can be accessed through the APIs of Cloudflare or GitHub, in which case no training on your data takes place.
- ilaksh 2y agoTrue.
- lappa 2y agoOpenAI clearly states that they train on your data https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is...
- lemming 2y agoBy default, we do not train on any inputs or outputs from our products for business users, including ChatGPT Team, ChatGPT Enterprise, and the API. We offer API customers a way to opt-in to share data with us, such as by providing feedback in the Playground, which we then use to improve our models. Unless they explicitly opt-in, organizations are opted out of data-sharing by default. The business bit is confusing, I guess they see the API as a business product, but they do not train on API data.
- ks2048 2y agoI think OpenAI should just have a single public facing "model" - all these names and versions are confusing. Imagine if Google, during it's accent, had a huge array of search engines with code names and notes about what it's doing behind the scenes. No, you open the page and type in box. If they can make it work better next month, great. (I understand this could not apply to developers or enterprise-type API usage).
- johanvts 2y agoThats the role of ChatGPT?
- sroussey 2y agoNope. That lets you choose a from seven models right now.
- ehfeng 2y agoEarly Google search only provided web links. Google Images, News, Video, Shopping, Maps, Finance used to be their own search boxes. Only later did Google start unifying their search experiences. Yelp suffered greatly in the early 2010s when Google started putting Google Maps listings (and their accompanying reviews) in their search results. OpenAI will eventually unify their products as well.
- Deegy 2y agoIf google had to face the reality that distilling their search engine into multiple case-specific engines would have resulted in vastly superior search results, they surely would done (or considered) it. Fortunately for them a monolith search engine was perfectly fine (and likely optimal due to accrued network effects). OpenAI is basically signaling that they need to distill their monolith in order to serve specific segments of the marketplace. They've explicitly said that they're targeting STEM with this one. I think that's a smart choice, the most passionate early adopters of this tech are clearly STEM users. If the tech was such that one monolith model was actually the optimal solution for all use cases, they would just do that. Actually, this is their stated mission: AGI. One monolith that's best at everything is basically what AGI is.
- dgfitz 2y agoOh look, another model. Yay.
- devindotcom 2y agoSure as a clock, tick follows tock. Can't imagine trying to build out cost structures, business plans, product launches etc on such rapidly shifting sands. Good that you get more for your money, I suppose. But I get the feeling no model or provider is worth committing to in any serious way.
- puffybunion 2y agothis is the best outcome, though, rather than a monopoly, which is exactly what everyone is hoping to have.
- thom 2y agoTerrible time to open a shovel store, amazing time to pick up a shovel.
- turnsout 2y agoHmm, not seeing it in my dashboard yet (Tier 4)
- throwaway314155 2y agoThis has happened to me with (I think) every single major model release (llm or image gen) from OpenAI. They just lie in their release announcements which leaves people scrambling on the day of.
- sunaookami 2y agoIt appeared just now for me on Tier 3.
- turnsout 2y agoSame—I'll be curious to check it out!
- thunkingdeep 2y agoI’ll take the China Deluxe instead, actually. I’ve been incredibly pleased with DeepSeek this past week. Wonderful product, I love seeing its brain when it’s thinking.
- mechagodzilla 2y agoBeing able to see the thinking trace in R1 is so useful, as you can go back and see if it's getting stuck, making a wrong assumption, missing data, etc. To me that makes it materially more useful than the OpenAI reasoning models, which seem impressive, but are much harder to inspect/debug.
- thot_experiment 2y agoRunning it locally lets you INTERJECT IN IT'S THINKING IN REALTIME and I cannot stress enough how useful that is.
- amarcheschi 2y agoOh this is so cool
- Gooblebrai 2y agoYou mean it reacts to you writing something while it's thinking of that you can stop it while it's thinking?
- hmottestad 2y agoYou can stop it at any time, then modify what it's written so far...then press continue and let it continue thinking and answering.
- thot_experiment 2y agoFundamentally the UI is up to you, I have a "typing-pauses-inference-and-starts-gaslighting" feature in my homebrew frontend, but in OpenWebUI/Sillytavern you can just pause it and edit the chain of thought and then have it continue from the edit.
- ofou 2y agoI find quite interesting they're releasing three compute levels (low, medium, high), I guess now there's some way to cap the thinking tokens when using their API. Pricing for o3-mini [1] is $1.10 / $4.40 per 1M tokens. [1]: https://platform.openai.com/docs/pricing#:~:text=o3%2Dmini https://platform.openai.com/docs/pricing#:~:text=o3%2Dmini
- kevinsundar 2y agoBTW if you want to stay up to date with these kinds of updates from OpenAI you can follow them here: https://www.getchangelog.com/?service=openai.com https://www.getchangelog.com/?service=openai.com It uses GPT-4o mini to extract updates from the website using scrapegraphai so this is kinda meta :). Maybe I'll switch to o3 mini depending on cost. It's reasoning abilities, with a lower cost than o1, could be quite powerful for web scraping.
- random3 2y agoI might be missing some context here - to what specific context does your comment refer to? I'm asking because I don't see you in the conversation and you comments seems an out of context self-promoting plug.
- kevinsundar 2y agoHey! I'm sorry you feel that way. There's several people who have subscribed to updates to OpenAI from my comment so there is clearly value to other commenters. I understand not everyone is interested though. It's just a free side project I built and I make no money. Additionally, I believe my contribution to the conversation is that gpt-4o-mini, the previous model advertised as low-cost, works pretty well for my use case (which in this case can help others here). I'm excited to try out gpt-03-mini depending on what the cost looks like for web scraping purposes. Happy to report back here once I try it out.
- ryanhecht 2y ago> While OpenAI o1 remains our broader general knowledge reasoning model, OpenAI o3-mini provides a specialized alternative for technical domains requiring precision and speed. I feel like this naming scheme is growing a little tired. o1 is for general knowledge reasoning, o3-mini replaces o1-mini but might be more specialized than o1 for certain technical domains...the "o" in "4o" is for "omni" (referring to its multimodality) but the reasoning models start with "o" ...but they can't use "o2" for trademark reasons so they skip straight to "o3" ...the word salad is getting really hard to follow!
- kingnothing 2y agoThey really need someone in marketing. If the model is for technical stuff, then call it the technical model. How is anyone supposed to know what these model names mean? The only page of theirs attempting to explain this is a total disaster. https://platform.openai.com/docs/models https://platform.openai.com/docs/models
- rowanG077 2y agoIf marketing terms from intel, AMD, Dell and other tech companies have taught me anything, it's that they need LESS of people in marketing.
- TeMPOraL 2y agoBut think of all the other marketers whose job is to produce blogspam explaining confusing product names!
- ninetyninenine 2y agoI bet you can get one of their models to fix that disaster.
- ryanhecht 2y agoBut what would we call that model?
- kaaskop 2y agoHow's this compare to Mistral Small 3?
- coder543 2y agoMistral Small 3 is roughly comparable in capabilities to 4o-mini (apart from 4o-mini's support for multimodality)... o1-mini was already better than GPT-4o (full size) for tasks like writing code, and this is supposedly better than o1 (full size) for those tasks, so... o3-mini is supposedly in a completely different league from Mistral Small 3, and it's not even close. Of course, the model has only been out for a few hours, so whether it lives up to the benchmarks or not isn't really known yet.
- highfrequency 2y agoAnyone else confused by inconsistency in performance numbers between this announcement and the concurrent system card? https://cdn.openai.com/o3-mini-system-card.pdf https://cdn.openai.com/o3-mini-system-card.pdf For example- GPQA diamond system card: o1-preview 0.68 GPQA diamond PR release: o1-preview 0.78 Also, how should we interpret the 3 different shading colors in the barplots (white, dotted, heavy dotted on top of white)...
- deleted 2y ago[deleted]
- kkzz99 2y agoActually sounds like benchslop to me.
- deleted 2y ago[deleted]
- airstrike 2y agoHopefully this is a big improvement from o1. o1 has been very disappointing after spending sufficient time with Claude Sonnet 3.5. It's like it actively tries to gaslight me and thinks it knows more than I do. It's too stubborn and confidently goes off in tangents, suggesting big changes to parts of the code that aren't the issue. Claude tends to be way better at putting the pieces together in its not-quite-mental-model, so to speak. I told o1 that a suggestion it gave me didn't work and it said "if it's still 'doesn't work' in your setup..." with "doesn't work" in quotes like it was doubting me... I've canceled my ChatGPT subscription and, when I really need to use it, just go with GPT-4o instead.
- Deegy 2y agoI've also noticed that with cGPT. That said I often run into a sort of opposite issue with Claude. It's very good at making me feel like a genius. Sometimes I'll suggest trying a specific strategy or trying to define a concept on my own, and Claude enthusiastically agrees and takes us down a 2-3 hour rabbit hole that ends up being quite a waste of time for me to back track out of. I'll then run a post-mortem through chatGPT and very often it points out the issue in my thinking very quickly. That said I keep coming back to sonnet-3.5 for reasons I can't perfectly articulate. Perhaps because I like how it fluffs my ego lol. ChatGPT on the other hand feels a bit more brash. I do wonder if I should be using o1 as my daily driver. I also don't have enough experience with o1 to determine if it would also take me down dead ends as well.
- bazmattaz 2y agoReally interesting point you make about Claude. I’ve experienced the same. What is interesting is that sometimes I’ll question it and say “would it not be better to do it this way” and all of a sudden Claude u-turns and says “yes great idea that’s actually a much better approach” which leaves me thinking; are you just stroking my ego, if it’s a better approach then why didn’t you suggest it? However I have suggested worse approaches on purpose and sometime Claude does pick them up as less than optimal
- airstrike 2y agoI agree with this but o1 will also confidently take you into rabbit holes. You'll just feel worse about it lol and when you ask Claude for a post mortem, it too will find the answer you missed quickly The truth is these models are very stochastic you have to try new chats whenever you even moderately suspect you're going awry
- ilaksh 2y agoIt looks like a pretty significant increase on SWE-Bench. Although that makes me wonder if there was some formatting or gotcha that was holding the results back before. If this will work for your use case then it could be a huge discount versus o1. Worth trying again if o1-mini couldn't handle the task before. $4/million output tokens versus $60. https://platform.openai.com/docs/pricing https://platform.openai.com/docs/pricing I am Tier 5 but I don't believe I have access to it in the API (at least it's not on the limits page and I haven't received an email). It says "rolling out to select Tier 3-5 customers" which means I will have to wait around and just be lucky I guess.
- TechDebtDevin 2y agoGenuinely curious, What made you choose OpenAI as your preferred api provider? Its always been the least attractive to me.
- TeMPOraL 2y agoUntil recently they were the only game in town, so maybe they accrued significant spend back then?
- ilaksh 2y agoI have mainly been using Claude 3.5/3.6 Sonnet via API in the last several months (or since 3.5 Sonnet came out). However, I was using o1 for a challenging task at one point, but last I tested it had issues with some extra backslashes for that application. I also have tested with DeepSeek R1 and will test some more with that although in a way Claude 3.6 with CoT is pretty good. Last time I tried to test R1 their API was out.
- ipaddr 2y agoWho else might be a good choice? Deepseek is down. Who has the cheapest gpt3.5 level or above api
- TechDebtDevin 2y agoIve personaly been using Deepseek (which has been better than for 3.5 for a really long time), and Perplexity, which is nice for their built in search. Ive actually been using Deepseek since it was free. Its been generally good for me. Ive mostly chosen both because of pricing as I generally dont use APIs for extermely complex prompts.
- georgewsinger 2y agoDid anyone else notice that o3-mini's SWE bench dropped from 61% in the leaked System Card earlier today to 49.3% in this blog post, which puts o3-mini back in line with Claude on real-world coding tasks? Am I missing something?
- logicchains 2y agoMaybe they found a need to quantize it further for release, or lobotomise it with more "alignment".
- kkzz99 2y agoOr the number was never real to begin with.
- ben_w 2y ago> lobotomise Anyone can write very fast software if you don't mind it sometimes crashing or having weird bugs. Why do people try to meme as if AI is different? It has unexpected outputs sometimes, getting it to not do that is 50% "more alignment" and 50% "hallucinate less". Just today I saw someone get the Amazon bot to roleplay furry erotica. Funny, sure, but it's still obviously a bug that a *sales bot* would do that. And given these models do actually get stuff wrong, is it really incorrect for them to refuse to help with things they might be dangerous if the user isn't already skilled, like Claude in this story about DIY fusion? https://www.corememory.com/p/a-young-man-used-ai-to-build-a-nuclear https://www.corememory.com/p/a-young-man-used-ai-to-build-a-...
- Rastonbury 2y agoThey are implying the release was rushed and they had to reduce the functionality of the model in order to make sure it did not teach people how to make dirty bombs
- bee_rider 2y agoIf somebody wants their Amazon bot to role play as an erotic furry, that’s up to them, right? Who cares. It is working as intended if it keeps them going back to the site and buying things I guess. I don’t know why somebody would want that, seems annoying. But I also don’t expect people to explain why they do this kind of stuff.
- OutOfHere 2y agoWake me up when the full o3 is out.
- therein 2y agoMy guess is it will happen right after Sam Altman's next public freakout about how dangerous this new model they have in store is and how it tried to escape from its confinement and kidnap the alignment operator.
- ls_stats 2y agoThat's pretty much what Altman said about GPT-3 (or 2, I don't remember), he said it was too dangerous to release to the public.
- deleted 2y ago[deleted]
- msp26 2y agoI wish they'd just reveal the CoT (like gemini and deepseek do), it's very helpful to see when the model gets misled by something in your prompt. Paying for tokens you aren't even allowed to see is peak OpenAI.
- tucnak 2y agoI'm sorry, but it's over for OpenAI. Some have predicted this; including me back in November[1] when I wrote "o1 is a revolution in accounting, not capability" which although tongue-in-cheek, has so far turned out to be correct. I'm only waiting to see what Google, Facebook et al. will accomplish now that R1-Zero result is out the bag. The nerve, the cheek of this hysterical o3-mini release—insisting to hide the COT from the consumer still, is telling us one thing and one thing alone: OpenAI is no longer able to adapt to the ever-changing landscape. Maybe the Chinese haven't beaten them yet, but Google, Facebook et al. absolutely will, & without having to resort to deception. [1]: https://old.reddit.com/r/LocalLLaMA/comments/1gna0nr/popular_opinion_o1_is_a_revolution_in_accounting/ https://old.reddit.com/r/LocalLLaMA/comments/1gna0nr/popular...
- mediaman 2y agoYou don't need to wait for Google. Their Jan 21 checkpoint for their fast reasoning model is available on AIStudio. It shows full reasoning traces. It's very good, much faster than R1, and although they haven't released pricing, based on flash it's going to be quite cheap.
- tucnak 2y agoSure, their 01-21 reasoning model is really good, but there's no pricing for it! I care mostly about batching in Vertex AI, which is 17-30x times cheaper than competition (whether you use prompt caching or not) while allowing for audio, video, and arbitrary document filetype inputs; unfortunately Gemini 1.5 Pro/Flash have remained the two so-called "stable" options that are available there. I can appreciate Google's experimental models for all I can, but I cannot take them seriously until they allow me to have my sweet, sweet batches.
- kumarm 2y agoI ran some quick programming tasks I have used O1 previously: 1. 1/4th time for reasoning for most tasks. 2. Far better results.
- CamperBob2 2y agoCompared to o1 or o1-pro?
- yzydserd 2y agoA few quick tasks look to me like o3-mini-high is 4-10x faster for 80% of the quality. It gives very good and sufficient fast reasoning about coding tasks, but I think I'd ask o1-pro to do the task ie provide the code. o3-mini-high can keep up with you at thinking / typing speed, whereas o1-pro can take several minutes. Just a quick view after playing for an hour.
- Bjorkbat 2y agoI have to admit I'm kind of surprised by the SWE-bench results. At the highest level of performance o3-mini's CodeForces score is, well, high. I've honestly never really sat down to understand how elo works, all I know is that it scored better than o1, which allegedly as better than ~90% of all competitors on CodeForces. So, you know, o3-mini is pretty good at CodeForces. But it's SWE-bench scores aren't meaningfully better than Claude, 49.3 vs Claude's 49.0 on the public leaderboard (might be higher now due to recent updates?) My immediate thoughts, CodeForces (and competitive programming in general) is a poor proxy for performance on general software engineering tasks. Besides that, for all the work put into OpenAI's most recent model it still has a hard time living up to an LLM initially released by Anthropic some time ago, at least according to this benchmark. Mind you, the Github issues that the problems in SWE-bench were based-off have been around long enough that it's pretty much a given that they've all found their way into the training data of most modern LLMs, so I'm really surprised that o3 isn't meaningfully better than Sonnet.
- dagelf 2y agoI think the innovation here is probably that its a much smaller and so cheaper model to run.
- aprilthird2021 2y ago> My immediate thoughts, CodeForces (and competitive programming in general) is a poor proxy for performance on general software engineering task Yep. A general software engineering task has a lot of information encoded in it that is either already known to a human or is contextually understood by a human. A competitive programming task often has to provide all the context as it's not based off an existing product or codebase or technology or paradigm known to the user
- vectorhacker 2y agoYeah, I no longer consider the SWE-bench useful because these models can just "memorize" the solutions to the PRs.
- usaar333 2y agoI'm not that surprised. Codeforces is a relatively low area of vocabulary knowledge needed. Real software has a lot more complexity and constraints, as well as ambiguity. Claude nowhere scores so high on codeforces, but crushes o1 on webarena: https://web.lmarena.ai/leaderboard https://web.lmarena.ai/leaderboard I also ran a hold out test myself for o3 mini - asked it to implement a function I need for Python 2.5. Claude and O1 get it - o3 mini hard incorrectly believes some functions are available in this certain that aren't. If I correct it, it's revised solution is very hacky (technically works, but I would take Claude's solution over it)
- _boffin_ 2y agowhy is o1-pro not mentioned in there?
- Oras 2y ago200k context window $1.1/m for input $4.4/m for output I assume thinking medium and hard would consume more tokens. I feel the timing is bad for this release especially when deepseek R1 is still peaking. People will compare and might get disappointed with this model.
- GaggiX 2y agoThe model looks quite a bit better in the benchmarks so unless they overfit the model on them it would probably perform better than deepseek.
- WiSaGaN 2y agoMy vibe question checking suggests otherwise. Even o3-mini-high is not as good as r1, even though it's faster than r1. Considering o3-mini is more expensive per token. It's not clear o3-mini-high is cheaper than r1 either even r1 probably consumes more token per answer.
- kandesbunzler 2y agowell in my anecdotal tests, o3 mini (free) performed better than r1
- kandesbunzler 2y agoI compared free o3 mini vs Deepseek R1 (on their website) and in my tests o3 performed better every time (did some coding tests)
- IMTDb 2y agoI really don't get the point of those oX-mini models for chat apps. (API is different, we can benchmark multiple models for a given recurring taks and choose the best one taking costs into consideration). As part of my job, I am trying to promote usage of AI in my company (~150 FTE); we have an OpenAI chatGPT plus subscription for all employees. Roughly speaking the message is: "use GPT-4o all the time, use o1 (soon o3) if you have more complex tasks". What am I supposed to answer when people ask "when am I supposed to use o3-mini ? . And what the heck is o3-mini-high, how do I know when to use it ?". People aren't gonna ask the same question to 5 different models and burn all their rate limits; yet it feels that what's openAI is hoping people will do. Put those weirs models in a sub-menu for advanced users if you really want to, but is you can use o1 there is probably no reason for you to hake o3-mini and o3-mini-high as additional options.
- oezi 2y agoWhy not promote o1? 4o is rather sloppy in comparison
- IMTDb 2y ago99% of what people use ChatGPT is for very mundane stuff. Think “translate this email to English”, “fix spelling mistakes”, “write this better for me”. Data extraction (list of emails) is big as well. You don’t need o1 for that; and people make lot of those requests per day. Additionally, o1 does not have access to search and multimodality and taking a screenshot of something and asking questions about it is also a big use case. It’s easy to overlook how widely ChatGPT is used for very small stuff. But compounded it’s still a game changer for many people.
- xinayder 2y ago"oh no DeepSeek copied our product it's not fair" > proceeds to release a product based on DeepSeek ah, alas the hypocrisy...
- feznyng 2y agoo3 was announced in December. R1 arguably builds off the rumored approach of o1 (LLM + RL) although with major efficiency gains. I'm not a big fan of OpenAI but it's the other way around.
- Rooster61 2y agoThe thing they previewed back in December before the whole Deepseek kerfuffle this week? Don't get me wrong, I'm laughing at OpenAI just like everyone else, but if they were really copying Deepseek, they'd be releasing a smaller model distilled from Deepseek API responses, and have it be open source to boot. This is neither
- famouswaffles 2y agoR1 seems to be the only of these reasoning models that seem to have had gains in the creative writing side.
- nimithryn 2y agoAm I the only one who thinks that R1 is awful at creative writing? I've seen a lot of very credulous posts on twitter that are super excited about excerpts written by DeepSeek that I think are absolutely absymal. Am I alone in this? Maybe people have very different tastes than I do? (I have no formal training in creative writing, though I do read a lot of literature. Not claiming my tastes are superior - genuinely curious if other people disagree).
- famouswaffles 2y agoI mean, do you think this is awful ? https://pastebin.com/Ja14mt6L https://pastebin.com/Ja14mt6L
- nimithryn 2y agoIf R1 one-shotted this then I revise my opinion somewhat. This doesn't give me a gut "this is awful" emotional reaction (although I don't think it's good - it's pretty cliche, and I found my eyes glossing over pretty quickly). I was somewhat turned off of DeepSeek (the first few questions I gave it, it returned 100% hallucinated answers). But maybe I'll have to look into it more, thanks.
- famouswaffles 2y agoI regenerated a page a few times but yeah, I gave no other instructions besides that. Also I approached things page by page. That was 3 pages. It's cliche but this was the prompt: I want you to write the first book of a fantasy series. The novel will be over 450 pages long with 30 chapters. Each chapter should have between 15 to 18 pages. Write the first page of the first chapter of this novel. Do not introduce the elements of the synopsis or worldbuilding and story details too quickly. Weave in the world, characters, and plot naturally. Pace it out properly. That means that several elements of the story may not come into light for several chapters. I had a lot of success with it coming up with decidedly not cliche world building elements after I arranged a sort of interview style interrogation (It asked me questions about what I was looking for generally and generated world building elements along the way). However, once you start giving a lot of information about the world etc in the prompt as well then the pacing gets weird.
- yapyap 2y agoThey sure scrambled something together after DeepSeek sweeped the market.
- GoatInGrey 2y agoIndeed. Everyone knows that one can cobble together a frontier model and deploy it within three weeks.
- TechDebtDevin 2y agoNot to mention the model has been available to researchers for a month.
- mhb 2y agoMaybe they can get some advice from the AWS instance naming group.
- throwaway314155 2y agoTypical OpenAI release announcement where it turns out they're _actually_ doing some sort of delayed rollout and despite what the announcement says, no - you can't use o3-mini today.
- feverzsj 2y agoIt's already a dead end for a while now, as they can't improve o1 meaningfully anymore. The market is also losing patience quickly.
- czk 2y agoim just glad it looks like o3-mini finally has internet access the o1 models were already so niche that i never used them, but not being able to search the web made them even more useless
- oytis 2y agoLet me guess - everyone is mindblown.
- estsauver 2y agoI couldn't find in the documentation anything that describes the relative number of tokens that you get for low/medium/high. I'm curious if anyone can find that, I'd be curious to see how it plays out relative to DeepSeeks thinking sections.
- isusmelj 2y agoDoes anyone know why GPT4 has knowledge cutoff December 2023 and all the other models (newer ones like 4o, O1, O3) seem to have knowledge cutoff October 2023? https://platform.openai.com/docs/models#o3-mini https://platform.openai.com/docs/models#o3-mini I understand that keeping the same data and curating it might be beneficial. But it sounds odd to roll back in time with the knowledge cutoff. AFAIK, the only event that happened around that time was the start of the Gaza conflict.
- kikki 2y agoI think trained knowledge is less and less important - as these multi-modal models have the ability to search the web and have much larger context windows.
- kittikitti 2y ago[flagged]
- andrewstuart 2y agoI find Claude to be vastly better than any OpenAI model as a programming assistant. In particular the "reasoning" models just seem to be less good and more slow.
- chad1n 2y agoI think that OpenAI should reduce the prices even further to be competitive with Qwen or Deepseek. There are a lot of vendors offering Deepseek R1 for $2-2.5 per 1 million tokens output.
- othello 2y agoWould you have specific recommendations of such vendors?
- chad1n 2y agoFor example, `https://deepinfra.com/ https://deepinfra.com/` which asks for $2.5 per million on output or https://nebius.com https://nebius.com which asks for $2.4 per million output tokens.
- BoorishBears 2y agoAs the sibling comment mentions, you're not getting anything production grade for less than $7 per million and that's on input and output. Nebius is single digit TPS. 31 seconds to reply to "What's 1+1". Hopefully Deepseek will make it out of their current situation because in a very ironic way, the thing the entire market lost its mind over is not actually usable at the pricing that drove the hype: https://openrouter.ai/deepseek/deepseek-r1 https://openrouter.ai/deepseek/deepseek-r1
- druskacik 2y agoWell, it's $2.19 per million output tokens even directly on deepseek platform. https://api-docs.deepseek.com/quick_start/pricing/ https://api-docs.deepseek.com/quick_start/pricing/
- BoorishBears 2y agoTheir API platform has been down for 48 hours at this point
- rsanek 2y ago
- secondcoming 2y agoAnyone else stuck in a Cloudflare 'verify you're a human' doom loop?
- stiltzkin 2y ago[dead]
- tempeler 2y agoThey made a discount; it's very impressive; they probably found a very efficient way, so it's discounted. I guess there's no need to build a very large nuclear power plant or a $9 trillion chip factory to run a single large language model. Efficiency has skyrocketed, or thanks to competition, OpenAI's all problems were solved.
- jen729w 2y ago> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a coin-flip.
- danielmarkbruce 2y agoRLUHF, U = useless.
- brookst 2y agoThose prompts are so irritating and so frequent that I’ve taken to just quickly picking whichever one looks worse at a cursory glance. I’m paying them, they shouldn’t expect high quality work from me.
- deleted 2y ago[deleted]
- apparent 2y agoHave you considered the possibility that your feedback is used to choose what type of response to give to you specifically in the future? I would not consider purposely giving inaccurate feedback for this reason alone.
- isaacremuant 2y agoAlternatively, I'll use the tool that is most user friendly and provides the most value for my money. Wasting time on an anti pattern is not value nor is it trying to outguess the way that selection mechanism is used.
- MattDaEskimo 2y agoI don't want a model that's customized to my preferences. My preferences and understanding changes all the time. I want a single source model that's grounded in base truth. I'll let the model know how to structure it in my prompt.
- EcommerceFlow 2y agoFirst thing I noticed on API and Chat for it is THIS THING IS FAST. That alone makes it a huge upgrade to o1-pro (not really comparable I know, just saying). Can't imagine how much I'll get done with this type of speed.
- GaggiX 2y agoThe API pricing is almost exactly double the deepseek ones.
- cheema33 2y agoI like deepseek a lot. But they are currently very glitchy. The API service goes up and down a lot. Maybe they'll sort that out soon.
- orbital-decay 2y agoApparently they're under a very targeted DDoS for almost a month, with technical details shared in Chinese but very little discussion in English. Which is surprising, it's not like major AI products are getting DDoSed out of existence every day.
- RobinL 2y agoWow - this is seriously fast (o3-mini), and my initial impressions are very favourable. I was asking it to layout quite a complex html form from a schema and it did a very good job. Looking at the comments on here and the benchmark results I was expecting it to be a bit meh, but initial impressions are quite the opposite I was expecting it to perhaps be a marginal improvement for complex things that need a lot of 'reasoning', but it seems it's a bit improvement for simple things that you need doing fast
- arresin 2y agoIt’s 2x the price of R1: https://x.com/deedydas/status/1885440582103031940/photo/1 https://x.com/deedydas/status/1885440582103031940/photo/1 Is it twice as good though?
- RobinL 2y agoWhilst I had tried R1 before, I hadn't paid attention to how fast it was. I just tried some similar prompts and was pretty impressed with speed and quality. I think o3-mini was still a bit quicker though.
- AISnakeOil 2y agoThe naming convention is so messed up. o1, o3-mini (no o2, no o3???)
- igravious 2y agohttps://www.perplexity.ai/search/new?q=list%20of%20all%20OpenAI%27s%20models https://www.perplexity.ai/search/new?q=list%20of%20all%20Ope... :) OpenAI has developed a variety of models that cater to different applications, from natural language processing to image generation and audio processing. Here’s a comprehensive list of the current models available: ## Language Models - \*GPT-4o\*: The flagship model capable of processing text, images, and audio. - \*GPT-4o mini\*: A smaller, more cost-effective version of GPT-4o. - \*GPT-4\*: An advanced model that improves upon GPT-3.5. - \*GPT-3.5\*: A set of models that enhance the capabilities of GPT-3. - \*GPT-3.5 Turbo\*: A faster variant designed for efficiency in chat applications. ## Reasoning Models - \*o1\*: Focused on reasoning tasks with improved accuracy. - \*o1-mini\*: A lightweight version of the o1 model. - \*o3\*: The successor to o1, currently in testing phases. - \*o3-mini\*: A lighter version of the o3 model. ## Audio Models - \*GPT-4o audio\*: Supports real-time audio interactions and audio generation. - \*Whisper\*: For transcribing and translating speech to text. ## Image Models - \*DALL-E\*: Generates images from textual descriptions. ## Embedding Models - \*Embeddings\*: Converts text into numerical vectors for similarity tasks. - \*Ada\*: An embedding model with various sizes (e.g., ada-002). ## Additional Models - \*Text to Speech (Preview)\*: Synthesizes spoken audio from text. These models are designed for various tasks, including coding assistance, image generation, and conversational AI, making OpenAI's offerings versatile for developers and businesses alike[1][2][4][5]. Citations: [1] https://learn.microsoft.com/vi-vn/azure/ai-services/openai/concepts/models [2] https://platform.openai.com/docs/models [3] https://llm.datasette.io/en/stable/openai-models.html [4] https://en.wikipedia.org/wiki/OpenAI_API [5] https://industrywired.com/open-ai-models-list-top-models-to-consider/ [6] https://holypython.com/python-api-tutorial/listing-all-available-openai-models-openai-api/ [7] https://en.wikipedia.org/wiki/GPT-3 [8] https://stackoverflow.com/questions/78122648/openai-api-how-do-i-get-a-list-of-all-available-openai-models/78122662
- ben_w 2y agoThere's an o1-mini, there's an o3 it just hasn't gone live yet: https://openai.com/12-days/#day-12 https://openai.com/12-days/#day-12 they can't call it o2 because: https://en.wikipedia.org/wiki/The_O2_Arena https://en.wikipedia.org/wiki/The_O2_Arena and the venue's sponsor: https://en.wikipedia.org/wiki/O2_(UK) https://en.wikipedia.org/wiki/O2_(UK)
- Y4wn 2y ago[flagged]
- thimabi 2y agoDoes anyone know the current usage limits for o3-mini and o3-mini-high when used through the ChatGPT interface? I tried to find them on the OpenAI Knowledgebase, but couldn’t find anything about that.
- keenmaster 2y agoFor Plus users the limits are: o3-mini-high: 50 messages per week (just like o1, but it seems like these are non-shared limits, so you can have 50 messages per week with o1, run out, and still have 50 messages with o3-mini-high to use) o3-mini: 150 messages per day Source for the latter is their press release. They were more vague about o3-mini-high, but people have already tested its limits just by using it, and got the pop-up for 25 messages left after sending 25 messages. It's nice not to worry about running out of o1 messages now and have a faster model that's mostly as good (potentially better in some areas?). OpenAI really needs to release a middle tier for 30 to $40 though that has the same models as Pro but without infinite usage. I hate not having the smartest model and I don't want to pay $200; there's probably a middle ground where they can make as much or more money from me on a subscription tier that gives limited access to o1-pro.
- deleted 2y ago[deleted]
- scarface_74 2y agoThis took 1:53 in o3-mini https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360d3 https://chatgpt.com/share/679d310d-6064-8010-ba78-6bd5ed3360... The 4o model without using the Python tool https://chatgpt.com/share/679d32bd-9ba8-8010-8f75-2f26a792e0d3 https://chatgpt.com/share/679d32bd-9ba8-8010-8f75-2f26a792e0... Trying to get accurate results with the paid version of 4o with the Python interpreter. https://chatgpt.com/share/679d31f3-21d4-8010-9932-7ecadd0b870f https://chatgpt.com/share/679d31f3-21d4-8010-9932-7ecadd0b87... The share link doesn’t show the output for some reason. But it did work correctly. I don’t know whether the ages are correct. I was testing whether it could handle ordering I have no idea what conclusion I should draw from this besides depending on the use case, 4o may be better with “tools” if you know your domain where you are using it. Tools are relatively easy to implement with LangChain or the native OpenAI SDK.
- margalabargala 2y agoThe 4o model's output is blatantly wrong. I'm not going to look up if it's the order or the ages that are incorrect, but: 36. Abraham Lincoln – 52 years, 20 days (1861) 37. James Garfield – 49 years, 105 days (1881) 38. Lyndon B. Johnson – 55 years, 87 days (1963) Basically everything after #15 in the list is scrambled.
- scarface_74 2y agoThat was the point. The 4o model without using Python was wrong. The o3 model worked correctly without needing an external tool
- BeetleB 2y agoI would not expect any LLM to get this right. I think people have too high expectations for it. Now if you asked it to write a Python program to list them in order, and have it enter all the names, birthdays, and year elected in a list to get the program to run - that's more reasonable.
- scarface_74 2y ago
- simonw 2y agoI just pushed a new release of my LLM CLI tool with support for the new model and the reasoning_effort option: https://llm.datasette.io/en/stable/changelog.html#v0-21 https://llm.datasette.io/en/stable/changelog.html#v0-21 Example usage: llm -m o3-mini 'write a poem about a pirate and a walrus' \ -o reasoning_effort high Output (comparing that with the default reasoning effort): https://github.com/simonw/llm/issues/728#issuecomment-2628321507 https://github.com/simonw/llm/issues/728#issuecomment-262832... (If anyone has a better demo prompt I'd love to hear about it)
- beklein 2y agoThank you for all the effort you put into this tool and keeping it up to date!
- theturtle32 2y agoA reasoning model is not meant for writing poetry. It's not very useful to evaluate it on such tasks.
- mediaman 2y agoIt's not clear that writing poetry is a bad use case. Reasoning models seem to actually do pretty well with creative writing and poetry. Deepseek's R1, for example, has much better poem structure than the underlying V3, and writers are saying R1 was the first model where they actually felt like it was a useful writing companion. R1 seems to think at length about word choice, correcting structure, pentameter, and so on.
- 1propionyl 2y agoIndeed. I would assume that a reasoning model would do far better at things like actually maintaining meter or rhyme scheme, something that models (even with good attention mechanisms) generally do very poorly at.
- theturtle32 2y agoOk, that makes some sense. I guess I was thinking more about the creative and abstract nature of poetry, the free flowing kind, not so much about rigid structures of meter and rhyme.
- hybrid_study 2y agoIs this version a prank?
- RivieraKid 2y agoNo.
- simonw 2y agoI used o3-mini to summarize this thread so far. Here's the result: https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae050 https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae... For 18,936 input, 2,905 output it cost 3.3612 cents. Here's the script I used to do it: https://til.simonwillison.net/llms/claude-hacker-news-themes#user-content-adding-a--m-model-option https://til.simonwillison.net/llms/claude-hacker-news-themes...
- deleted 2y ago[deleted]
- layman51 2y agoI noticed that it thought that GoatInGrey wrote “openai is no longer relevant.” However, they were just quoting a different user (buyucu) who was the person who first wrote that.
- simonw 2y agoGood catch. That's likely an artifact of the way I flatten the nested JSON from the comments API. I originally did that to save on tokens but modern models have much larger input windows so I may not need to do that any more.
- acka 2y agoThis is why I keep advocating that JSON should die, or at least no longer be used with LLMs. LLMs (and human brains) are simply not wired up for counting nested curly brackets across long spans of quoted text joined by colons and commas, and it is far too easy for humans to make mistakes when chunking JSON. IMO, (Strict)YAML is a very good alternative, it has even been suggested to me by multiple LLMs when I asked them what they thought the best format for presenting conversations to an LLM would be. It is very easy to chunk simple YAML and present it to an LLM directly off the wire: you only need to remember to repeat the indentation and names of all higher level keys (properties) pertaining to the current chunk at the top of the chunk, then start a text block containing the remaining text in the chunk, and the LLM will happily take it from there: topic: subtopic: text: | Subtopic text for this chunk. If you want to make sure that the LLM understands that it is dealing with chunks of a larger body of text, you can start and end the text blocks of the chunks with an ellipsis ('...').
- jajko 2y agoA random idea - train one of those models on you, keep it aside, let it somehow work out your intricacies, moods, details, childhood memories, personality, flaws, strengths. Methods can be various - initial dump of social networks, personal photos and videos, maybe some intense conversation to grok rough you, then polish over time. A first step to digital immortality, could be a nice startup of some personalized product for rich, and then even regular folks. Immortality not in ourselves as meat bags of course, we die regardless, but digital copy and memento that our children can use if feeling lonely and can carry with themselves anywhere, or later descendants out of curiosity to hold massive events like weddings. One could 'invite' long lost ancestors. Maybe your grand-grand father would be a cool guy you could easily click with these days via verbal input. Heck even 3D detailed model. An additional service, 'perpetually' paid - keeping your data model safe, taking care of it, backups, heck even maybe give it a bit of computing power to to receive current news in some light fashion and evolve, could be extras. Different tiers for different level of services and care. Or am I decade or two ahead? I can see this as universally interesting across many if not all cultures.
- silverlake 2y agoO3-mini solved this prompt. DeepSeek R1 had a mental breakdown. The prompt: “Bob is facing forward. To his left is Ann, to his right is Cathy. Ann and Cathy are facing backwards. Who is on Ann’s left?”
- mark_l_watson 2y agoThanks for that example! Using 'ollama run' these models all failed: deepseek-r1:32b, mistral-small:latest, qwq:latest, and qwen2.5:14b.
- modeless 2y agoR1 or R1-Distill? They are not the same thing. I think DeepSeek made a mistake releasing them at the same time and calling them all R1. Full R1 solves this prompt easily for me.
- silverlake 2y agoI used R1 hosted at NVidia here: https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard https://build.nvidia.com/deepseek-ai/deepseek-r1/modelcard
- SparkyMcUnicorn 2y agoSame. With the recommended settings, it got it right. I regenerated a bunch of times, and it did suggest Cathy once or twice. R1 70b also got it right just as many times for me.
- modeless 2y agoHuh, that one got it wrong for me too. I don't have patience to try it 10 times each to see if it was a coincidence, but it is absolutely true that not all implementations of LLMs produce the same outputs. It is in fact common for subtle bugs to happen that cause the outputs to be worse but not catastrophically bad, and therefore go unnoticed. So I wouldn't trust any implementation but the original for benchmarking or even general use unless I tested it extensively.
- 2y ago
- EternalFury 2y agoo1-preview, o1, o1-mini, o3-mini, o3-mini (low), o3-mini (medium), o3-mini (high)... What's next? o4-mini (wet socks), o5-Eeny-meeny-miny-moe? I thought they had a product manager over there. They only need 2 names, right? ChatGPT and o. ChatGPT-5 and o4 would be next. This multiplication of the LLM loaves and fishes is kind of silly.
- mark_l_watson 2y agoOh, sweet: both o3-mini low and high support integrated web search. No integrated web search with o1. I prefer, for philosophical reasons, open weight and open process/science models, but OpenAI has done a very good job at productizing ChatGPT. I also use their 4o-mini API because it is cheap and compares well to using open models on Groq Cloud. I really love running local models with Ollama but the API venders keep the price so low that I understand most people not wanting the hasssle if running Deepseek-R, etc., locally.
- swyx 2y agofor those interested, updated my o3-mini price chart to compare the cost-intelligence frontier with deepseek: https://x.com/swyx/status/1885432031896887335 https://x.com/swyx/status/1885432031896887335
- bix6 2y agoThey use the word reasoning a lot in the post. Is this reasoning or statistical prediction?
- jokoon 2y agoDoes that mean I can use this on my recent gaming AMD gpu?
- rednafi 2y agoThe most important detail for me was that in coding, it’s weaker than 4o and stronger than o1-mini. So I don’t have much use for it.
- usaar333 2y agoWhere are you reading that?
- xmichael909 2y agoSo can I ditch the $200 a month o1 pro account, and go back to the $20 account with 03-mini?
- genidoi 2y agoWith o1 pro you're paying for unlimited compute that you don't get with $20 + capped o1.
- modeless 2y agoInitial vibes are not living up to the hype. It fails my pet prompt, and the Cursor devs say they still prefer Sonnet[1]. I'm sure it will have its uses but it is not going to dominate. [1] https://x.com/cursor_ai/status/1885415392677675337 https://x.com/cursor_ai/status/1885415392677675337
- sandos 2y agoHow many benchmarks for LLMs are there out there? Is there any evidence of over-fitting on benchmarks, or is there truely hidden parts to them?
- diegocg 2y agoI hope chatgpt reconsiders the naming of their models some time. I have troubles deciding which model is the one I should use.
- badgersnake 2y ago56% is pretty close to ‘don’t give a toss’
- mvdtnz 2y agoWake up honey a new lie generator just dropped.
- sourcecodeplz 2y agoEven for free users, that is nice
- vok 2y agoWell, o3-mini-high just successfully found the root cause of a seg fault that o1 missed: mistakenly using _mm512_store_si512 for an unaligned store that should have been _mm512_storeu_si512.
- nextworddev 2y agorip development jobs /s.. or not /s
- glouwbug 2y agoThe output of LLMs will be suboptimal for some time. Companies stand to earn more by hiring juniors who can reason without LLMs than juniors who copy paste from LLMs
- throw83288 2y agoHow do I avoid the angst about this stuff as a student in computer science? I love this field but frankly I've been at a loss since the rapid development of these models.
- jumploops 2y agoLLMs are the new compilers. As a student, you should continue to focus on fundamentals, but also adapt LLMs into your workflow where you can. Skip writing the assembly (now curly braces and semicolons), and focus on what the software you’re building actually does, who it serves, and how it works. Programming is both changing a lot, and not at all. The mechanics may look different, but the purpose is still the same: effectively telling computers what to do.
- unification_fan 2y ago> LLMs are the new compilers. This shows a grave misunderstanding of what compilers and LLMs are. They're fundamentally opposite concepts. Compilers are about optimizing abstract code down to the most efficient representation possible for some hardware. LLMs are about wasting petaflops (made possible by compiler engineers) to produce random statements that don't have any static guarantees.
- ern 2y agoI haven’t bothered with o3 mini, because who wants an “inferior” product? I was using 4o as a “smarter Google” until DeepSeek appeared (although its web search is being hammered now and I’m just using Google ). o1 seems to have been neutered in the last week lots of disclaimers and butt-covering in its responses. I also had an annoying discussion with o1 about the DC plane crash..it doesn’t have web access and its cutoff is 2024, so I don’t expect it know about the crash. However, after saying such an event is extremely unlikely and being almost patronisingly reassuring, it treated pasted news articles and links (which to be sure, it can’t access) as “fictionalized”, instead of acknowledging its own cut-off date, and that it could have been wrong. In contrast DeepSeek (with web search turned off) was less dismissive of the risks in DC airspace, and more aware of its own knowledge cut-off. Coupled with the limited number of o1 responses for ChatGPT Plus, I’ve cancelled my subscription for now.
- deleted 2y ago[deleted]
- profsummergig 2y agoCan someone please share the logic behind their version naming convention?
- cyounkins 2y agoI switched an agent from Sonnet V2 to o3-mini (default medium mode) and got strangely poor results: only calling 1 tool at a time despite being asked to call multiple, not actually doing any work, and reporting that it did things it didn't
- binary132 2y agoNot really impressed by the answers I just got.
- sshh12 2y agoI built a silly political simulation game with this: https://state.sshh.io/ https://state.sshh.io/ https://github.com/sshh12/state-sandbox https://github.com/sshh12/state-sandbox
- n0id34 2y agoIs AI fizzing out or just me? I feel like they're trying to smash out new models as fast as they can but in reality they're barely any different, it's turning into the smartphone market. New iPhone with a slightly better camera and slightly differently bevelled edges, get it NOW! But doesn't actually do anything better than the iPhone 6. Claude, GPT 4 onwards, and DeepSeek all feel the same to me. Okay to a point, then kinda useless. More like a more convenient specialised Google that you need to double check the results of.
- nextworddev 2y agoon the contrary, it's accelerating since they unlocked a new paradigm of scaling
- ktzar 2y agoI don’t think they’ve improved much for common use since GPT-3.5, to be frank. They’re cheaper and more ubiquitous, yes, but when it comes to summarizing and generating basic text, they’re pretty much the same as they were back then Maybe we're just getting more used to make it part of our workflow.
- lordofgibbons 2y agoBoiling frog. The advances are happening so rapidly, but incrementally, that it's not being registered. It just seems like the normal state. Compare LLMs from a year or two ago with the ones out today on practically any task. It's night and day difference. This is specially so when you start taking into account these "reasoning" models. It's mind blowing how much better they are than "non-reasoning" models for tasks like planning and coding. https://aider.chat/docs/leaderboards/#aider-polyglot-benchmark-results https://aider.chat/docs/leaderboards/#aider-polyglot-benchma...
- n0id34 2y agoHmmm I guess it's the way I use them then, because the latest models feel almost less intelligent than the likes of GPT4. Certainly not "night and day" difference from my daily or every other day use case experience. I guess it's probably far more noticeable on benchmarks and far more advanced stuff than I'm using, but I would have assumed that would be the minority and that the majority of people use it similar to how I do.
- Alifatisk 2y agoAny comparison with other models yet?
- antirez 2y agoJust tested two complicated coding tasks, and surprisingly o3-mini-high nailed it while Sonnet 3.5 failed it. Will do more tests tomorrow.
- lenerdenator 2y agoNo self-host, no care.
- deleted 2y ago[deleted]
- AutistiCoder 2y agothe o3-mini model would be useful to me if coding's the only thing I need to do in a chat log. When I use ChatGPT these days, it's to help me write coding videos and then the social media posts around those videos. So that's two specialties in one chat log.
- anotherpaulg 2y agoFor AI coding, o3-mini scored similarly to o1 at 10X less cost on the aider polyglot benchmark [0]. This comparison was with both models using high reasoning effort. o3-mini with medium effort scored in between R1 and Sonnet. 62% $186 o1 high 60% $18 o3-mini high 57% $5 DeepSeek R1 54% $9 o3-mini medium 52% $14 Sonnet 48% $0 DeepSeek V3 [0] https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- throw83288 2y agoWhat do you expect to come from full o3 in terms of automating software engineering?
- nextworddev 2y agoo3 (high) might score 80%+
- stavros 2y agoDo you have plans to try o3-mini-high as the architect and Sonnet as the model?
- seunosewa 2y agoYou're conflating the low price of the o3-mini medium effort model with the high performance of the o3-mini high effort model. OpenAI hasn't listed the price for the o3-mini high effort model separately on their pricing page.
- mvkel 2y agoI've been using cursor since it launched, sticking almost exclusively to claude-3.5-sonnet because it is incredibly consistent, and rarely loses the plot. As subsequent models have been released, most of which claim to be better at coding, I've switched cursor to it to give them a try. o1, o1-pro, deepseek-r1, and the now o3-mini. All of these models suffer from the exact same "adhd." As an example, in a NextJS app, if I do a composer prompt like "on page.tsx [15 LOC], using shadcn components wherever possible, update this page to have a better visual hierarchy." sonnet nails it almost perfectly every time, but suffers from some date cutoff issues like thinking that shadcn-ui@latest is the repo name. Every single other model, doesn't matter which, does the following: it starts writing (from scratch), radix-ui components. I will interrupt it and say "DO NOT use radix-ui, use shadcn!" -- it will respond with "ok!" then begin writing its own components from scratch, again not using shadcn. This is still problematic with o3-mini. I can't believe it's the models. It must be the instruction-set that cursor is giving it behind the scenes, right? No amount of .cursorrules, or other instruction, seems to get cursor "locked in" the way sonnet just seems to be naturally. It sucks being stuck on the (now ancient) sonnet, but inexplicably, it remains the only viable coding option for me. Has anyone found a workaround?
- jvanderbot 2y agoI've found cursor to be too thin a wrapper. Aider is somehow significantly more functional. Try that.
- dhc02 2y agoAider, with o1 or R1 as the architect and Claude 3.5 as the implementer, is so much better than anything you can accomplish with a single model. It's pretty amazing. Aider is at least one order of magnitude more effective for me than using the chat interface in Cursor. (I still use Cursor for quick edits and tab completions, to be clear).
- ChadNauseam 2y agoI normally use aider by just typing in what I want and it magically does it. How do I use o1 or R1 to play the role of the "architect"?
- energy123 2y agoHow to disable the LLM summarized thought traces that get spammed into my chat window with o3-mini-high? Very annoying now having to manually press the "^" to hide the verbose thought traces every single question I ask, totally breaks flow.
- simonw 2y agoNow that the dust is settling a little bit, I have published my notes so far on o3-mini here: https://simonwillison.net/2025/Jan/31/o3-mini/ https://simonwillison.net/2025/Jan/31/o3-mini/ To save you the click: I think the most interesting things about this model are the price - less than half that of GPT-4o while being better for many things, most notably code - and the increased length limits. 200,000 tokens input and 100,000 output (compared to 128k/16k for GPT-4o and just 8k for DeepSeek R2 and Claude 3.5 on output) could open up some interesting new applications, especially at that low price.
- nprateem 2y agoI've found with sonnet that the longer its output the less it adheres to the prompt. Ask it to write something and not sound robotic and it's exceptional up to 1000 words. Then it starts becoming more and more robotic. At 3500 words every last paragraph starts "Remember,...", etc
- zora_goron 2y agoDoes anyone know, how "reasoning effort" is implemented technically - does this involve differences in the pre-training, RL, or prompting phases (or all)?
- deleted 2y ago[deleted]
- Havoc 2y ago5 hours in 500 odd comments. Definitely feels like this has less wow factor than previous OAI releases.
- jiocrag 2y agoThis is.... underwhelming.
- mohsen1 2y agoIt's funny because I asked it to fix my script to show deepseek's chain of thoughts in the script but it refuses to answer hahaha
- catigula 2y agoIt's actually a bit comforting that it isn't very good.
- waynecochran 2y agoI just had it convert Swift code to Kotlin and was surprised at how the comment was translated. It "knew" the author of the paper and what is was doing!? That is wild. Swift: // // Double Reflection Algorithm from Table I (page 7) // in Section 4 of https://tinyurl.com/yft2674p // for i in 1 ..< N { let X1 = spine[i] ... Kotlin: // Use the Double Reflection Algorithm (from Wang et al.) to compute subsequent frames. for (i in 1 until N) { val X1 = Vector3f(spine[i]) ...
- smallerize 2y agoWow, haven't seen a viglink in a while.
- waynecochran 2y agoNot sure I meant that to happen. I am sure I just went to tinyurl to make the link -- I don't know how it ended up this way.
- mattmanser 2y agoSomeone else has written this exact code on the internet, OpenAI stole it, and now chatgpt is regurgitating it. Just like it can regurgitate whole articles. You need to stop being wow'd by human intelligence masquerading as AI!
- smallerfish 2y agoWhere? Searching for "Use the Double Reflection Algorithm (from Wang et al.)" only returns this post.
- waynecochran 2y agoUnlikely. I wrote this myself. Of course it is in a private github repo.
- throwuxiytayq 2y agoyou are so confused it’s not even funny
- deleted 2y ago[deleted]
- prompt_overflow 2y agoPlot twist: 1. they are trying to obfuscate deepscrape success 2. they are trying to confuse you. the benchmark margins are minimal (and meaningless) 3. they are trying to get time (with investors) releasing nothing-special-models in a predicted schedule (jan -> o3, feb -> o3-pro-max, march -> o7-ultra, and in 2026 -> OMG! we've reached singularity! (after spending $500B)) - And at the end of the day, nothing changes for me and neither for you. enjoy your time out of this sickness ai hype. bruh!
- zone411 2y agoIt scores 72.4 on NYT Connections, a significant improvement over the o1-mini (42.2) and surpassing DeepSeek R1 (54.4), but it falls short of the o1 (90.7). (https://github.com/lechmazur/nyt-connections/ https://github.com/lechmazur/nyt-connections/)
- Mr_Bees69 2y agoCan't wait till deepseek gets their hands on this
- aussieguy1234 2y agoJust gave it a go using open-webui. One immediate difference I noticed is that o3-mini actually observes the system prompt you set. So if I say it's a Staff Engineer at Google, it'll stay in character. That was not possible with o1-mini, it ignored system prompts completely.
- revskill 2y agoModels should be better at clarifying the prompt before actually spamming with bad answers.
- energy123 2y agoThis might be the best publicly available model for coding: https://livebench.ai/#/?Coding=as https://livebench.ai/#/?Coding=as
- system7rocks 2y agoAI is absolute trash. It's a scam. Like Bitcoin. And LibreOffice.
- anoncow 2y agoHow long before o3 is available in GitHub copilot?
- sirolimus 2y agoNot open source, nothing to see here.
- sirolimus 2y agoProprietary AI models are not going to last. Not scalable, not moral and not worth the money.
- llamataboot 2y agoI wish they picked different descriptors than various ox with turbo mini preview etc and some of them shockingly different (o1 preview versus o1 regular for example) just give us three or four buckets of combos of cheap and fast and good :p it’s to the point where I have to look up prices and abilities and decide on a model for every use case and realize I made a mistake in the one I was using the day before.
- deleted 2y ago[deleted]
- 0xFEE1DEAD 2y agoAfter o3 was announced, with the numbers suggesting it was a major breakthrough, I have to say I’m absolutely not impressed with this version. I think o1 works significantly better, and that makes me think the timing is more than just a coincidence. Last week Nvidia lost 600 billion because of DeepSeek R1, and now OpenAI comes out with a new release which feels like it has nothing to do with the promises that were being made about o3.
- llm_trw 2y agoHaving tried using it, it is much worse than r1. Both the standard and high effort version.
- CryptoBanker 2y agoIf it’s actually available, it can’t be that much worse than R1 which currently only completes a response about 50% of the time for me.
- SkyPuncher 2y agoYea, o3-mini was a massive step down from Sonnet for coding tasks. R1 is my cost effective programmer. Sonnet is my hard problem model still.
- yohbho 2y ago"we are releasing", and by releasing, we do not mean releasing. Is this company worth anything?
- cranberryturkey 2y agoWasn’t available on the $20 plan for me.
- _giorgio_ 2y agoIt is going to be available. I only got it some hours after the official release
- gmaster1440 2y agoi think it says, amongst other things, that there is a salient difference between competitive programming like codeforce and real-world programming. u can train a model to hillclimb elo ratings on codeforce, but that won't necessarily directly translate to working on a prod javascript codebase. anthropic figured out something about real world coding that openai is still trying to catch up to, o3-mini-high notwithstanding.
- resource_waste 2y agoFurther confirmation that LLMs are deep into diminishing returns. It seemed to be true with every 'advancement' after ChatGPT4. Layers are cool, but that isnt AI/LLMs.
- sixhobbits 2y agoFor years I've been asking all the models this mixed up version of the classic riddle and they 99% of the time get it wrong and insist on taking the goat across first. Even the other reasoning models would reason about how it was wrong, figure out the answer, and then still conclude goat. o3-mini is the first one to get it right for me. Transcript: Me: I have a wolf, a goat, and a cabbage and a boat. I want to get them all across the river. If I leave the goat and the wolf together, the goat will eat the wolf. If I leave the wolf and the cabbage together, the wolf will eat the cabbage. I can only take one at a time. How do I do it? o3-mini 1. Take the wolf across to the right bank. 2. Return alone to the left bank. 3. Take the goat across to the right bank. 4. Bring the wolf back to the left bank. 5. Take the cabbage across to the right bank. 6. Return alone to the left bank. 7. Finally, take the wolf across to the right bank. Me: I have a wolf, a goat, and a cabbage and a boat. I want to get them all across the river. If I leave the goat and the cabbage together, the cabbage will eat the goat. If I leave the wolf and the cabbage together, the wolf will eat the cabbage. I can only take one at a time. How do I do it? o3-mini 1. Take the cabbage to the right bank. 2. Go back empty to the left bank. 3. Take the wolf to the right bank. 4. Bring the cabbage back to the left bank. 5. Take the goat to the right bank. 6. Go back empty to the left bank. 7. Finally, take the cabbage to the right bank.
- nwienert 2y agoIf you’ve talked to OpenAI about this riddle before it’s certain that conversation is now in the training set.
- martin-t 2y agoAt first I was curious if OP tried arguing with it or telling it the correct answer. If he just asked the question without any further interaction, it would still have no data that it generated the wrong answer. But as another comment says, many people have tried similar things and posted about it online, that information is siphoned by now and the LLMs weights get updated.
- 2y ago
- deleted 2y ago[deleted]
- rasengan 2y agoo1-pro got turned off? and o3-mini-high is significantly worse than o1-pro :(
- kizer 2y agoFirst AI model to pass my test on the first try (I used o3-mini-high). Prompt: Write an interpreter for a simple but practical scripting language. Write the interpreter in JavaScript to be run on the Node.JS platform. You can import any of the standard Node.JS modules. Churned out ~750 lines and a sample source code file to run the interpreter on. Ran on the first try completely correctly. Definitely a step up. Perhaps it's in the training data. I don't know. But no other model has ever produced an error-free and semantically correct program on the first try, and I don't think any ever managed to implement closures.
- airbreather 2y agoI tried to get it to build me a slightly challenging app to break out data from a fairly obscure file format for some PLC code, after having tried with Claude. o3-mini produced volumes of code more quickly and more of it, but Claude still had greater insight in to the problem and decoded the format to a noticeably greater degree. Whereas 03-mini quickly got to a certain point, it wasn't long before it was obvious it wasn't really going any further - like it's big cousin, but in it's own way, it was lazy and forgetful, seeming at times more interested in telling me what I might try than actually trying itself. Interestingly, even when I gave it a copy of Claude's code it still wasn't able to get to the same depth of understanding.
- wseqyrku 2y agoOpenAI o3-mini "brought" to you by deepseek.