9 ms·
how do we know it's not a quantized version of o3? what's stopping these firms from announcing the full model to perform well on the benchmarks and then gradual
by behnamoh 1y ago
how do we know it's not a quantized version of o3? what's stopping these firms from announcing the full model to perform well on the benchmarks and then gradually quantizing it (first at Q8 so no one notices, then Q6, then Q4, ...).
I have a suspicion that's how they were able to get gpt-4-turbo so fast. In practice, I found it inferior to the original GPT-4 but the company probably benchmaxxed the hell out of the turbo and 4o versions so even though they were worse models, users found them more pleasing.
- esafak 1y agoAre there any benchmarks that track historical performance?
- behnamoh 1y agogood question, and I don't know of any, although it's a no brainer that someone should make it. a proxy to that may be the anecdotal evidence of users who report back in a month that model X has gotten dumber (started with gpt-4 and keeps happening, esp. with Anthro and OpenAI models). I haven't heard such anecdotal stories about Gemini, R1, etc.
- SparkyMcUnicorn 1y agoAider has one, but it hasn't been updated in months. People kept claiming models were getting worse, but the results proved that they weren't.
- esafak 1y agohttps://aider.chat/docs/leaderboards/by-release-date.html https://aider.chat/docs/leaderboards/by-release-date.html
- __mharrison__ 1y agoUpdated yesterday... https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- vitaflo 1y agoThat Deepseek price is always hilarious to see in these charts.
- SparkyMcUnicorn 1y agoThat's not the one I'm referring to. See my other comments or your sibling comment.
- benterix 1y ago> users found them more pleasing. Some users. For me the drop was so huge it became almost unusable for the things I had used it for.
- behnamoh 1y agoSame here. One of my apps straight out stopped working because the gpt-4o outputs were noticeably worse than the gpt-4 that I built the app based on.
- lispisok 1y agoI swear every time a new model is released it's great at first but then performance gets worse over time. I figured they were fine-tuning it to get rid of bad output which also nerfed the really good output. Now I'm wondering if they were quantizing it.
- codr7 1y ago[flagged]
- bboygravity 1y agoBut OpenAI breathes honesty. They're open source! They would never do such a thing. /s
- daseiner1 1y agoIt's still a very competitive marketplace
- mathgradthrow 1y agohonestly refreshing take.
- nabla9 1y agoIt seems that least Google is overselling their compute capacity. You pay monthly fee, but Gemini is completely jammed 5-6 hours when North America is working.
- baq 1y agoGemini is simply that good. I’m trying out Claude 4 every now and then and go back to Gemini to fix its mess…
- fasterthanlime 1y agoFunny, I have the exact opposite experience! I use Claude to fix Gemini’s mess.
- coconut08 1y agoQuantization is a massive efficiency gain for near negligible drop in quality. If the tradeoff is quantization for an 80 percent price drop I would take that any day of the week.
- behnamoh 1y ago> for near negligible drop in quality Hmm, that's evidently and anecdotally wrong: https://github.com/ggml-org/llama.cpp/discussions/4110 https://github.com/ggml-org/llama.cpp/discussions/4110
- spiderice 1y agoYou may be right that the tradeoff is worth it, but it should be advertised as such. You shouldn't think you're paying for full o3, even if they're heavily discounting it.
- code_biologist 1y agoI would like the option to pay for the unquantized version. For creative or story writing (D&D campaign materials and such) quantization seems to end up in much weaker word selection and phrasing. There are small semantic missteps that break the illusion the LLM understands what it's writing. I find it jarring and deeply immersion breaking. I'd prefer prototype prompts on a cheaper quantized version, but I want to be able to spend 50 cents an API call to get golden output.
- CSMastermind 1y agoThis is almost certainly what they're doing and rebranding the original o3 model as "o3-pro"
- behnamoh 1y ago> rebranding the original o3 model as "o3-pro" interesting take, I wouldn't be surprised if they did that.
- anticensor 1y ago-pro models appear to be a best-of-10 sampling of the original full size model
- Szpadel 1y agohow do you sample it behind the scenes? usually best of X means you generate X outputs and you choose best result. if you could do this automatically, it would be game changer as you could run top 5 best models in parallel and select best answer every time but it's not practical because you are the bottleneck as you have to read all 5 solutions and compare them
- joshstrange 1y agoI think the idea is they use another/same model to judge all the results and only return the best one to the user.
- anticensor 1y agoI think the idea is they just feed each to the RLHF reward model used to train the model and return the most rewarded answer.
- anticensor 1y ago> if you could do this automatically, it would be game changer as you could run top 5 best models in parallel and select best answer every time remember they have access to the RLHF reward model, against which they can evaluate all N outputs and have the most "rewarded" answer picked and sent
- ants_everywhere 1y agoIs this what happened to Gemini 2.5 Pro? It used to be very good, but it's started struggling on basic tasks. The thing that gets me is it seems to be lying about fetching a web page. It will say things are there that were never on any version of the page and it sometimes takes multiple screenshots of the page to convince it that it's wrong.
- SparkyMcUnicorn 1y agoThe Aider discord community has proposed and disproven the theory that 2.5 Pro became worse, several times, through many benchmark runs. It had a few bugs here or there when they pushed updates, but it didn't get worse.
- ants_everywhere 1y agoGemini is objectively exhibiting new behavior with the same prompts and that behavior is unwelcome. It includes hallucinating information and refusing to believe it's wrong. My question is not whether this is true (it is) but why it's happening. I am willing to believe the aider community has found that Gemini has maintained approximately equivalent performance on fixed benchmarks. That's reasonable considering they probably use a/b testing on benchmarks to tell them whether training or architectural changes need to be reverted. But all versions of aider I've tested, including the most recent one, don't handle Gemini correctly so I'm skeptical that they're the state of the art with respect to bench-marking Gemini.
- SparkyMcUnicorn 1y agoGemini 2.5 Pro is the highest ranking model on the aider benchmarks leaderboard. For benchmarks, either Gemini writes code that adheres to the required edit format, builds successfully, and passes unit tests, or it doesn't. I primarily use aider + 2.5 pro for planning/spec files, and occasionally have it do file edits directly. Works great, other than stopping it mid-execution once in a while.
- code_biologist 1y ago
- jstummbillig 1y agoYou can just give it a go for very little money (in Windsurf it's 1x right now), and see what it does. There is no room for conspiracy here, because you can simple look at what it does. If you don't like it, so won't others, and then people will not use it. People are obviously very capable of (collectively) forming opinions on models, and then vote with their wallet.
- resters 1y agoIt's probably optimized in some way, but if the optimizations degrade performance, let's hope it is reflected in various benchmarks. One alternative hypothesis is that it's the same model, but in the early days they make it think "harder" and run a meta-process to collect training data for reinforcement learning for use on future models.
- SparkyMcUnicorn 1y agoIt's a bit dated now, but it would be cool if people submitted PRs for this one: https://aider.chat/docs/leaderboards/by-release-date.html https://aider.chat/docs/leaderboards/by-release-date.html
- __mharrison__ 1y agoDated? This was updated yesterday https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/
- SparkyMcUnicorn 1y agoMy link is to the benchmark results _over time_. The main leaderboard page that you linked to is updated quite frequently, but it doesn't contain multiple benchmarks for the same exact model.
- carter-0 1y agoAn OpenAI researcher claims it's the exact same model on X: https://x.com/aidan_mclau/status/1932507602216497608 https://x.com/aidan_mclau/status/1932507602216497608
- segmondy 1y agoyou don't, so run your own model.
- EnPissant 1y agoThe API lists o3 and o3-2025-04-16 as the same thing with the same price. The date based models are set in stone.
- hyperknot 1y agoI got 700+ tokens/sec on o3 after the announcement, I suspect it's very much a quantized version. https://x.com/hyperknot/status/1932476190608036243 https://x.com/hyperknot/status/1932476190608036243
- dist-epoch 1y agoOr maybe they just brought online much faster much cheaper hardware.
- az226 1y agoOr they are using a speedy add-on decoder.
- zackangelo 1y agoIs that input tokens or output tokens/s?
- beering 1y agoDo you also have numbers on intelligence before and after?
- Bjorkbat 1y agoRelated, when o3 finally came out ARC-AGI updated their graph because it didn’t perform nearly as well as the version of o3 that “beat” the benchmark. https://arcprize.org/blog/analyzing-o3-with-arc-agi https://arcprize.org/blog/analyzing-o3-with-arc-agi
- beering 1y agoThe o3-preview test was with very expensive amounts of compute, right? I remember it was north of $10k so makes sense it did better
- Bjorkbat 1y agoPoint remains though, they crushed the benchmark using a specialized model that you’ll probably never have access to, whether personally or through a company. They inflated expectations and then released to the public a model that underperforms
- throwaway314155 1y agoThey revealed the price points for running those evaluations. IIRC the "high" level of reasoning cost tens of thousands of dollars if not more. I don't think they really inflated expectations. In fact a lot of what we learned is that ARC-AGI probably isn't a very good AGI evaluation (it claims to not be one, but the name suggests otherwise).
- ctoth 1y agoFrom the announcement email: > Today, we dropped the price of OpenAI o3 by 80%, bringing the cost down to $2 / 1M input tokens and $8 / 1M output tokens. > We optimized our inference stack that serves o3—this is the same exact model, just cheaper.
- smusamashah 1y agoHw about testing same input vs output with same seed on different dates. If its a different model it will return different output.
- zomnoys 1y agoIsn’t this not true since these models run with a non-zero temperature?
- smusamashah 1y agoYou can set the temperature too.
- luke-stanley 1y agoI think the API has some special IDs to check for reproducibility of the environment.
- tedsanders 1y agoIt's the same model, no quantization, no gimmicks. In the API, we never make silent changes to models, as that would be super annoying to API developers [1]. In ChatGPT, it's a little less clear when we update models because we don't want to bombard regular users with version numbers in the UI, but it's still not totally silent/opaque - we document all model updates in the ChatGPT release notes [2]. [1] chatgpt-4o-latest is an exception; we explicitly update this model pointer without warning. [2] ChatGPT Release Notes document our updates to gpt-4o and other models: https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... (I work at OpenAI.)
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- az226 1y agoEven classic GPT-4 from March 2023 was quantized to 4.5 bits.
- rfoo 1y agoI don't work for OAI so obviously I can't say for them. But we don't do this. We don't make hobbyist mistakes of randomly YOLO trying various "quantization" methods that only happen after all training and claim it a day, at all. Quantization was done before it went live.