6 ms·
Just posted my initial impressions, took a couple of hours to write them up because there's a lot in this release! https://simonwillison.net/2025/Aug/5/gpt-oss/
by simonw 1y ago
Just posted my initial impressions, took a couple of hours to write them up because there's a lot in this release! https://simonwillison.net/2025/Aug/5/gpt-oss/ https://simonwillison.net/2025/Aug/5/gpt-oss/
TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs. Will be interesting to see if independent benchmarks resolve in that direction as well.
The 20B model runs on my Mac laptop using less than 15GB of RAM.
- GodelNumbering 1y ago> The 20B model runs on my Mac laptop using less than 15GB of RAM. I was about to try the same. What TPS are you getting and on which processor? Thanks!
- hrpnk 1y agogpt-oss-20b: 9 threads, 131072 context window, 4 experts - 35-37 tok/s on M2 Max via LM Studio.
- mekpro 1y agoi got 70 token/s on m4 max
- mhitza 1y agoThat M4 Max is really something else, I get also 70 tokens/second on eval on a RTX 4000 SFF Ada server GPU.
- mdz4040 1y ago55 token/s here on m4 pro, turning on flash attention puts it to 60/s.
- coltonv 1y agoWhat did you set the context window to? That's been my main issue with models on my macbook, you have to set the context window so short that they are way less useful than the hosted models. Is there something I'm misisng there?
- rmonvfer 1y agoI’m also very interested to know how well these models handle tool calling as I haven’t been able to make it work after playing with them for a few hours. Looks promising tho.
- rmonvfer 1y agoupdate: I’ve tried to use lm-studio (like the author) and the tool request kept failing due to a mismatch in the prompt template. I guess they’ll fix it but seems sloppy from lm-studio not having tested this before release.
- month13 1y agoI was road testing tool calling in LM Studio a week ago against a few models marked with tool support, none worked, so I believe it may be a bug. Had much better luck with llama.cpp’s llama-server.
- hrpnk 1y agoI tried to generate a streamlit dashboard with MACD, RSI, MA(200). 1:0 for qwen3 here. qwen3-coder-30b 4-bit mlx took on the task w/o any hiccups with a fully working dashboard, graphs, and recent data fetched from yfinance. gpt-oss-20b mxfp4's code had a missing datatime import and when fixed delivered a dashboard without any data and with starting date of Aug 2020. Having adjusted the date, the update methods did not work and displayed error messages.
- teitoklien 1y agofor now, i wouldnt rank any model from openai in coding benchmarks, despite all the false messaging they are giving, almost every single model openai has launched even the high end o3 expensive models are absolutely monumentally horrible at coding tasks. So this is expected. If its decent in other tasks, which i do find openai often being better than others at, then i think its a win, especially a win for the open source community that even AI labs that pionered the hype of Gen AI who didnt want to ever launch open models are now being forced to launch them. That is definitely a win, and not something that was certain before.
- dongobread 1y agoIt is absolutely awful at writing and general knowledge. IMO coding is its greatest strength by far.
- mplewis 1y agoSure sounds like they're not good at anything in particular, then.
- xwolfi 1y agowelcome to 3DTV hype, LLM are useless...
- teitoklien 1y ago
- throwdbaaway 1y agoThere is no way that gpt-oss-120b can beat the much larger Kimi-K2-Instruct, Qwen3 Coder/Instruct/Thinking, or GLM-4.5. How did you arrive at this rather ridiculous conclusion? The current sentiment in r/LocalLLaMA is that gpt-oss-120b is around Llama-4 Scout level. But it is indeed the best in refusal.
- irthomasthomas 1y agoMy llm agent is currently running an experiment generating many pelicans. It will compare various small model consortiums against the same model running solo. It should push new pelicans to the repo after run. The horizon-beta is up already, not small or opensource but tested it anyway, and you can already see an improvement using 2+1 (2 models + the arbiter) for that model. https://irthomasthomas.github.io/Pelicans-consortium/ https://irthomasthomas.github.io/Pelicans-consortium/ https://github.com/irthomasthomas/Pelicans-consortium https://github.com/irthomasthomas/Pelicans-consortium
- markasoftware 1y agoThe space invaders game seems like a poor benchmark. Both models understood the prompt and generated valid, functional javascript. One just added more fancy graphics. It might just have "use fancy graphics" in its system prompt for all we know.
- simonw 1y agoThe way I run these prompts excludes a system prompt - I'm hitting the models directly.
- markasoftware 1y agostill, if you ask this open model to generate a fancy space invaders game with polish, and then ask the other model to generate a bare-bones space invaders game with the fewest lines of code, I think there's a good chance they'd switch places. This doesn't really test the models ability to generate a space invaders game, so much as it tests their tendency to make an elaborate vs simple solution.
- simonw 1y agoMy main goal with that benchmark is to see if it can produce HTML and JavaScript code that runs without errors for a moderately complex challenge. It's not a comprehensive benchmark - there are many ways you could run it in ways that would be much more informative and robust. It's great as a quick single sentence prompt to get a feeling for if the model can produce working JavaScript or not.
- deleted 1y ago[deleted]
- dennisy 1y agoNot really the other commenters are correct I feel and this is not really proving anything about the fundamental capability of the model. It’s just a hello world benchmark adding no real value, just driving blog traffic for you.
- mudkipdev 1y agoHasn't nailed the strawberry test yet
- pxc 1y agoI found this surprising because that's such an old test that it must certainly be in the training data. I just tried to reproduce and I've been unable to get it (20B model, lowest "reasoning" budget) to fail that test (with a few different words).
- quatonion 1y agoI am starting to get the impression the strawberry test is an OpenAI watermark, more than an actual problem. It is a good way to detect if another model was trained on your data for example, or is a distillation/quant/ablation.
- h4ny 1y ago> TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs. That's just straight up not the case. Not sure how you can jump to that conclusion not least when you stated that you haven't tested tool calling in your post too. Many people in the community are finding it substantially lobotomized to the point that there are "safe" memes everywhere now. Maybe you need to develop better tests that and pay more attention to benchmaxxing. There are good things that came out of these release from OpenAI but we'd appreciate more objective analyses...
- simonw 1y agoIf you read my full post, it ends with this: > I’m waiting for the dust to settle and the independent benchmarks (that are more credible than my ridiculous pelicans) to roll out, but I think it’s likely that OpenAI now offer the best available open weights models. You told me off for jumping to conclusions and in the same comment quoted me saying "I think OpenAI may have taken" - that's not a conclusion, it's tentative speculation.
- h4ny 1y agoI did read that and it doesn't change what I said about your comment on HN, I was calling out the fact that you are making a very bold statement without having done careful analysis. You know you have a significant audience, so don't act like you don't know what you're doing when you chose to say "TLDR: I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs" then defend what I was calling out based on word choices like "conclusions" (I'm sure you have read conclusions in academic journals?), "I think", and "speculation".
- simonw 1y agoI'm going to double down on "I think OpenAI may have taken the medal..." not being a "bold statement". I try to be careful about my choice of words, even in forum comments.
- 1y ago
- kgeist 1y ago>I think OpenAI may have taken the medal for best available open weight model back from the Chinese AI labs I have a bunch of scripts that use tool calling. Qwen-3-32B handles everything flawlessly at 60 tok/sec. Gpt-oss-120B breaks in some cases and runs at mere 35 tok/sec (doesn't fit on the GPU). But I hope there's still some ironing out to do in llama.cpp and in the quants. So far it feels lackluster compared to Qwen3-32B and GLM-4.5-Air
- EagnaIonat 1y agoNice write up! One test I do is to give a common riddle but word it slightly to see if it can actually reason. For example: "Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ???" The 20B model kept picking the answer of the original riddle, even after explaining extra information to it. The original riddle is: "Janes dad has five daughters, Lala, Lele, Lili, Lolo and ???"
- clbrmbr 1y agoI don’t get it. Wouldn’t it be Lulu in both cases?
- blueplanet200 1y agoOriginal riddle the answer is Jane.
- throwaway106382 1y agoIt’s Bob or Jane. The dad of <Person> has 5 daughters. Four are listed off. So the answer for the fifth is <Person>.
- philipwhiuk 1y agoExcept having five daughters doesn't prevent them also having 20 sons one of whom is called Bob.
- throwaway106382 1y agoThat’s why it’s a riddle.
- stavros 1y agoBut that makes no sense, if you ask me that John's dad has two daughters, Mary and ???, there's no way I will say "John" as an answer. I'll simply assume he also has at least one son.
- MichaelKSpencer 1y ago[dead]