13 ms·
Running a model comparable to o3 on a 24GB Mac Mini is absolutely wild. Seems like yesterday the idea of running frontier (at the time) models locally or on a m
by x187463 1y ago
Running a model comparable to o3 on a 24GB Mac Mini is absolutely wild. Seems like yesterday the idea of running frontier (at the time) models locally or on a mobile device was 5+ years out. At this rate, we'll be running such models in the next phone cycle.
- tedivm 1y agoIt only seems like that if you haven't been following other open source efforts. Models like Qwen perform ridiculously well and do so on very restricted hardware. I'm looking forward to seeing benchmarks to see how these new open source models compare.
- Rhubarrbb 1y agoAgreed, these models seem relatively mediocre to Qwen3 / GLM 4.5
- modeless 1y agoNah, these are much smaller models than Qwen3 and GLM 4.5 with similar performance. Fewer parameters and fewer bits per parameter. They are much more impressive and will run on garden variety gaming PCs at more than usable speed. I can't wait to try on my 4090 at home. There's basically no reason to run other open source models now that these are available, at least for non-multimodal tasks.
- tedivm 1y agoQwen3 has multiple variants ranging from larger (230B) than these models to significantly smaller (0.6b), with a huge number of options in between. For each of those models they also release quantized versions (your "fewer bits per parameter). I'm still withholding judgement until I see benchmarks, but every point you tried to make regarding model size and parameter size is wrong. Qwen has more variety on every level, and performs extremely well. That's before getting into the MoE variants of the models.
- modeless 1y agoThe benchmarks of the OpenAI models are comparable to the largest variants of other open models. The smaller variants of other open models are much worse.
- mrbungie 1y agoI would wait for neutral benchmarks before making any conclusions.
- bigyabai 1y agoWith all due respect, you need to actually test out Qwen3 2507 or GLM 4.5 before making these sorts of claims. Both of them are comparable to OpenAI's largest models and even bench favorably to Deepseek and Opus: https://cdn-uploads.huggingface.co/production/uploads/62430a8522549d0917bfeb5a/0d7zztq4GB7G2ZYowO-dQ.jpeg https://cdn-uploads.huggingface.co/production/uploads/62430a... It's cool to see OpenAI throw their hat in the ring, but you're smoking straight hopium if you think there's "no reason to run other open source models now" in earnest. If OpenAI never released these models, the state-of-the-art would not look significantly different for local LLMs. This is almost a nothingburger if not for the simple novelty of OpenAI releasing an Open AI for once in their life.
- modeless 1y ago> Both of them are comparable to OpenAI's largest models and even bench favorably to Deepseek and Opus So are/do the new OpenAI models, except they're much smaller.
- UrineSqueegee 1y agoI'd really wait for additional neutral benchmarks, I asked the 20b model on low reasoning effort which number is larger 9.9 or 9.11 and it got it wrong. Qwen-0.6b gets it right.
- bigyabai 1y agoAccording to the early benchmarks, it's looking like you're just flat-out wrong: https://blog.brokk.ai/a-first-look-at-gpt-oss-120bs-coding-ability/ https://blog.brokk.ai/a-first-look-at-gpt-oss-120bs-coding-a...
- sourcecodeplz 1y agoFrom my initial web developer test on https://www.gpt-oss.com/ https://www.gpt-oss.com/ the 120b is kind of meh. Even qwen3-coder 30b-a3b is better. have to test more.
- thegeomaster 1y agoThey have worse scores than recent open source releases on a number of agentic and coding benchmarks, so if absolute quality is what you're after and not just cost/efficiency, you'd probably still be running those models. Let's not forget, this is a thinking model that has a significantly worse scores on Aider-Polyglot than the non-thinking Qwen3-235B-A22B-Instruct-2507, a worse TAUBench score than the smaller GLM-4.5 Air, and a worse SWE-Bench verified score than the (3x the size) GLM-4.5. So the results, at least in terms of benchmarks, are not really clear-cut. From a vibes perspective, the non-reasoners Kimi-K2-Instruct and the aforementioned non-thinking Qwen3 235B are much better at frontend design. (Tested privately, but fully expecting DesignArena to back me up in the following weeks.) OpenAI has delivered something astonishing for the size, for sure. But your claim is just an exaggeration. And OpenAI have, unsurprisingly, highlighted only the benchmarks where they do _really_ well.
- moralestapia 1y agoYou can always get your $0 back.
- Imustaskforhelp 1y agoI have never agreed with a comment so much but we are all addicted to open source models now.
- recursive 1y agoNot all of us. I've yet to get much use out of any of the models. This may be a personal failing. But still.
- satvikpendem 1y agoDepends on how much you paid for the hardware to run em on
- cvadict 1y agoYes, but they are suuuuper safe. /s So far I have mixed impressions, but they do indeed seem noticeably weaker than comparably-sized Qwen3 / GLM4.5 models. Part of the reason may be that the oai models do appear to be much more lobotomized than their Chinese counterparts (which are surprisingly uncensored). There's research showing that "aligning" a model makes it dumber.
- xwolfi 1y agoThe censorship here in China is only about public discussions / spaces. You cannot like have a website telling you about the crimes of the party. But downloading some compressed matrix re-spouting the said crimes, nobody gives a damn. We seem to censor organized large scale complaints and viral mind virii, but we never quite forbid people at home to read some generated knowledge from an obscure hard to use software.
- echelon 1y agoThis might mean there's no moat for anything. Kind of a P=NP, but for software deliverability.
- CamperBob2 1y agoOn the subject of who has a moat and who doesn't, it's interesting to look the role of patents in the early development of wireless technology. There was WWI, and there was WWII, but the players in the nascent radio industry had serious beef with each other. I imagine the same conflicts will ramp up over the next few years, especially once the silly money starts to dry up.
- a_wild_dandan 1y agoRight? I still remember the safety outrage of releasing Llama. Now? My 96 GB of (V)RAM MacBook will be running a 120B parameter frontier lab model. So excited to get my hands on the MLX quants and see how it feels compared to GLM-4.5-air.
- 4b6442477b1280b 1y agoin that era, OpenAI and Anthropic were still deluding themselves into thinking they would be the "stewards" of generative AI, and the last US administration was very keen on regoolating everything under the sun, so "safety" was just an angle for regulatory capture. God bless China.
- narrator 1y agoYeah, China is e/acc. Nice cheap solar panels too. Thanks China. The problem is their ominous policies like not allowing almost any immigration, and their domestic Han Supremacist propaganda, and all that make it look a bit like this might be Han Supremacy e/acc. Is it better than wester/decel? Hard to say, but at least the western/decel people are now starting to talk about building power plants, at least for datacenters, and things like that instead of demanding whole branches of computer science be classified, as they were threatening to Marc Andreessen when he visited the Biden admin last year.
- 01HNNWZ0MV43FF 1y agoI wish we had voter support for a hydrocarbon tax, though. It would level out the prices and then the AI companies can decide whether they want to pay double to burn pollutants or invest in solar and wind and batteries
- AtlasBarfed 1y agoOh poor oppressed marc andreesen. Someone save him!
- a_wild_dandan 1y agoOh absolutely, AI labs certainly talk their books, including any safety angles. The controversy/outrage extended far beyond those incentivized companies too. Many people had good faith worries about Llama. Open-weight models are now vastly more powerful than Llama-1, yet the sky hasn't fallen. It's just fascinating to me how apocalyptic people are. I just feel lucky to be around in what's likely the most important decade in human history. Shit odds on that, so I'm basically a lotto winner. Wild times.
- bogtog 1y agoWhen people talk about running a (quantized) medium-sized model on a Mac Mini, what types of latency and throughput times are they talking about? Do they mean like 5 tokens per second or at an actually usable speed?
- deleted 1y ago[deleted]
- n42 1y agohere's a quick recording from the 20b model on my 128GB M4 Max MBP: https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr and the 120b: https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM I am, um, floored
- Davidzheng 1y agothe active param count is low so it should be fast.
- Rhubarrbb 1y agoGeneration is usually fast, but prompt processing is the main limitation with local agents. I also have a 128 GB M4 Max. How is the prompt processing on long prompts? processing the system prompt for Goose always takes quite a while for me. I haven't been able to download the 120B yet, but I'm looking to switch to either that or the GLM-4.5-Air for my main driver.
- anonymoushn 1y agoit's odd that the result of this processing cannot be cached.
- lostmsu 1y agoIt can be and it is by most good processing frameworks.
- 1y ago
- tyho 1y agoWhat's the easiest way to get these local models browsing the web right now?
- dizhn 1y agoaider uses Playwright. I don't know what everybody is using but that's a good starting point.
- keheliya 1y agoUsing LM-Studio as the frontend and Playwright-powered MCP tools for browser access. I've had success with one such MCP: https://github.com/instavm/coderunner https://github.com/instavm/coderunner It has a tool called navigate_and_get_all_visible_text, for example. Here's a demo of this functionality: https://www.youtube.com/watch?v=9mLrGcuDifo https://www.youtube.com/watch?v=9mLrGcuDifo
- Imustaskforhelp 1y agoOkay I will be honest, I was so hyped up about This model but then I went to localllama and saw it that the: 120 B model is worse at coding compared to qwen 3 coder and glm45 air and even grok 3... (https://www.reddit.com/r/LocalLLaMA/comments/1mig58x/gptoss120b_below_glm45air_and_qwen_3_coder_at/ https://www.reddit.com/r/LocalLLaMA/comments/1mig58x/gptoss1...)
- logicchains 1y agoIt's only got around 5 billion active parameters; it'd be a miracle if it was competitive at coding with SOTA models that have significantly more.
- jph00 1y agoOn this bench it underperforms vs glm-4.5-air, which is an MoE with fewer total params but more active params.
- ascorbic 1y agoThat's SVGBench, which is a useful benchmark but isn't much of a test of general coding
- Imustaskforhelp 1y agoHm alright, I will see how this model actually plays around instead of forming quick opinions.. Thanks.
- pxc 1y agoQwen3 Coder is 4x its size! Grok 3 is over 22x its size! What does the resource usage look like for GLM 4.5 Air? Is that benchmark in FP16? GPT-OSS-120B will be using between 1/4 and 1/2 the VRAM that GLM-4.5 Air does, right? It seems like a good showing to me, even though Qwen3 Coder and GLM 4.5 Air might be preferable for some use cases.
- larodi 1y agoWe be running them in PIs off spare juice in no time, and they be billions given how chips and embedded spreads…