6 ms·
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. I am running Qwen3.6-35B-A3B locally on my 16GB mac
by dwa3592 3mo ago
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
- embedding-shape 3mo ago> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat https://github.com/deepanwadhwa/samosa-chat It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;) Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
- dwa3592 3mo ago>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that that would have run into a race condition unfortunately ;) but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
- embedding-shape 3mo ago> that would have run into a race condition unfortunately ;) Not really, you start small, bootstrap as soon as you can, and off you go. Requires a good model though ;)
- smith7018 3mo agoNothing says they're using Qwen for local development. They could be using it to for conversations, knowledge, or "creative writing."
- embedding-shape 3mo ago> Nothing says they're using Qwen for local development I know! That's my point! You're a poor salesman of a coding environment/tool if you don't even use it yourself for coding...
- reidrac 3mo ago> on a decent speed But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
- dwa3592 3mo agoi am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
- jckahn 3mo agoIgnore the haters. What you've done is incredible!
- Aurornis 3mo agoIf you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri It says so right in the readme. They’re not hiding anything.
- bigyabai 3mo agoYep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac. Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
- LeifCarrotson 3mo agoI think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare! https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done. Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
- deleted 3mo ago[deleted]
- rvz 3mo ago> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. That is no where near decent at all.
- dwa3592 3mo agoit's a 16GB machine. i am proud of this machine so far.
- haute_cuisine 3mo agoHow are the thermals? I noticed that running any serious workload locally heats system fast.
- dwa3592 3mo agoi have been optimizing for that. for now samosa is capped at using half of the avaiable cores and switching between them, which keeps the system 'less hot' as it would have been. i will also release better thermal control in the next release. at this point its basically sacrificing about 20% of the speed to keep the hardware less stressed (and hot).
- Capricorn2481 3mo agoI tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.
- Azantys 3mo agoYou are comparing a 35B models to a 635B+ frontier model, of course thats not even close
- Capricorn2481 3mo agoI'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.
- IsTom 3mo agoTo be fair, they don't have the text editor highlighting all the matching parens. I'd be lost too.
- Capricorn2481 3mo agoYeah prediction models and many parentheses are probably not a good combination, but we're not talking about anything exceptionally complicated here. I have had syntax issues in Python as well.
- Azantys 3mo agoIt really depends on the language, popular languages work pretty good
- segmondy 3mo agotry q8, check your parameters. qwen3.6-35b-a3b should definitely be able to do so with no issues at all.
- 3mo ago
- smeeth 3mo agoProbably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one. It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
- dwa3592 3mo agoagreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
- dTal 3mo ago>in case they start to ban open source AI development Is that likely, do you think?
- dwa3592 3mo agodario has been saying open source models are dangerous. who knows who is listening to him.
- mdp2021 3mo agoThe one time in which I saw Juergen Schmidhuber in perfect nervous control, "coolness" they may say westward, was when he replied to one member of the audience, "The same observation was made when they invented fire: oh, it's dangerous. But in the end, now it's here (shrugh)". There is a proposal in the USA to restrict LLM access. This will only have us depend more and more on open source models and their providers. And cause a drain of research in those areas in which it will be impeded.
- 3mo ago
- notnullorvoid 3mo agoI mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).
- Aurornis 3mo ago> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware. This prediction alone isn’t useful at all without a bound on speed and maybe quantization. You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded. If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it. > This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air. Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses. You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe. You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token. Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
- dwa3592 3mo ago>>Yes, it’s technically running, but not in a way that would be useful by normal LLM standards. What are the LLM standards? Do you know how many people use perplexity? I know many people who are not software engineers or tech workers and have a LLM subscription for rewriting their stuff (non-native english speakers) in english. There are many use cases for running good models locally. Maybe not for you, but someone might find this beneficial.
- Aurornis 3mo agoI have a free perplexity account from some promotion. Not sure what comparison you’re trying to make because Perplexity’s whole thing is that it’s really fast. It launches the search with parallel agents and then even seems to render some of the output paragraphs with parallel sessions to get the results. Doing the same thing at 7-9 tokens per second, concurrency of 1, would take ages for all of the tool calling and subsequent processing. It wouldn’t compare in any meaningful way, because perplexity delivers instant results. That’s what I meant by modern standards of LLM usefulness.
- typon 3mo agoBy early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s
- kennywinker 3mo agoUnless there are major improvements to how much hardware it takes to run a 1T model, this is deeply unrealistic. First because why release hardware that puts your biggest customers (data centers) out of business. Second because as I understand it the data centers have bought up all the high end chip production capacity for at least the next year and unless the bubble pops that'll continue for a while.
- iwontberude 3mo agoBecause for the company that will actually do it, their biggest customers aren’t data centers they are iPhone owners.
- kennywinker 3mo agoFirst off the math doesn’t math. Datacenters are willing to pay $50k for a single high end GPU. If you have unlimited capacity, yeah sell millions for $100 a pop or $10 a pop or whatever the bom cost of a phone GPU would be - but if you have limited capacity, you’re gonna sell all of that to the customer who is willing to pay the most PER UNIT. Second off, this doesn’t work from a power consumption standpoint. When I run qwen3.6-35b, a far smaller model than op is suggesting, power usage spikes to 150-200W during inference. To fit a 1T model in the palm of my hand, the amount of processing required doesn’t fit the amount of power available. Now I’m not saying this will never happen - there are some great leads, e.g. burning models directly on to a chip - but op’s scenario is definitely not happening in two years. Maybe 5, a lot more likely 10, unless of course local ai is made illegal
- andrekandre 3mo ago> Datacenters are willing to pay $50k for a single high end GPU. its true for now, because capital is flowing like a torrent, but how long will that last if returns start to be expected (aka the bubble pops)?
- TYPE_FASTER 3mo agoDownloading now just 'cause the repo name
- deleted 3mo ago[deleted]
- felineflock 3mo agoThat is awesome! I am curious about the decision to not use GPU since this is for Apple Silicon. Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?
- dwa3592 3mo agoIt will, but the process at this point is SSD bound rather than compute bound. On a bigger machine, Apple silicon must help but I don't have a bigger machine. I can think about this more and will make changes if that helps.
- deleted 3mo ago[deleted]
- razster 3mo agoI sense the same. Running Ornith 35b with Pi and hitting 50+toks, and now that I learned Pi can do search and fetch, I have not needed to visit any of the large models as a search function. Soon we will be seeing new models drop this month and next that will change some of the landscape. Exciting times. P.s. Try Ornith, well worth it.
- cyanydeez 3mo agosetup deer-flow: https://github.com/bytedance/deer-flow https://github.com/bytedance/deer-flow its pretty good for building any kind of research report using search engines and it manages context corrextly.
- deleted 3mo ago[deleted]
- deleted 3mo ago[deleted]
- VladVladikoff 3mo agoIsn’t 9 tok/s unbearable? I frequently see Claude sessions hit 1Mtok in less than a day. It seems that 9tok/s would be really slow for actual work.
- driverdan 3mo agoYou can't use it real time but you can have it run in the background and come back to it.
- mseri 3mo agoNice! I had done the exact same with gemma4 26b, both for my Intel laptop and for my M1 with 8Gb RAM (with also q4 and turboquant). I don’t use it much since there are dumber but way faster models to run, but I should clean up the code and make it available
- mseri 3mo agoOk, done: https://github.com/mseri/zunzuncito https://github.com/mseri/zunzuncito My main focus is systems with very low ram. On my M1 with 8Gb or RAM llama.cpp cannot run gemma4 26b, but this implementation works fine with 5-6 tok/s. I also replaced my crap implementation of the oai server following yours, I hope it's all right. I did add a mention to your project and this fact in the README. It is a lot more barebones than what you have, but I have to admit that it works really nicely for me
- dwa3592 3mo agoglad to know. thanks :)
- javier2 3mo agoHow do I install without curl-piping to bash?
- damienmeur 3mo agoJust wake me u when I can run fable or gpt 5.6 for free my raspberry-pi 4 at 2000 tokens/s
- minkeymaniac 3mo agoI just downloaded llama. ran this llama-cli -hf ggml-org/gemma-3-1b-it-GGUF I am getting [ Prompt: 91.3 t/s | Generation: 171.8 t/s ] This is on a GPU (RTX 4060) Is this decent?