5 ms·
Is the output coherent though? I am yet to see a local model working on consumer grade hardware being actually useful.
by bossyTeacher 6mo ago
Is the output coherent though? I am yet to see a local model working on consumer grade hardware being actually useful.
- lrvick 6mo agoI run qwen3.5 122b on a Framework Desktop at 35/ts as a daily driver doing security and OS systems and software engineering. Never paid an LLM provider and I have no reason to ever start.
- mixermachine 6mo agoWhat spec of Framework Desktop do you run this on?
- breisa 6mo agoThere is only one and for this model you need the one with 128GiB RAM.
- the_pwner224 6mo agoIf you're looking to buy new hardware, also consider the Asus Rog Flow Z13. It has the same chip as the Framework desktop and is ~20% cheaper ($2,700) for the 128 GB spec while coming in a tablet/laptop form factor. It's capped at a slightly lower power but Strix Halo scales down very well in TDP - I never even use the max power mode on my Z13 because you don't really get any extra perf. The only downside is that I suspect the Framework would be a decent bit quieter under load (not that this thing is abnormally loud). As well as you're limited to a single M.2 2230 internal SSD slot in this (I believe Micron recently launched a 4 TB model, but generally you'll max out at 2 TB without using an external enclosure). I don't have anything against the Framework, I'm sure it's a great machine, but the Z13 is an incredible portable all-in-one device that can handle everything from general PC use to gaming to tablet/entertainment to LLMs & high perf.
- lrvick 6mo agoThe main reason to get the framework is the DIY edition for the mini-itx form factor board and support. If you do not care about those then any cheap proprietary format box with the Strix Halo chip will do. I put my boards in mini itx rack mounts personally so framework is the only option.
- a_paddy 6mo agoI can try it for you
- jfoster 6mo agoIt can write (some) code that works. Just roughly guessing from my use, but I think of it as being a bit like ChatGPT circa-2024 in terms of capability & speed. Disappointing if you compare it to anything else from 2026, but fairly impressive for something that can run locally at an OK speed.
- fsiefken 6mo agoQwen3.5-9b and Qwen3.5-27b are pretty coherent on my 24G android phone
- dpacmittal 6mo agoWhich android phone has 24G?
- fsiefken 6mo agoFor example: ASUS ROG Phone 9 Pro, Nubia RedMagic 10 Pro, OnePlus 13, Realme GT 5
- logicallee 6mo agoIt's highly coherent (see my other comment for an example of its text output) and yes, it's useful. I am starting to use Gemma 4:e4b as my daily driver for simple commands it definitely knows, things that are too simple to use ChatGPT for. It is also able to code through moderately difficult coding tasks. If you want to see it in action, I posted a video about it here[1] (the 10 GB one is at the 2 minute mark and the 20 GB one says hello at 5 minutes 45 seconds into the video.) You can see its speed and output on simple consumer grade hardware, in this case a Mac Mini M4 with 24 GB of RAM. [1] https://youtube.com/live/G5OVcKO70ns https://youtube.com/live/G5OVcKO70ns
- jeroenhd 6mo agoGoogle's models work quite well on my Android phone. I haven't found a use case beyond generating shitposts, but the model does its job pretty well. It's not exactly ChatGPT, but minor things like "alter the tone of this email to make it more professional" work like a charm. You need a relatively beefy phone to run this stuff on large amounts of text, though, and you can't have every app run it because your battery wouldn't last more than an hour. I think the real use case for apps is more like going to be something like tiny, purpose-trained models, like the 270M models Google wants people to train and use: https://developers.googleblog.com/on-device-function-calling-in-google-ai-edge-gallery/ https://developers.googleblog.com/on-device-function-calling... With these things, you can set up somewhat intelligent situational automation without having to work out logic trees and edge cases beforehand.
- the_pwner224 6mo agoI have a 128 GB Strix Halo tablet (same as the other commenter here with the Framework Desktop). I'm using the larger Gemma 4 26B-A4B model (only 28 GB @ Q8) and it's been working great and runs very fast. It's a 100% replacement for free ChatGPT/Gemini. Compared to the paid pro/thinking models... Gemma does have reasoning, and I have used the reasoning mode for some tax & legal/accounting advice recently as well as other misc problems. It's worked well for that, but I haven't tried any real difficult tasks. From what I've heard re. agentic coding, the open weight models are ~18-24 months behind Anthropic & Google's SOTA. Qwen 3.5 122B-A10B should just fit into 128 GB with a Q4/5 and may be a bit smarter. There's apparently also a similar sized Gemma 4 model but they haven't released it yet, the 26B was the largest released.
- zozbot234 6mo agoThere's a 31B dense model in the Gemma 4 series that's obviously going to be smarter (though a whole lot slower) than the MoE 26A4B.
- the_pwner224 6mo agoI tried it and it was unusably slow at ~5-6 TPS. 26A4B gets close to 40 TPS which is faster than you can read, and still pretty quick with reasoning enabled.
- veunes 6mo agoSure, 26B models on beefy desktop silicon are finally nipping at the heels of commercial APIs, but this is a mobile thread. On a phone with 8GB of RAM and passive cooling, your tokens per second (t/s) are going to fall off a cliff after the first minute of sustained compute
- RobMurray 6mo agoI'm working on a visual description app for the blind. even Gemma 4 E2B can give very useful image descriptions while at the same time taking questions as audio. It's also much faster than most of the current popular cloud based apps like Be My Eyes.