4 ms·
I can (barely, but sustainably) run Q3.5 397B on my Mac Studio with 256GB unified. It cost $10,000 but that's well within reach for most people who are here, I
by kridsdale3 6mo ago
I can (barely, but sustainably) run Q3.5 397B on my Mac Studio with 256GB unified. It cost $10,000 but that's well within reach for most people who are here, I expect.
- toxik 6mo ago$10k is well outside my budget for frivolous computer purchases.
- bdangubic 6mo ago99.97% of HN users are nodding… :)
- hparadiz 6mo agoThere are way too many good uses of these models for local that I fully expect a standard workstation 10 years from now to start at 128GB of RAM and have at least a workstation inference device.
- bdangubic 6mo agoor if you believe a lot of HN crowd we are in AI bubble and in 10 years inference will be dirt cheap when all of this crashes and we have all this hardware in data centers and it won't make any sense to run monster workstations at home (I work 128GB M4 but not run inference, just too many electron apps running at the same time...) :)
- hparadiz 6mo agoInference will be dirt cheap for things like coding but you'll want much more compute for architectural planning, personal assistants with persistent real time "thinking / memory", as well as real time multimedia. I could put 10 M4s to work right now and it won't be enough for what I've been cooking.
- bigiain 6mo ago> I work 128GB M4 but not run inference, just too many electron apps running at the same time. This is somewhat depressing - needing a couple of thousand bucks worth of ram just to run your chat app and code/text editor and API doco tool and forum app and notetaking app all at the same time...
- mananaysiempre 6mo agoCrucial (Micron) sold 128GB of DDR5-5600 in SODIMM form for $280 a year ago. It would be slower tham the same amount on an M4 Mac, but still, I object to characterizing either as “a couple thousand bucks worth”.
- bigiain 6mo agoI( get that number by optioning up a Mac Studio to 128GB at the Apple Store. (Admittedly, Apple should be facing criminal price gouging law suits for their ram pricing.)
- fragmede 6mo agoThat's kind of a specific percentage. What numbers did you use to get there?
- stefs 6mo agoyeah, but if you really really wanted to and/or your livelyhood depended on it, you probably could afford it.
- zozbot234 6mo agoIt would be plenty in-budget if the software part of local AI was a bit more full-featured than it is at present. I want stuff like SSD offload for cold expert weights and/or for saved/cached KV-context, dynamic context sizing, NPU use for prefill, distributed inference over the network, etc. etc. to all be things that just work for most users, without them having to set anything up in an overly error-prone way. The system should not just explode when someone tries to run something slightly larger; it should undergo graceful degradation and let them figure out where the reasonable limits are.
- fragmede 6mo agoJust have to reclassify it as non-frivolous then. $10k's not a lot for something as important as a car, if you live somewhere where one is required. Housing is typically gonna cost you more than $10k to own. I probably spend close to $10k for food for 1.5 years. So if you just huff enough of the AI Kool aid, you too can own a Mac Studio. Or an M5 MacBook. Or a dual 3090 rig.
- rwmj 6mo agoBut it's well within the budget of a small company that wants to run a model locally. There are plenty of reasons to run one locally even if it's not state of the art, such as for privacy, being able to do unlimited local experiments, or refining it to solve niche problems.
- SlavikCA 6mo agoI'm running it on my Intel Xeon W5 with 256GB of DDR5 and Nvidia 72GB VRAM. Paid $7-8k for this system. Probably cost twice as much now. Using UD-IQ4_NL quants. Getting 13 t/s. Using it with thinking disabled.
- GrayShade 6mo agoI get 20 t/s on the UD-Q6_K_XL quant, Radeon 6800 XT.
- qlm 6mo agoHacker News moment
- rwmj 6mo agoFor some reason you were being downvoted but I enjoy hearing how people are running open weights models at home (NOT in the cloud), and what kind of hardware they need, even if it's out of my price range.
- kylehotchkiss 6mo agoyou have proved my point
- lpnam0201 6mo agoIn where I am living, 10k USD is a little more than 3 years worth of rent, for a relatively new and convenient 2 bedroom apartment.
- gbgarbeb 6mo ago$277 a month for a two bedroom is literally 6-10% of what someone in the SF Bagholder Area pays. Either you're in Africa, southeast Asia or south/central Amarica. How do you even afford internet?