2 ms·
Aren't there a bunch of models that run OK on consumer hardware now?
by procaryote 11mo ago
Aren't there a bunch of models that run OK on consumer hardware now?
- lukan 11mo agoHopefully my new GPU will arrive tomorrow, then I can confirm myself, but if you look around online, there are lots of private people out there running their own models. A 16 GB GPU starts at 270€, which lets you run something like deepseek r.14, 32 GB GPUs start at 1200 € and then it goes further up, in model quality and price. (Top models require something like 60- 200 GB of GPU memory I think) So for sure any medium sized company could afford to run their own LLMs, also at scale if they want to make the investment. The question is, how much they value their confidential data. (I would not trust any of the big AI companies). And you don't usually need cutting edge reasoning and coding abilities to process basic information.
- FuckButtons 10mo agoA 120gb ram MacBook Pro will run gpt-oss-120b at a very respectable clip and I’ve found it to be quite serviceable for a lot of tasks.
- adastra22 10mo agoI bought one for this purpose, but LM Studio doesn't seem to want to run even the most quantized versions. Any suggestions?
- FuckButtons 10mo agoAre you using the mix quantized versions? Also, there’s a setting that disables the memory allocation guardrails they put in place, which are irrelevant since macOS handles oom quite gracefully.