3 ms·
his 128gb Ram laptop is quite extreme
by AgentMasterRace 2mo ago
his 128gb Ram laptop is quite extreme
- simonw 2mo agoIt should just about be usable in 32GB.
- npodbielski 2mo agoIt is. I am running it on R9700
- krzyk 2mo agoOn a consumer hardware it would be nicer. With no GPU/iGPU or a 6-8GB VRAM.
- bakraman 2mo agoRAM is never the issue, it's always the compute power
- spider-mario 2mo agoRAM is not “never” the issue. My iPhone and MacBook Air could both run larger and more capable models if they had more RAM.
- geek_at 2mo agoand memory bandwidth
- tuetuopay 2mo agoQuite the opposite, RAM is always the issue. More specifically, high bandwidth RAM.
- mhaberl 2mo agowhat??? not true! for inference the compute is the last thing we need more of. memory bandwidth is the numebr one blocker, after that the inefficiencies that where introduced with MoE models (and all new large models are made that way) Here is a quick read: https://news.ycombinator.com/item?id=49324600 https://news.ycombinator.com/item?id=49324600
- CamouflagedKiwi 2mo agoIt's absolutely not for these models. There are plenty of consumer GPUs out there with 8 or 12GB VRAM - they are comparatively very fast at inference but just aren't big enough to run lots of the models you want. Also context management is a massive pain.
- DanielHB 2mo agoI run qwen3.5-9B on an RTX 3080 with 10GB of vram. It runs at ~77tk/s with around 50k context size. As soon as I switch to a model that doesn't fully fit into vram it tanks to <10tk/s which makes it unusable for me for most tasks.
- piva00 2mo agoRAM bandwidth is the main issue for running LLMs on consumer hardware...
- aizk 2mo agoGive it 6 months, the capabilities will increase even further.
- deleted 2mo ago[deleted]