5 ms·
While 192GB of ram is appealing, it's also quite expensive at $6000. For that price I rather buy a system with 5 used 3090s, which while being "only" 120GB of V
by nodja 2y ago
While 192GB of ram is appealing, it's also quite expensive at $6000. For that price I rather buy a system with 5 used 3090s, which while being "only" 120GB of VRAM, you benefit from much faster tokens/s and prompt processing speed (the macs are notoriously slow at consuming large contexts).
- miohtama 2y ago5x 3090 is also much more power hungry?
- karamanolev 2y agoFor personal usage, does it matter though? In most places residential electricity is cheap compared to everything else. In a DC context I feel it matters a lot more compared to the capex.
- thenickdude 2y ago1x 3090 (350W power limit) already makes it feel like I'm running a fan heater under my desk, 5x would be nuts.
- TeMPOraL 2y agoPlace and time your use right, and you'll save a bit on heating at winter and/or at nights.
- heelix 2y agoI think the last time any of my computers had a case was back when I realized the pair of 900gx2 cards I was running was turning my computer into an easy bake.
- ryao 2y agoWhen running inference workloads via something like llama.cpp, only 1 GPU is ever used at a time, so you would have 1 active GPU and 4 idle GPUs. That should make the power usage less insane in practice than you expect.
- orf 2y agoCan I use that on the train though? I can with a 128GB MacBook, without it sounding like a helicopter taking off as well.
- zem 2y agoyou don't need to take ai training quite so literally (:
- jchw 2y agoHonestly, if you have a residence of some kind and an Internet connection, you don't need to bring your beefy computer with you everywhere. It is cool to be able to have ridiculously powerful mobile computers, but I don't think I would ever be willing to take a $6,000 laptop anywhere it has a decent chance of being stolen.
- orf 2y agoDo you live in a third world country? If so I might agree, but otherwise trains are perfectly safe.
- jchw 2y agoI am very happy for you, but laptops get stolen in public in most countries.
- orf 2y agoLaptops get stolen on a train? An enclosed, single-direction space that only occasionally allows you to exit between infrequent, long-distance stops? A thing that contains ticket inspectors and a literal guard? How many laptops have you personally seen be stolen on a train?
- rob_c 2y agoPeople stabbed maybe, but that tends to be more sports related than laptop related. (Yes on a national line(!))
- danielhanchen 2y agoThe good thing is since MoEs are mainly memory bound, we just need (VRAM + RAM) to be in the range of 80GB or so in my tests for at least 5 tokens or so /s. It's better to get (VRAM + RAM) >= 140GB for at least 30 to 40 tokens/s, and if VRAM >= 140GB, then it can approach 140 tokens/s! Another trick is to accept more than 8 experts per pass - it'll be slower, but might be more accurate. You could even try reducing the # of experts to say 6 or 7 for low FLOP machines!
- yobid20 2y agoThe power requirement for 5x5090s is 10x higher , so you'll spend far more than $6000 in electricity over time.
- deleted 2y ago[deleted]
- xbmcuser 2y agoI think just getting nvidia Project Digits might be the best option. A lot of people when it was announced were underwhelmed. But I think now it could be just the thing for people making their own ai home servers. https://www.nvidia.com/en-us/project-digits/ https://www.nvidia.com/en-us/project-digits/
- sourcecodeplz 2y agoYep, I think the same. With 128GB fast memory one could run this.
- phkahler 2y ago>> While 192GB of ram is appealing, it's also quite expensive at $6000. That's because it's Apple. It time to start moving to AMD systems with shared memory. My Zen 3 APU system has 64GB these days and its a mini ITX board.
- Maakuth 2y agoWhat is the performance in ML workloads like on AMD APUs compared to Apple Silicon?