4 ms·
Considering that the topic is approachability and energy efficiency, that Mac Studio will do reasonably fast inference while consuming <200W at full load. The
by Lalabadie 3y ago
Considering that the topic is approachability and energy efficiency, that Mac Studio will do reasonably fast inference while consuming <200W at full load.
The speed is certainly not comparable to dedicated GPUs, but the power efficiency is ridiculous for a very usable speed and no hardware setup.
- Applejinx 3y agoThis, and then you get to have a Mac Studio. I have one, where I selected an M1 Ultra and 128G RAM to facilitate just this sort of thing. But in practice, I'm spending much more time using it to edit 4K video, and as a recording studio/to develop audio plugins on, and to livestream while doing these things. Turns out it's good at these things, and since I have the LLAMA 70b language model at home and can run it directly unquantized (not at blinding speed, of course, but it'll run just fine), I'm naturally interested in learning how to fine tune it :)
- int_19h 3y agoYep, I also got mine specifically for LLMs and ended up using it as a second desktop for other things; actually strongly considering making it my primary at this point. I still wouldn't recommend it to someone just looking for a powerful desktop, just because $3K is way overpriced for what you get (non-replaceable 1Tb SSD is so Apple!). But it's certainly great if you already have it...