3 ms·
Interesting, so this is a Mac-specific solution? That's pretty cool. I assume, then, that the primary goal would be to drop in the beefiest GPU possible when o
by kulahan 2y ago
Interesting, so this is a Mac-specific solution? That's pretty cool.
I assume, then, that the primary goal would be to drop in the beefiest GPU possible when on windows/linux?
- lysace 2y agoWith Windows/Linux I think the issue is that NVidia is artificially limiting the amount of onboard RAM (they want to sell those devices for 10x more to openai, etc) and that AMD for whatever reason can't get their shit together. I'm sure that there are other much more knowledgeable people here though, on this topic.
- rustcleaner 2y agoThis is why the DMCA must be repealed.
- evilduck 2y agoThere's nothing Mac specific about running LLMs locally, they just happen to be a convenient way to get a ton of VRAM in a single small power efficient package. In Windows and Linux, yes you'll want at least 12GB of VRAM to have much of any utility but the beefiest consumer GPUs are still topping out at 24GB which is still pretty limiting.