4 ms·

Are we at or approaching the point where people can self host an useful LLM for personal use on affordable hardware?

by newsclues 3y ago

adastra22 3y ago
Is a MacBook Air affordable? Because we’re already at that point.
- newsclues 3y ago
  The base model? Given the article about GPUs, I think I was thinking more along the lines of a desktop with a consumer GPU.
  - brucethemoose2 3y ago
    Intel (and AMD) reportedly have big M-Pro-like iGPUs in the pipe, and Intel has the Battlemage GPU line coming up, so we shall see.
  - adastra22 3y ago
    By base model do you mean off-the-shelf? I think the bigger limitation there would be RAM, as the machine comes with just 8GB soldered on board. You can expand to 24GB for +$400 though, and increase the GPU cores +25% for an extra +$100. The CPU/GPU speed of the Air is the same as the MacBook Pro base model though. The only difference is the lack of active cooling, which for large workloads can result in performance degradation. Chat apps are intrinsically interactive though, only using bursts of GPU when it is performing inference. Shouldn't be an issue. The expensive PRO/MAX variants of the MacBook Pro would be 2x - 4x faster. But the plain M2 in the MacBook Air is sufficient to get real-time speeds on sizable models. When the M3 comes out for the MacBook Air, discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference.
    - washadjeffmad 3y ago
      > discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference At the point of purchase of the lowest cost configuration with 24GB Unified Memory, you've already paid the an equivalent of over 2200 hours of GPU compute time on an RTX 4090 24GB, with a performance that exceeds the MacBook by around 1200% (it/s). If you buy that MacBook for AI, you would have to run continuous generative inference on it for over a decade to match the return of just having used runpod instead. Even Apple doesn't use Macs to do AI - they use GCP TPUs. Buy the Mac if you like it, by all means, but be realistic. LLM performance on Apple Silicon is decent for small models but does not scale. If any Mac architecture were cost effective at any scale for AI, we would be putting them in data centers, and we're not.
      - adastra22 3y ago
        This sub thread isn’t about hosted inference though. I interpreted the question as “is there a reasonable laptop out there that can run large models locally in real time?” And a MacBook Air does fit the bill for that. It’s an entry-level Mac, and it is capable of doing inference jobs. If you have around $1k available to spend on a daily driver laptop, but you also want to do some AI inference experiments, the MacBook Air is what I’d recommend. The AMD Framework laptop might be a good alternative. But I don’t know how well its integrated GPU is supported by llama.cpp.
brucethemoose2 3y ago
I run Yi 34B at 45K-70K context as a GPT 3.5 replacement on a Nvidia 3090. Its quite smart, and fast. The whole rig cost me $2.1K, but it could have easily been $1.5K without splurging on certain parts like I did. And its 10 Liters, small enough to move around.
- washadjeffmad 3y ago
  Same, but mine is 11L (A4-H2O). Which case did you get?!
  - brucethemoose2 3y ago
    A Node 202! I wanted something flat enough for a suitcase. https://www.reddit.com/r/sffpc/comments/18a7mal/ducted_3090_ftw3_stuffed_in_a_node_202/ https://www.reddit.com/r/sffpc/comments/18a7mal/ducted_3090_...
montebicyclelo 3y ago
Consumer laptops, such as apple silicon macbooks with just 32gb ram, can run fairly large models, such as Mixtral-8x7b, an open source model that's comparable to gpt-3.5, well enough for interactive chat, (and seems likely models of this size will keep improving) - ofc, 64gb ram preferable, because it unlocks larger models. And 3090s are the other way to to go, which are faster but not quite as much vram, though you can use multiple.
tudorw 3y ago
I'm using GPT4ALL with Mistral on a $400 Intel NUC, so yes, if the use fits!
Const-me 3y ago
If you use Windows and you have a discrete GPU from any vendor with at least 6GB VRAM, you could test my implementation of Mistral model: https://github.com/Const-me/Cgml https://github.com/Const-me/Cgml Screenshot: https://github.com/Const-me/Cgml/blob/master/Mistral/MistralChat/screenshot.png https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...

Are we at or approaching the point where people can self host an useful LLM for personal use on affordable hardware?

adastra22 3y ago

Is a MacBook Air affordable? Because we’re already at that point.

newsclues 3y ago
The base model? Given the article about GPUs, I think I was thinking more along the lines of a desktop with a consumer GPU.
- brucethemoose2 3y ago
  Intel (and AMD) reportedly have big M-Pro-like iGPUs in the pipe, and Intel has the Battlemage GPU line coming up, so we shall see.
- adastra22 3y ago
  By base model do you mean off-the-shelf? I think the bigger limitation there would be RAM, as the machine comes with just 8GB soldered on board. You can expand to 24GB for +$400 though, and increase the GPU cores +25% for an extra +$100. The CPU/GPU speed of the Air is the same as the MacBook Pro base model though. The only difference is the lack of active cooling, which for large workloads can result in performance degradation. Chat apps are intrinsically interactive though, only using bursts of GPU when it is performing inference. Shouldn't be an issue. The expensive PRO/MAX variants of the MacBook Pro would be 2x - 4x faster. But the plain M2 in the MacBook Air is sufficient to get real-time speeds on sizable models. When the M3 comes out for the MacBook Air, discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference.
  - washadjeffmad 3y ago
    > discounted M2 machines, new or refurbished, should be an ideal entry-level machine for local inference At the point of purchase of the lowest cost configuration with 24GB Unified Memory, you've already paid the an equivalent of over 2200 hours of GPU compute time on an RTX 4090 24GB, with a performance that exceeds the MacBook by around 1200% (it/s). If you buy that MacBook for AI, you would have to run continuous generative inference on it for over a decade to match the return of just having used runpod instead. Even Apple doesn't use Macs to do AI - they use GCP TPUs. Buy the Mac if you like it, by all means, but be realistic. LLM performance on Apple Silicon is decent for small models but does not scale. If any Mac architecture were cost effective at any scale for AI, we would be putting them in data centers, and we're not.
    - adastra22 3y ago
      This sub thread isn’t about hosted inference though. I interpreted the question as “is there a reasonable laptop out there that can run large models locally in real time?” And a MacBook Air does fit the bill for that. It’s an entry-level Mac, and it is capable of doing inference jobs. If you have around $1k available to spend on a daily driver laptop, but you also want to do some AI inference experiments, the MacBook Air is what I’d recommend. The AMD Framework laptop might be a good alternative. But I don’t know how well its integrated GPU is supported by llama.cpp.

brucethemoose2 3y ago

I run Yi 34B at 45K-70K context as a GPT 3.5 replacement on a Nvidia 3090. Its quite smart, and fast. The whole rig cost me $2.1K, but it could have easily been $1.5K without splurging on certain parts like I did. And its 10 Liters, small enough to move around.

washadjeffmad 3y ago
Same, but mine is 11L (A4-H2O). Which case did you get?!
- brucethemoose2 3y ago
  A Node 202! I wanted something flat enough for a suitcase. https://www.reddit.com/r/sffpc/comments/18a7mal/ducted_3090_ftw3_stuffed_in_a_node_202/ https://www.reddit.com/r/sffpc/comments/18a7mal/ducted_3090_...

montebicyclelo 3y ago

Consumer laptops, such as apple silicon macbooks with just 32gb ram, can run fairly large models, such as Mixtral-8x7b, an open source model that's comparable to gpt-3.5, well enough for interactive chat, (and seems likely models of this size will keep improving) - ofc, 64gb ram preferable, because it unlocks larger models. And 3090s are the other way to to go, which are faster but not quite as much vram, though you can use multiple.

tudorw 3y ago

I'm using GPT4ALL with Mistral on a $400 Intel NUC, so yes, if the use fits!

Const-me 3y ago

If you use Windows and you have a discrete GPU from any vendor with at least 6GB VRAM, you could test my implementation of Mistral model: https://github.com/Const-me/Cgml https://github.com/Const-me/Cgml Screenshot: https://github.com/Const-me/Cgml/blob/master/Mistral/MistralChat/screenshot.png https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...