9 ms·

Sorry for the off topic question, but does anyone know how to buy consumer hardware optimal for running emerging open source chat models with the largest param

Sorry for the off topic question, but does anyone know how to buy consumer hardware optimal for running emerging open source chat models with the largest parameter chat models possible? Would it be more cost effective to try to buy an absurd amount of ram and run on the cpu? Or buy an Nvidia card with the biggest capacity available? Or maybe buy a Mac with the most memory you can get?

moffkalast 3y ago
Well for running the average model as-is without spending a few days figuring out why you're getting strange errors and can't get it working you more or less need CUDA support. As much VRAM as you can get is probably also a good idea. For reference I can seemingly run Vicuna-7B (I think the 4 bit version) on my 6G 1660 Ti at roughly 1.5 tokens per second. Way too slow for anything useful, so you can imagine what CPU inference would look like.
- londons_explore 3y ago
  CPU inference is only a little slower. GPU's aren't good for a batch size of 1 and everything quantised.
  - execveat 3y ago
    I get 3 tokens per second on M1 Max running 30B models compared to 1 token per second on a GPU (P40), both quantized to 4bit. So, in my opinion CPUs are better for inference (at least fast CPUs with DDR 5 versus cheapest GPUs). The reason why GPUs seem to be the standard de facto is that they scale better, are more power efficient and are better supported by pytorch & co. Also, academia cares more about getting the best quality for their benchmarks, than about the performance and accessibility.
    - londons_explore 3y ago
      GPU's win for training... And those who write papers and publish code tend to do lots of training and only a little inference.
gtirloni 3y ago
Check this article out to build a perspective a out how consumer hardware performs against more business solutions in the machine learning area: https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVIDIA-A100-40-GB-(PCIe)/579vs592 https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVI...
riter 3y ago
there are a few decent YT videos on this topic (dated 2022 though) vast.ai is decent if you want to rent.
execveat 3y ago
ARM-based Macs are the easiest way to get an acceptable performance without the headaches right now, if you can afford the price. Install https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui, update pytorch and llamacpp-python, and you should be able to run pretty much all models out there, in all formats, both on GPU and CPU. CPU on a MAC gives you the fastest speed, but you should pass the correct --threads argument (investigate how many performance cores you've got). GPU is slower, but more energy efficient. https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm gives me way better GPU performance compared to oobabooga, but they only support a couple of models right now, - it's worth following their progress though. If you're after the raw performance, I suggest using GGML models (meant for llama.cpp, but it's bundled in textgen, so you can use it there with the convenience of a web ui). q4_0 is the fastest quantization, while the q5_1 is the best quality right now. If the GGML is not available, you can generate it quite easily from the safetensors yourself (not the you need enough RAM to load the model in pytorch though). With 16GB RAM you can run any 13G model, as long as it's quantized to 4/5 bits. 32GB RAM allows you running 30/33G models and 64GB RAM - 65G models. 30G and 60G models are way more useful for real world tasks, but they are more expensive to train, so there aren't as many to choose from compared to 7/13. 7B and anything less is a toy in my opinion while 13B is good enough for experimentation and prototyping.
- BaculumMeumEst 3y ago
  Thanks, that makes sense and helps a lot. I have a 16gb m1 that I got llama 13b running on. It works really well but I really want to run bigger models, so your examples of ram -> model size are super helpful. I’ll probably just end up getting a higher capacity Mac in the next few years. Right now 96gb configurations seem to be around 4k, if that comes down a bit in the future i’ll probably pick something up. I’m not really looking to train myself so training cost isn’t an issue for me personally, I just want to be able to run the best of what the open source community comes up with (or contribute to a pool to train models, if that becomes a thing).
  - thewataccount 3y ago
    Just a heads up - GPU's are a looooot faster then CPU's depending on what models you're running, especially if you were looking at running the image models. Admittedly I'm not sure how well they work if you stream/batch to the GPU (say 96GB of system ram + 24GB GPU). I've heard used nvidia workstation cards are reasonably cheap for >24GB VRAM. A 3090/4090 have 24GB of vram and can run up the 30B models with some optimizations, and this is the easiest way to run the 30B models which are essentially the highest end any consumer card can run. If you also play games and have money then this is the way to go IMO. If you were to get a GPU, it must have CUDA support (so nvidia only) unless you want a headache.
MacsHeadroom 3y ago
I don't know how anyone hasn't mentioned this yet, the $180 Nvidia Tesla P40 24GB is about as capable as a 4090 for running LLMs (~70% of the token throughput for 8x cheaper). You can even run two or more in SLI to run 65B or larger models. Just search eBay for Nvidia P40. Be sure to add an aftermarket cooling fan ($15 on eBay), as the P40 does not come with its own. The P40 is a LOT faster than an ARM Mac, and a lot cheaper. (Note: Do not go older than a P40. Pascal or newer is required to run 4bit quantizatized models. For example. the $100 M40 24GB is effectively only 6GB as it must run models in 16bit.)
- oersted 3y ago
  Can you provide sources for this claim? If true, how? What is it that the 4090 has that the P40 doesn't to justify the price? I understand that 4090 is aimed at gaming and has a lot of extra bells and whistles like the RTX cores. But it is also consumer electronics and much cheaper than the enterprise GPU lines for the same power. According to this 4090 already has double the raw flop performance of the V100 and is competitive with the most powerful GPUs in the market from last year. https://www.aime.info/blog/en/deep-learning-gpu-benchmarks-2022/ https://www.aime.info/blog/en/deep-learning-gpu-benchmarks-2... And according to this the V100 is ~60% faster than the P40 https://ai-benchmark.com/ranking_deeplearning_detailed.html https://ai-benchmark.com/ranking_deeplearning_detailed.html Not that these sources look particularly reliable, but still, consistent with intuition.
  - KaoruAoiShiho 3y ago
    The claim of 70% of 4090 is very strange, my 4090 runs a 30b at roughly 25 tokens/second compared to the 1token/second claimed by the p40 user here: https://news.ycombinator.com/item?id=35861360 https://news.ycombinator.com/item?id=35861360
    - MacsHeadroom 3y ago
      >compared to the 1token/second claimed by the p40 user That user is doing something wrong. They may not be cooling it and are getting thermal throttled. That would be my guess. The P40 is capable of upwards of 10 tokens/second with 30b.

moffkalast 3y ago

Well for running the average model as-is without spending a few days figuring out why you're getting strange errors and can't get it working you more or less need CUDA support. As much VRAM as you can get is probably also a good idea. For reference I can seemingly run Vicuna-7B (I think the 4 bit version) on my 6G 1660 Ti at roughly 1.5 tokens per second. Way too slow for anything useful, so you can imagine what CPU inference would look like.

londons_explore 3y ago
CPU inference is only a little slower. GPU's aren't good for a batch size of 1 and everything quantised.
- execveat 3y ago
  I get 3 tokens per second on M1 Max running 30B models compared to 1 token per second on a GPU (P40), both quantized to 4bit. So, in my opinion CPUs are better for inference (at least fast CPUs with DDR 5 versus cheapest GPUs). The reason why GPUs seem to be the standard de facto is that they scale better, are more power efficient and are better supported by pytorch & co. Also, academia cares more about getting the best quality for their benchmarks, than about the performance and accessibility.
  - londons_explore 3y ago
    GPU's win for training... And those who write papers and publish code tend to do lots of training and only a little inference.

gtirloni 3y ago

Check this article out to build a perspective a out how consumer hardware performs against more business solutions in the machine learning area: https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVIDIA-A100-40-GB-(PCIe)/579vs592 https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVI...

riter 3y ago

there are a few decent YT videos on this topic (dated 2022 though) vast.ai is decent if you want to rent.

execveat 3y ago

ARM-based Macs are the easiest way to get an acceptable performance without the headaches right now, if you can afford the price. Install https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui, update pytorch and llamacpp-python, and you should be able to run pretty much all models out there, in all formats, both on GPU and CPU. CPU on a MAC gives you the fastest speed, but you should pass the correct --threads argument (investigate how many performance cores you've got). GPU is slower, but more energy efficient. https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/mlc-llm gives me way better GPU performance compared to oobabooga, but they only support a couple of models right now, - it's worth following their progress though. If you're after the raw performance, I suggest using GGML models (meant for llama.cpp, but it's bundled in textgen, so you can use it there with the convenience of a web ui). q4_0 is the fastest quantization, while the q5_1 is the best quality right now. If the GGML is not available, you can generate it quite easily from the safetensors yourself (not the you need enough RAM to load the model in pytorch though). With 16GB RAM you can run any 13G model, as long as it's quantized to 4/5 bits. 32GB RAM allows you running 30/33G models and 64GB RAM - 65G models. 30G and 60G models are way more useful for real world tasks, but they are more expensive to train, so there aren't as many to choose from compared to 7/13. 7B and anything less is a toy in my opinion while 13B is good enough for experimentation and prototyping.

BaculumMeumEst 3y ago
Thanks, that makes sense and helps a lot. I have a 16gb m1 that I got llama 13b running on. It works really well but I really want to run bigger models, so your examples of ram -> model size are super helpful. I’ll probably just end up getting a higher capacity Mac in the next few years. Right now 96gb configurations seem to be around 4k, if that comes down a bit in the future i’ll probably pick something up. I’m not really looking to train myself so training cost isn’t an issue for me personally, I just want to be able to run the best of what the open source community comes up with (or contribute to a pool to train models, if that becomes a thing).
- thewataccount 3y ago
  Just a heads up - GPU's are a looooot faster then CPU's depending on what models you're running, especially if you were looking at running the image models. Admittedly I'm not sure how well they work if you stream/batch to the GPU (say 96GB of system ram + 24GB GPU). I've heard used nvidia workstation cards are reasonably cheap for >24GB VRAM. A 3090/4090 have 24GB of vram and can run up the 30B models with some optimizations, and this is the easiest way to run the 30B models which are essentially the highest end any consumer card can run. If you also play games and have money then this is the way to go IMO. If you were to get a GPU, it must have CUDA support (so nvidia only) unless you want a headache.

MacsHeadroom 3y ago

I don't know how anyone hasn't mentioned this yet, the $180 Nvidia Tesla P40 24GB is about as capable as a 4090 for running LLMs (~70% of the token throughput for 8x cheaper). You can even run two or more in SLI to run 65B or larger models. Just search eBay for Nvidia P40. Be sure to add an aftermarket cooling fan ($15 on eBay), as the P40 does not come with its own. The P40 is a LOT faster than an ARM Mac, and a lot cheaper. (Note: Do not go older than a P40. Pascal or newer is required to run 4bit quantizatized models. For example. the $100 M40 24GB is effectively only 6GB as it must run models in 16bit.)

oersted 3y ago
Can you provide sources for this claim? If true, how? What is it that the 4090 has that the P40 doesn't to justify the price? I understand that 4090 is aimed at gaming and has a lot of extra bells and whistles like the RTX cores. But it is also consumer electronics and much cheaper than the enterprise GPU lines for the same power. According to this 4090 already has double the raw flop performance of the V100 and is competitive with the most powerful GPUs in the market from last year. https://www.aime.info/blog/en/deep-learning-gpu-benchmarks-2022/ https://www.aime.info/blog/en/deep-learning-gpu-benchmarks-2... And according to this the V100 is ~60% faster than the P40 https://ai-benchmark.com/ranking_deeplearning_detailed.html https://ai-benchmark.com/ranking_deeplearning_detailed.html Not that these sources look particularly reliable, but still, consistent with intuition.
- KaoruAoiShiho 3y ago
  The claim of 70% of 4090 is very strange, my 4090 runs a 30b at roughly 25 tokens/second compared to the 1token/second claimed by the p40 user here: https://news.ycombinator.com/item?id=35861360 https://news.ycombinator.com/item?id=35861360
  - MacsHeadroom 3y ago
    >compared to the 1token/second claimed by the p40 user That user is doing something wrong. They may not be cooling it and are getting thermal throttled. That would be my guess. The P40 is capable of upwards of 10 tokens/second with 30b.