6 ms·

I think it would be more interesting doing this with smaller models (33b-70b) and see if you could get 5-10 tokens/sec on a budget. I've desperately wanted some

I think it would be more interesting doing this with smaller models (33b-70b) and see if you could get 5-10 tokens/sec on a budget. I've desperately wanted something locally thats around the same level of 4o, but I'm not in a hurry to spend $3k on an overpriced GPU or $2k on this

firtoz 2y ago
Would it be something like this? > OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi https://x.com/geerlingguy/status/1884994878477623485 https://x.com/geerlingguy/status/1884994878477623485 I haven't tried it myself or haven't verified the creds, but seems exciting at least
- gliptic 2y ago
  That's 1.2 t/s for the 14B Qwen finetune, not the real R1. Unless you go with the GPU with the extra cost, but hardly anyone but Jeff Geerling is going to run a dedicated GPU on a Pi.
- etra0 2y ago
  it's using a Raspberry Pi with a.... USD$1k gpu, which kinda defeat the purpose of using the RPI in the first place imo. or well, I guess you save a bit on power usage.
  - firtoz 2y ago
    Oh, I was naive to think that the Pi was capable of some kind of magic (sweaty smile emoji goes here)
  - unethical_ban 2y ago
    I suppose it makes sense, for extremely GPU centric applications, that the pi be used essentially as a controller for the 3090.
gliptic 2y ago
Your best bet for 33B is already having a computer and buying a used RTX 3090 for <$1k. I don't think there's currently any cheap options for 70B that would give you >5. High memory bandwidth is just too expensive. Strix Halo might give you >5 once it comes out, but will probably be significantly more than $1k for 64 GB RAM.
- jjallen 2y ago
  How does inference happen on a GPU with such limited memory compared with the full requirements of the model? This is something I’ve been wondering for a while
  - Gracana 2y ago
    You can run a quantized version of the model to reduce the memory requirements, and you can do partial offload, where some of the model is on GPU and some is on CPU. If you are running a 70B Q4, that’s 40-ish GB including some context cache, and you can offload at least half onto a 3090, which will run its portion of the load very fast. It makes a huge difference even if you can’t fit every layer on the GPU.
    - jjallen 2y ago
      So the more GPUs we have the faster it will be and we don't have to have the model run solely CPU or GPU -- it can be combined. Very cool. Think that is how it's running now with my single 4090.
- ants_everywhere 2y ago
  With used GPUs do you have to be concerned that they're close to EOL due to high utilization in a Bitcoin or AI rig?
  - gliptic 2y ago
    I guess it will be a bigger issue the longer it's been since they stopped making them, but most I've heard (including me) haven't had any issue. Crypto rigs don't necessarily break GPUs faster because they care about power consumption and run the cards at a pretty even temperature. What probably breaks first is the fans. You might also have to open the card up and repaste/repad them to keep the cooling under control.
api 2y ago
Apple M chips with their unified GPU memory are not terrible. I have one of the first M1 Max laptops with 64G and it can run up to 70B models at very useful speeds. Newer M series are going to be faster and they offer more RAM now. Are there any other laptops around other than the larger M series Macs that can run 30-70B LLMs at usable speeds that also have useful battery life and don’t sound like a jet taxiing to the runway? For non-portables I bet a huge desktop or server CPU with fast RAM beats the Mac Mini and Studio for price performance, but I’d be curious to see benchmarks comparing fast many core CPU performance to a large M series GPU with unified RAM.
jenny91 2y ago
As a data point: you can get an RTX 3090 for ~$1.2k and it runs deepseek-r1:32b perfectly fine via Ollama + open webui at ~35 tok/s in an OpenAI-like web app and basically as fast as 4o.
- kevinak 2y ago
  You mean Qwen 32b fine-tuned on Deepseek :) There is only one model of Deepseek (671b), all others are fine-tunes of other models
- driverdan 2y ago
  > you can get an RTX 3090 for ~$1.2k If you're paying that much you're being ripped off. They're $800-900 on eBay and IMO are still overpriced.
bick_nyers 2y ago
It will be slower for a 70b model since Deepseek is an MoE that only activates 37b at a time. That's what makes CPU inference remotely feasible here.
Svoka 2y ago
You can run smaller models on MacbookPro with ollama with those speeds. Even with several 3k GPUs it won't come close to 4o level.
spaceport 2y ago
I put together a $350 build with a 3060 12GB and its still my favorite build. I run llama 3.2 11b q4 on it and its a really efficient way to get started and the tps is great.

firtoz 2y ago

Would it be something like this? > OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi https://x.com/geerlingguy/status/1884994878477623485 https://x.com/geerlingguy/status/1884994878477623485 I haven't tried it myself or haven't verified the creds, but seems exciting at least

gliptic 2y ago
That's 1.2 t/s for the 14B Qwen finetune, not the real R1. Unless you go with the GPU with the extra cost, but hardly anyone but Jeff Geerling is going to run a dedicated GPU on a Pi.
etra0 2y ago
it's using a Raspberry Pi with a.... USD$1k gpu, which kinda defeat the purpose of using the RPI in the first place imo. or well, I guess you save a bit on power usage.
- firtoz 2y ago
  Oh, I was naive to think that the Pi was capable of some kind of magic (sweaty smile emoji goes here)
- unethical_ban 2y ago
  I suppose it makes sense, for extremely GPU centric applications, that the pi be used essentially as a controller for the 3090.

gliptic 2y ago

Your best bet for 33B is already having a computer and buying a used RTX 3090 for <$1k. I don't think there's currently any cheap options for 70B that would give you >5. High memory bandwidth is just too expensive. Strix Halo might give you >5 once it comes out, but will probably be significantly more than $1k for 64 GB RAM.

jjallen 2y ago
How does inference happen on a GPU with such limited memory compared with the full requirements of the model? This is something I’ve been wondering for a while
- Gracana 2y ago
  You can run a quantized version of the model to reduce the memory requirements, and you can do partial offload, where some of the model is on GPU and some is on CPU. If you are running a 70B Q4, that’s 40-ish GB including some context cache, and you can offload at least half onto a 3090, which will run its portion of the load very fast. It makes a huge difference even if you can’t fit every layer on the GPU.
  - jjallen 2y ago
    So the more GPUs we have the faster it will be and we don't have to have the model run solely CPU or GPU -- it can be combined. Very cool. Think that is how it's running now with my single 4090.
ants_everywhere 2y ago
With used GPUs do you have to be concerned that they're close to EOL due to high utilization in a Bitcoin or AI rig?
- gliptic 2y ago
  I guess it will be a bigger issue the longer it's been since they stopped making them, but most I've heard (including me) haven't had any issue. Crypto rigs don't necessarily break GPUs faster because they care about power consumption and run the cards at a pretty even temperature. What probably breaks first is the fans. You might also have to open the card up and repaste/repad them to keep the cooling under control.

api 2y ago

Apple M chips with their unified GPU memory are not terrible. I have one of the first M1 Max laptops with 64G and it can run up to 70B models at very useful speeds. Newer M series are going to be faster and they offer more RAM now. Are there any other laptops around other than the larger M series Macs that can run 30-70B LLMs at usable speeds that also have useful battery life and don’t sound like a jet taxiing to the runway? For non-portables I bet a huge desktop or server CPU with fast RAM beats the Mac Mini and Studio for price performance, but I’d be curious to see benchmarks comparing fast many core CPU performance to a large M series GPU with unified RAM.

jenny91 2y ago

As a data point: you can get an RTX 3090 for ~$1.2k and it runs deepseek-r1:32b perfectly fine via Ollama + open webui at ~35 tok/s in an OpenAI-like web app and basically as fast as 4o.

kevinak 2y ago
You mean Qwen 32b fine-tuned on Deepseek :) There is only one model of Deepseek (671b), all others are fine-tunes of other models
driverdan 2y ago
> you can get an RTX 3090 for ~$1.2k If you're paying that much you're being ripped off. They're $800-900 on eBay and IMO are still overpriced.

bick_nyers 2y ago

It will be slower for a 70b model since Deepseek is an MoE that only activates 37b at a time. That's what makes CPU inference remotely feasible here.

Svoka 2y ago

You can run smaller models on MacbookPro with ollama with those speeds. Even with several 3k GPUs it won't come close to 4o level.

spaceport 2y ago

I put together a $350 build with a 3060 12GB and its still my favorite build. I run llama 3.2 11b q4 on it and its a really efficient way to get started and the tps is great.