4 ms·

What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second)

frozenport 2y ago
M2 macs can do it: https://twitter.com/junrushao/status/1681828325923389440 https://twitter.com/junrushao/status/1681828325923389440 in practice 10 tokens per second is kinda annoyingly slow most local people would opt for a smaller 7b model
- zarzavat 2y ago
  Have been playing around with Llama3 7b today, it’s not very good. I’m sure that Facebook put everything they could into making it good, but 7B is apparently just not enough parameters.
  - pennomi 2y ago
    I assume you mean 8B? There is no Llama 3 7B.
    - sieszpak 2y ago
      Llama 3 8B seems sad to answer... this is the first model in a long time that has had trouble telling me how much is 3! - (factorial)
  - mistrial9 2y ago
    llava-v1.5-7b-q4.llamafile yes agree that the impression is poor overall
  - d-z-m 2y ago
    not very good compared to what? Hard to reconcile your comment with its outsize performance on arena/glowing praise from others comparing it to much larger models.
    - zarzavat 2y ago
      Trying the 8B on translation gives some hilariously garbled results. It’s a small model so perhaps not unexpected but it’s definitely nowhere close to GPT-3. I’d be cautious of synthetic benchmarks, you never know how much of the scores are due to contamination or survivorship bias.
      - anon373839 2y ago
        Languages other than English are “out of scope” according to the model card, so I wouldn’t expect strong translation performance. In English, though, it’s incredibly capable for its size.
      - deleted 2y ago
        [deleted]
      - d-z-m 2y ago
        I didn't think arena was a synthetic benchmark?
  - frozenport 2y ago
    Yeah thats why OpenAI, etc don't run 7B But for scrapping, and tool calling it represents a substantial gain.
jart 2y ago
Any Macbook with 32GB should be able to run Meta-Llama-3-70B-Instruct.Q2_K.llamafile which I uploaded a few minutes ago. It's smart enough to solve math riddles, but at this level of quantization you should expect hallucinations. If you want to run Q4_0 you'll probably be able to squeeze it on a $3,999.00 Macbook Pro M3 Max w/ 48GB of RAM. If you want to run Q5_K_M or or Q8_0 the best choice is probably Mac Studio. I have an Apple M2 Ultra w/ 24‑core CPU, 60‑core GPU, 128GB RAM. It cost me $8000 with the monitor. If I run Meta-Llama-3-70B-Instruct.Q4_0.llamafile then I get 14 tok/sec (prompt eval is 82 tok/sec) thanks to the Metal GPU. You could alternatively go on vast.ai and rent a system with 4x RTX 4090's for a few bucks an hour. That'll run 70b. Or you could build your own, but the graphics cards alone will cost $10k+. AMD Threadripper Pro 7995WX ($10k) does a good job too. I get 5.9 tok/sec eval with Q4_0 and 49 tok/sec prompt. If I use F16 weights then prompt eval goes 65 tok/sec.
- jtrn 2y ago
  Thank you for this. I am looking for as much information on Lama 70b running locally, but not finding as much as I would like. How come you have access to both 7995WX and M3 Max? Professional setting or hobbyist? I'm not knowledgeable with regards to running LLMs on CPU. Do I understand correctly that Threadripper treats RAM as unified memory in the same way as Apple silicon, so that you could in theory dump 256GB of RAM into the system and have the FP16 llama3 70b actually run and load?
- FloatArtifact 2y ago
  I'm going to wait for late 2024 gpus or 2025 cpus and do a cost analysis.
jacobfnl 2y ago
I have 70B at q4_0 running at 20 tokens/s inference using 2 Founders Edition RTX3090s with NVLink. It's an older AMD 5950x with a motherboard with 4-slot spacing between the PCI-e (iirc, the motherboard is an ASRock x570 Taichi). For comparison, I tried the same rig with a single RTX 4090, but it generated an abysmal 2.4 tokens/s, which I would consider nigh unusable. The difference between all GPU VRAM is significant.
angoragoats 2y ago
Ignore anyone telling you to buy a $4000 Mac. That will work if you’ve already got one, but is not the cheapest option by a longshot. The cheapest option is to get a recent intel or AMD system, and one or two 3090 cards used on the secondary market ($700-800 each). Make sure you have at least 32GB of system RAM, or preferably 64GB if you have only one 3090. If you’re frugal and don’t buy the latest and greatest CPUs, you can get by on a setup costing less than $1500, or less than about $2300 with two GPUs. Make sure you get a beefy power supply, though you can throttle the 3090s down to ~225W each and not lose much performance. With a single 3090: * Roughly half of the model layers will handled by the GPU * The rest will use CPU and system RAM * Performance may be unacceptably slow for you (maybe like 4 tokens/sec or so, but I haven’t tested) With two GPUs: * Most of the layers will run on the GPUs * Inference will be a lot faster than with single GPU * Depending on the quantization of the specific model file you download, a small portion of the model may still be run from system RAM. Not only is the above option significantly cheaper than any Mac with 64GB+ of RAM, but with two 3090s it’s also way faster. Edited to add: I recognize that I’m comparing the price of a new Mac to the price of used PC equipment, which might bother some. But in my experience, higher end Macs with 64GB or more hold their value extremely well, so I’m assuming for purposes of this comparison that a $4000 (new) Mac is going to be no more than 10-20% cheaper used. If you can find a high end Studio/MBP/etc for a lot less than this, it might be worth considering.
- tucnak 2y ago
  They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have just got one, although I also had to pay a slight premium because Europe. Now, anybody who's suggesting 2x3090 is either bad at maths, or have not been paying attention. That only gets you 48GB which is barely enough to run 70b models in 4-bit, let alone 8-bit which is what you want to be doing ideally. K/V cache is not free, and should you off-load a bunch of stuff to RAM, it would effectively kill 3090 bandwidth. Mind you, to remain future-proof you would be looking at 3x3090 over a KWh in power (electricity isn't free) also good luck finding a motherboard that would sincerely support 48 lanes, and a CPU that would be able to saturate that. At the end of the day, you would be looking at a 5-6k build to start with. Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.

frozenport 2y ago

M2 macs can do it: https://twitter.com/junrushao/status/1681828325923389440 https://twitter.com/junrushao/status/1681828325923389440 in practice 10 tokens per second is kinda annoyingly slow most local people would opt for a smaller 7b model

zarzavat 2y ago
Have been playing around with Llama3 7b today, it’s not very good. I’m sure that Facebook put everything they could into making it good, but 7B is apparently just not enough parameters.
- pennomi 2y ago
  I assume you mean 8B? There is no Llama 3 7B.
  - sieszpak 2y ago
    Llama 3 8B seems sad to answer... this is the first model in a long time that has had trouble telling me how much is 3! - (factorial)
- mistrial9 2y ago
  llava-v1.5-7b-q4.llamafile yes agree that the impression is poor overall
- d-z-m 2y ago
  not very good compared to what? Hard to reconcile your comment with its outsize performance on arena/glowing praise from others comparing it to much larger models.
  - zarzavat 2y ago
    Trying the 8B on translation gives some hilariously garbled results. It’s a small model so perhaps not unexpected but it’s definitely nowhere close to GPT-3. I’d be cautious of synthetic benchmarks, you never know how much of the scores are due to contamination or survivorship bias.
    - anon373839 2y ago
      Languages other than English are “out of scope” according to the model card, so I wouldn’t expect strong translation performance. In English, though, it’s incredibly capable for its size.
    - deleted 2y ago
      [deleted]
    - d-z-m 2y ago
      I didn't think arena was a synthetic benchmark?
- frozenport 2y ago
  Yeah thats why OpenAI, etc don't run 7B But for scrapping, and tool calling it represents a substantial gain.

jart 2y ago

Any Macbook with 32GB should be able to run Meta-Llama-3-70B-Instruct.Q2_K.llamafile which I uploaded a few minutes ago. It's smart enough to solve math riddles, but at this level of quantization you should expect hallucinations. If you want to run Q4_0 you'll probably be able to squeeze it on a $3,999.00 Macbook Pro M3 Max w/ 48GB of RAM. If you want to run Q5_K_M or or Q8_0 the best choice is probably Mac Studio. I have an Apple M2 Ultra w/ 24‑core CPU, 60‑core GPU, 128GB RAM. It cost me $8000 with the monitor. If I run Meta-Llama-3-70B-Instruct.Q4_0.llamafile then I get 14 tok/sec (prompt eval is 82 tok/sec) thanks to the Metal GPU. You could alternatively go on vast.ai and rent a system with 4x RTX 4090's for a few bucks an hour. That'll run 70b. Or you could build your own, but the graphics cards alone will cost $10k+. AMD Threadripper Pro 7995WX ($10k) does a good job too. I get 5.9 tok/sec eval with Q4_0 and 49 tok/sec prompt. If I use F16 weights then prompt eval goes 65 tok/sec.

jtrn 2y ago
Thank you for this. I am looking for as much information on Lama 70b running locally, but not finding as much as I would like. How come you have access to both 7995WX and M3 Max? Professional setting or hobbyist? I'm not knowledgeable with regards to running LLMs on CPU. Do I understand correctly that Threadripper treats RAM as unified memory in the same way as Apple silicon, so that you could in theory dump 256GB of RAM into the system and have the FP16 llama3 70b actually run and load?
FloatArtifact 2y ago
I'm going to wait for late 2024 gpus or 2025 cpus and do a cost analysis.

jacobfnl 2y ago

I have 70B at q4_0 running at 20 tokens/s inference using 2 Founders Edition RTX3090s with NVLink. It's an older AMD 5950x with a motherboard with 4-slot spacing between the PCI-e (iirc, the motherboard is an ASRock x570 Taichi). For comparison, I tried the same rig with a single RTX 4090, but it generated an abysmal 2.4 tokens/s, which I would consider nigh unusable. The difference between all GPU VRAM is significant.

angoragoats 2y ago

Ignore anyone telling you to buy a $4000 Mac. That will work if you’ve already got one, but is not the cheapest option by a longshot. The cheapest option is to get a recent intel or AMD system, and one or two 3090 cards used on the secondary market ($700-800 each). Make sure you have at least 32GB of system RAM, or preferably 64GB if you have only one 3090. If you’re frugal and don’t buy the latest and greatest CPUs, you can get by on a setup costing less than $1500, or less than about $2300 with two GPUs. Make sure you get a beefy power supply, though you can throttle the 3090s down to ~225W each and not lose much performance. With a single 3090: * Roughly half of the model layers will handled by the GPU * The rest will use CPU and system RAM * Performance may be unacceptably slow for you (maybe like 4 tokens/sec or so, but I haven’t tested) With two GPUs: * Most of the layers will run on the GPUs * Inference will be a lot faster than with single GPU * Depending on the quantization of the specific model file you download, a small portion of the model may still be run from system RAM. Not only is the above option significantly cheaper than any Mac with 64GB+ of RAM, but with two 3090s it’s also way faster. Edited to add: I recognize that I’m comparing the price of a new Mac to the price of used PC equipment, which might bother some. But in my experience, higher end Macs with 64GB or more hold their value extremely well, so I’m assuming for purposes of this comparison that a $4000 (new) Mac is going to be no more than 10-20% cheaper used. If you can find a high end Studio/MBP/etc for a lot less than this, it might be worth considering.

tucnak 2y ago
They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have just got one, although I also had to pay a slight premium because Europe. Now, anybody who's suggesting 2x3090 is either bad at maths, or have not been paying attention. That only gets you 48GB which is barely enough to run 70b models in 4-bit, let alone 8-bit which is what you want to be doing ideally. K/V cache is not free, and should you off-load a bunch of stuff to RAM, it would effectively kill 3090 bandwidth. Mind you, to remain future-proof you would be looking at 3x3090 over a KWh in power (electricity isn't free) also good luck finding a motherboard that would sincerely support 48 lanes, and a CPU that would be able to saturate that. At the end of the day, you would be looking at a 5-6k build to start with. Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.