3 ms·

Ignore anyone telling you to buy a $4000 Mac. That will work if you’ve already got one, but is not the cheapest option by a longshot. The cheapest option is to

Ignore anyone telling you to buy a $4000 Mac. That will work if you’ve already got one, but is not the cheapest option by a longshot. The cheapest option is to get a recent intel or AMD system, and one or two 3090 cards used on the secondary market ($700-800 each). Make sure you have at least 32GB of system RAM, or preferably 64GB if you have only one 3090. If you’re frugal and don’t buy the latest and greatest CPUs, you can get by on a setup costing less than $1500, or less than about $2300 with two GPUs. Make sure you get a beefy power supply, though you can throttle the 3090s down to ~225W each and not lose much performance. With a single 3090: * Roughly half of the model layers will handled by the GPU * The rest will use CPU and system RAM * Performance may be unacceptably slow for you (maybe like 4 tokens/sec or so, but I haven’t tested) With two GPUs: * Most of the layers will run on the GPUs * Inference will be a lot faster than with single GPU * Depending on the quantization of the specific model file you download, a small portion of the model may still be run from system RAM. Not only is the above option significantly cheaper than any Mac with 64GB+ of RAM, but with two 3090s it’s also way faster. Edited to add: I recognize that I’m comparing the price of a new Mac to the price of used PC equipment, which might bother some. But in my experience, higher end Macs with 64GB or more hold their value extremely well, so I’m assuming for purposes of this comparison that a $4000 (new) Mac is going to be no more than 10-20% cheaper used. If you can find a high end Studio/MBP/etc for a lot less than this, it might be worth considering.

tucnak 2y ago
They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have just got one, although I also had to pay a slight premium because Europe. Now, anybody who's suggesting 2x3090 is either bad at maths, or have not been paying attention. That only gets you 48GB which is barely enough to run 70b models in 4-bit, let alone 8-bit which is what you want to be doing ideally. K/V cache is not free, and should you off-load a bunch of stuff to RAM, it would effectively kill 3090 bandwidth. Mind you, to remain future-proof you would be looking at 3x3090 over a KWh in power (electricity isn't free) also good luck finding a motherboard that would sincerely support 48 lanes, and a CPU that would be able to saturate that. At the end of the day, you would be looking at a 5-6k build to start with. Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.
- angoragoats 2y ago
  I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second) That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said. Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.

tucnak 2y ago

They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have just got one, although I also had to pay a slight premium because Europe. Now, anybody who's suggesting 2x3090 is either bad at maths, or have not been paying attention. That only gets you 48GB which is barely enough to run 70b models in 4-bit, let alone 8-bit which is what you want to be doing ideally. K/V cache is not free, and should you off-load a bunch of stuff to RAM, it would effectively kill 3090 bandwidth. Mind you, to remain future-proof you would be looking at 3x3090 over a KWh in power (electricity isn't free) also good luck finding a motherboard that would sincerely support 48 lanes, and a CPU that would be able to saturate that. At the end of the day, you would be looking at a 5-6k build to start with. Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.

angoragoats 2y ago
I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second) That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said. Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.