3 ms·

They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have

by tucnak 2y ago

They should ignore anyone telling them to ignore Mac advice. First of all, there's plenty of refurbished M2 Max 96GB+ Studio's out there going for $3000, I have just got one, although I also had to pay a slight premium because Europe. Now, anybody who's suggesting 2x3090 is either bad at maths, or have not been paying attention. That only gets you 48GB which is barely enough to run 70b models in 4-bit, let alone 8-bit which is what you want to be doing ideally. K/V cache is not free, and should you off-load a bunch of stuff to RAM, it would effectively kill 3090 bandwidth. Mind you, to remain future-proof you would be looking at 3x3090 over a KWh in power (electricity isn't free) also good luck finding a motherboard that would sincerely support 48 lanes, and a CPU that would be able to saturate that. At the end of the day, you would be looking at a 5-6k build to start with. Like it or not, 96/128/192GB macs are the most affordable in VRAM/dollar.

angoragoats 2y ago
I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second) That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said. Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.

by tucnak 2y ago

angoragoats 2y ago

I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second) That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said. Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.