I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that c

I agree with just about everything you say, except maybe the final/concluding sentence, but the parent post asked: > What's the cheapest hardware setup that can run a 70B model at tolerably interactive rates? (say 10 characters a second) That is the question I was answering. I have personally seen Llama3 70B, Q4, running via Ollama on a dual 3090 setup, and it runs about as fast as ChatGPT 4 does, and does not use main memory as far as I can tell. Yes, it's a quantized model, but the quality is pretty good, and the OP didn't specify that they must use 8 bit or greater so I figured they were going for cheap, like they said. Edit: also, some of your figures are off: instantaneous power is measured in watts, not watt-hours or kWh, and if you want to be more efficient you can comfortably drop the max power of 3090s down to about 225W each and not lose much performance with LLM loads, as I mentioned above. Additionally, each 3090 only needs 4 lanes of PCIe 4.0 bandwidth (there are diminishing returns beyond that), so a motherboard with 48 (or even 16) free lanes is completely unnecessary. Again, I'm trying to answer the question of what the cheapest setup would be, not the ideal set up or the one with the most/fastest memory.