PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you

PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother. PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device. max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-technical-analysis https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff. > How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? $10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.