On servers you're right: but for local LLM inference, I think you're wrong. For local LLMs most people are bottlenecked by not having enough VRAM: pretty much n

On servers you're right: but for local LLM inference, I think you're wrong. For local LLMs most people are bottlenecked by not having enough VRAM: pretty much no one is running a 70b model on Nvidia GPUs locally, just due to the expense. You don't need maximum performance: you need it to run at all, which most people can't do for the good models — at least, not without heavy quantization that pretty badly lobotomizes them. Apple is the king right now of local LLM inference, just because of their unified memory architecture meaning that people can get large amounts of "VRAM" (since all RAM is VRAM). They're not as fast as Nvidia — not even close to an H100, for example. But they don't need to be. No consumer can afford an H100, but they can afford a Mac.