Your statement is only partially correct. For single batch inference, something we do locally, the bottleneck is generally bandwidth specially for llm. But no o

Your statement is only partially correct. For single batch inference, something we do locally, the bottleneck is generally bandwidth specially for llm. But no one does single batch inference in server. Also training is bottlenecked by compute and not memory bandwidth.