You can predict the token generation performance of a GPU or CPU by dividing the memory bandwidth by the size of the active parameters. By definition, that is a

You can predict the token generation performance of a GPU or CPU by dividing the memory bandwidth by the size of the active parameters. By definition, that is a memory bandwidth bottleneck. I have no idea why you think it is not. Anyone who has worked on inference code knows that memory bandwidth is the principal bottleneck for token generation. For example: https://github.com/ryao/llama3.c https://github.com/ryao/llama3.c