That's correct, but also compute to some degree. The larger the model the more of a bottleneck memory becomes. There are some older HBM cards with very high ba

That's correct, but also compute to some degree. The larger the model the more of a bottleneck memory becomes. There are some older HBM cards with very high bandwidth like the Radeon Pro VII which has 1TB/s of bandwidth like the RTX 3090 and 4090, but is notably slower at inference for smaller models since has less compute in comparison. At least I think that was the consensus of some benchmarks people ran.