Speculative decoding is software, though. How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hidi

Speculative decoding is software, though. How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.