3 ms·It’s because the model weights and KV cache are stored in SRAM. It’s extremely expensive per token.by xadhominemx 1y agoIt’s because the model weights and KV cache are stored in SRAM. It’s extremely expensive per token.