3 ms·
It’s because the model weights and KV cache are stored in SRAM. It’s extremely expensive per token.
by xadhominemx 11mo ago
It’s because the model weights and KV cache are stored in SRAM. It’s extremely expensive per token.
loading thread…
3 ms·