2 ms·
Indeed. Almost all of the inference memory overhead comes from attention matrices and the KV cache.
by woadwarrior01 3y ago
Indeed. Almost all of the inference memory overhead comes from attention matrices and the KV cache.
loading thread…
2 ms·