19 ms·
> Note: using a 256k context window requires at least 64GB of memory. Using a 1M+ context window requires significantly more (100s of GBs). The ram requirement
by superkuh 3y ago
> Note: using a 256k context window requires at least 64GB of memory. Using a 1M+ context window requires significantly more (100s of GBs).
The ram requirements at 200k seem to be using an assumed 16 bits? With a 4 bit quantization it's more like ~12GB of kv-cache for for 200k sequence length. Unless I'm missing something?
- datadrivenangel 3y agoHow much does this do to performance though?
- deleted 3y ago[deleted]
- sebzim4500 3y agoAre people quantizing the activations? I was under the impression that people generally quantize the parameters but leave the activations unchanged.
- brrrrrm 3y agoYou can quantize KV caches