Depends on quantization etc. But there are good calculators that will calculate for your KV cache etc as well: https://apxml.com/tools/vram-calculator https://a

Depends on quantization etc. But there are good calculators that will calculate for your KV cache etc as well: https://apxml.com/tools/vram-calculator https://apxml.com/tools/vram-calculator.