3 ms·

In general you can swap B for GB (and use the q8 quantization), so 8GB VRAM can probably just about work.

• lolinder 2y ago
  If you want to not quantize at all, you need to double it for fp16—16GB.
  • cjbprime 2y ago
    Yes, but I think it's standard to do inference at q8, not fp16.