3 ms·

how much vram does the 8B model use?

• cjbprime 3y ago
  In general you can swap B for GB (and use the q8 quantization), so 8GB VRAM can probably just about work.
  • lolinder 3y ago
    If you want to not quantize at all, you need to double it for fp16—16GB.
    • cjbprime 3y ago
      Yes, but I think it's standard to do inference at q8, not fp16.
• derac 3y ago
  You can use 5 bits per parameter with negligible loss of capability as a general rule. 4 bits for a tiny bit worse results. This is subject to changes in how good quantization is in general and on the specific model.