5 ms·

Can this be quantized to run on a device?

• ben_w 3y ago
  "quantized" yes; "on device" depends very much on your device and if you're OK with the slowdown from using normal (or worse, virtual) RAM. > The 8-bit and 4-bit quantized versions of Falcon 180B show almost no difference in evaluation with respect to the bfloat16 reference! This is very good news for inference, as you can confidently use a quantized version to reduce hardware requirements. Keep in mind, though, that 8-bit inference is much faster than running the model in 4-bit.
  • sp332 3y ago
    I didn't expect that, why would 4 bit be slower?
    • ben_w 3y ago
      I also didn't expect it; my guess is that the library is doing 8 (or more) bit arithmetic, and using bit masking and shifting to un/compress the values?
    • qeternity 3y ago
      It isn't. It's just that 4bit implementations are really bad. Exllama has shown how properly optimized and fused kernels can perform.