3 ms·

How is 2.5 bits possible? Is it an average between 2 and 3 over all weights?

• ftxbro 3y ago
  > Is it an average between 2 and 3 over all weights? Yes I think it's an average where different quantization levels are used for different layers or weights. Here are more details about the quantization scheme: https://github.com/turboderp/exllamav2#exl2-quantization https://github.com/turboderp/exllamav2#exl2-quantization