8 bit (and 4 bit?) quantization is low hanging fruit, assuming its not already running in 8 bit. An 8GB requirement kinda sounds like they have already quantiz

8 bit (and 4 bit?) quantization is low hanging fruit, assuming its not already running in 8 bit. An 8GB requirement kinda sounds like they have already quantized the model though.