3 ms·

This smells very badly of quantization - the extra commentary is a failure mode I observe frequently when dropping down from FP16 down to 4 bits.

• aledalgrande 3y ago
  Which base model do you work with?