Astonishing work, I was muttering to myself last week about quantization and thought useful 4 bit might never happen. IIUC there's a loss of 6-16% which is tota

Astonishing work, I was muttering to myself last week about quantization and thought useful 4 bit might never happen. IIUC there's a loss of 6-16% which is totally reasonable and means my 4.8 GB Mistral model looks more like 2.4 GB once this trickles through to MLC. That means on device GPT 3.25ish at 30 tkns/sec...