Thats just one method... - bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quant

Thats just one method... - bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quantize ahead of time, but its not resource intense. - mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization