12 ms·
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
by mike-the-brain 1mo ago
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
- smcleod 1mo agoUnsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.
- verdverm 1mo agoQuantization is typically very cheap and fast. It can even be done on hardware that does not fit the model, by processing the weights layer by layer. I use this project: https://github.com/vllm-project/llm-compressor https://github.com/vllm-project/llm-compressor
- kristjansson 1mo agoJust quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating. [0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...