3 ms·
ggml quantization is very easy with the official llama.cpp repo. Its quick and mostly dependency free, and you can pick the perfect size for your CPU/GPU pool.
by brucethemoose2 3y ago
ggml quantization is very easy with the official llama.cpp repo. Its quick and mostly dependency free, and you can pick the perfect size for your CPU/GPU pool.
But don't get me wrong, TheBloke is a hero.
- ynniv 3y agoSome of the newer models have slightly different architectures, so he explains any differences and shows a llama.cpp invocation. Plus you can avoid pulling the larger dataset.
- brucethemoose2 3y agoYeah. Keeping up wkth the changes is madness, and those FP16 weights are huge.
- int_19h 3y agoWhile we're at it, the GGML file format has been deprecated in favor of GGUF. https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/ggerganov/llama.cpp/pull/2398 https://github.com/ggerganov/llama.cpp/pull/2398