4 ms·
Every time a new model hits I'm waiting for his ggmls
by ynniv 3y ago
Every time a new model hits I'm waiting for his ggmls
- brucethemoose2 3y agoggml quantization is very easy with the official llama.cpp repo. Its quick and mostly dependency free, and you can pick the perfect size for your CPU/GPU pool. But don't get me wrong, TheBloke is a hero.
- ynniv 3y agoSome of the newer models have slightly different architectures, so he explains any differences and shows a llama.cpp invocation. Plus you can avoid pulling the larger dataset.
- brucethemoose2 3y agoYeah. Keeping up wkth the changes is madness, and those FP16 weights are huge.
- int_19h 3y agoWhile we're at it, the GGML file format has been deprecated in favor of GGUF. https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md https://github.com/ggerganov/llama.cpp/pull/2398 https://github.com/ggerganov/llama.cpp/pull/2398