3 ms·
> similar to the AI Horde kudos What they are referencing, which is super cool and (IMO) criminally underused: https://lite.koboldai.net/ https://lite.kobolda
by brucethemoose2 3y ago
> similar to the AI Horde kudos
What they are referencing, which is super cool and (IMO) criminally underused:
https://lite.koboldai.net/ https://lite.koboldai.net/
https://tinybots.net/artbot https://tinybots.net/artbot
https://aihorde.net/ https://aihorde.net/
In fact, I can host a 13B-70B finetune in the afternoon if anyone on HN wants to test a particular one out:
https://huggingface.co/models?sort=modified&search=70B+gguf https://huggingface.co/models?sort=modified&search=70B+gguf
- swyx 3y ago> GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp. GGUF offers numerous advantages over GGML, such as better tokenisation, and support for special tokens. It is also supports metadata, and is designed to be extensible is there a more canonical blogpost or link to learn more about the technical decisions here?
- brucethemoose2 3y agohttps://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#historical-state-of-affairs https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md#... It is (IMO) a necessary and good change. I just specified gguf because my 3090 cannot host a 70B model without offloading outside of exLlama's very new ~2 bit quantization. And pre quantized gguf is a much smaller download than raw fp16 for conversion.
- swyx 3y agothanks very much!