4 ms·
there's also ollama, which I haven't used much yet. they used to have llama.cpp as the only backend, but it appears they've now started to include their own cod
by exe34 2y ago
there's also ollama, which I haven't used much yet. they used to have llama.cpp as the only backend, but it appears they've now started to include their own code.
- ekianjo 2y agoyes, with Llama3.2 vision they are diverging from llama.cpp backend only.
- pulse7 2y agoIf I understood correctly they just added image preprocessing and then they feed that into llama.cpp...
- polotics 2y agoOllama is kind of ok to get started, but as I understand it they don't give you a choice in the quantisation you'll use. Please correct me if I'm wrong. One thing I am sure about it is they store large model files renamed as large globally unique identifier, and I still haven't understood that part of the design as anything but some silly obfuscating embrace... And here again, I'd love to be shown how I'm wrong.
- lurking_swe 2y agoyou can, when you search for a model on the ollama website there is a drop down that lets you select a “tag”. Sort of like a docker container tag. This lets you pick the quantization you want. example: https://ollama.com/library/llama3.2/tags https://ollama.com/library/llama3.2/tags
- solarkraft 2y agoAll in the name of UX. It’s modeled after Docker, so it defaults to doing things that way. Really does make for great ease of use, imo.
- mseri 2y agoYou can choose the quantization by appending the right tag to the model name, but they don't support other more advanced useful features (e.g. you need a special flag to enable flash attention and you cannot use KV cache quantization for large contexts).