3 ms·
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense. Things w
by LeBit 2mo ago
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.
Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"
- chorizo 2mo agoHad a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.