3 ms·
Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Stud
by pplonski86 2mo ago
Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.
- chii 2mo ago> I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth what model was it that you were able to run with the rtx 3070?
- pplonski86 2mo agoI was able to fit only small models Qwen3.5-4B in RTX3070 which is not very useful for Python and SQL generation thought. When I wan to test larger open LLM models I often just use cloud resources.
- walrus01 2mo agoJust FYI lm-studio is a GUI wrapper on top of a copy of llama-server that the lm-studio developers compile and distribute