3 ms·
I have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead. llama.cpp basically dr
by buyucu 2y ago
I have unfortunately been disappointed with the llama.cpp/ollama ecosystem of late, and thinking about moving my things to vllm instead.
llama.cpp basically dropped support for multimodal visual models. ollama still does support them, but only a handful. Also ollama still does not support vulkan eventhough llama.cpp had vulkan support for a long long time now.
This has been very sad to watch. I'm more and more convinced that vllm is the way to go, not ollama.
- jacooper 2y agoBut can you run llms that easily with vllm? do you have to fiddle with formats to get it to run?
- buyucu 2y agoI'm still in early stages of exploration, but vllm seems to be compatible with most models on huggingface.
- mistercheph 2y agoOllama maintainers seem a bit burned out