5 ms·
Called it. It's very unfortunate that the local inference community has aggregated around Ollama when it's clear that's not their long term priority or strateg
by dcreater 1y ago
Called it.
It's very unfortunate that the local inference community has aggregated around Ollama when it's clear that's not their long term priority or strategy.
Its imperative we move away ASAP
- mchiang 1y agohmm, how so? Ollama is open and the pricing is completely optional for users who want additional GPUs. Is it bad to fairly charge money for selling GPUs that cost us money too, and use that money to grow the core open-source project? At one point, it just has to be reasonable. I'd like to believe by having a conscientious, we can create something great.
- tomrod 1y agoEveryone just wants to solarpunk this up.
- dcreater 1y agoIn an ideal world yes - as we should - especially for us Californian/Bay Area people, that's literally our spirit animal. But I understand that is idle dreaming. What I believe certainly is within reach is a state that is much better than what we are in.
- dcreater 1y agoFirst, I must say I appreciate you taking the time to be engaged on this thread and responding to so many of us. What I'm referring to is a broader pattern that I (and several) others have been seeing. Of the top of my head: not crediting llama.cpp previously, still not crediting llama.cpp now and saying you are using your own inference engine when you are still using ggml and the core of what Georgi made, most importantly why even create your own version - is it not better for the community to just contribute to llama.cpp?, making your own propreitary model storage platform disallowing using weights with other local engines requiring people to duplicate downloads and more. I dont know how to regard these other than being largely motivated out of self interest. I think what Jeff and you have built have been enormously helpful to us - Ollama is how I got started running models locally and have enjoyed using it for years now. For that, I think you guys should be paid millions. But what I fear is going to happen is you guys will go the way of the current dogma of capturing users (at least in mindshare) and then continually squeezing more. I would love to be wrong, but I am not going to stick around to find out as its risk I cannot take.
- idiotsecant 1y agoOh no this is a positively diabolical development, offering...hosting services tailored to a specific use case at a reasonable price ...
- SV_BubbleTime 1y agoThey can’t keep getting away with this.
- mrcwinn 1y agoYes, better to get free sh*t unsustainably. By the way, you're free to create an open source alternative and pour your time into that so we can all benefit. But when you don't — remember I called it!
- rpdillon 1y agoWhat? The obvious move is to never have switched to Ollama and just use Llama.cpp directly, which I've been doing for years. Llama.cpp was created first, is the foundation for this product, and is actually open source.
- wkat4242 1y agoBut there's much less that works with that. OpenWebUI for example.
- vntok 1y agoOpen WebUI works perfectly fine with llama.cpp though. They have very detailed quick start docs on it: https://docs.openwebui.com/getting-started/quick-start/starting-with-llama-cpp https://docs.openwebui.com/getting-started/quick-start/start...
- wkat4242 1y agoOh thanks I didn't know that :O I do also need an API server though. The one built into OpenWebUI is no good because it always reloads the model if you use it first from the web console and then run an API call using the same model (like literally the same model from the workspace). Very weird but I avoid it for that reason.
- rpdillon 1y agollama.cpp is what you want. It offers both a web UI and an API on the same port. I use llama.cpp's webui with gpt-oss-20b, and I also leverage it as an OpenAI-compatible server with gptel for Emacs. Very good product.
- tarruda 1y agoLlama.cpp (library which ollama uses under the hoods) has its own server, and it is fully compatible with open-webui. I moved away from ollama in favor of llama-server a couple of months ago and never missed anything, since I'm still using the same UI.
- A4ET8a8uTh0_v2 1y agoInteresting, admittedly, I am slowly getting to the point, where ollama's defaults get a little restrictive. If the setup is not too onerous, I would not mind trying. Where did you start?
- tarruda 1y agoDownload llama-server from llama.cpp Github and install it some PATH directory. AFAIK they don't have an automated installer, so that can be intimidating to some people Assuming you have llama-server installed, you can download + run a hugging face model with something like llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja And access http://localhost:8080 http://localhost:8080
- mchiang 1y agototally respect your choice, and it's a great project too. Of course as a maintainer of Ollama, my preference is to win you over with Ollama. If it doesn't meet your needs, it's okay. We are more energized than ever to keep improving Ollama. Hopefully one day we will win you back. Ollama does not use llama.cpp anymore; we do still keep it and occasionally update it to remain compatible for older models for when we used it. The team is great, we just have features we want to build, and want to implement the models directly in Ollama. (We do use GGML and ask partners to help it. This is a project that also powers llama.cpp and is maintained by that same team)
- tarruda 1y ago> Ollama does not use llama.cpp anymore That is interesting, did Ollama develop its own proprietary inference engine or did you move to something else? Any specific reason why you moved away from llama.cpp?
- janalsncm 1y agoHuggingface also offers a cloud product, but that doesn’t take away from downloading weights and running them locally.
- sitkack 1y agoI believe that is what https://github.com/containers/ramalama https://github.com/containers/ramalama set out to do.
- cchance 1y agoI stopped using them when they started doing the weird model naming bullshit stuck with lmstudio since
- Aurornis 1y ago> Its imperative we move away ASAP Why? If the tool works then use it. They’re not forcing you to use the cloud.
- dcreater 1y agoThere are many, many FOSS apps that use Ollama as a dependency. If Ollama rugs, then all those projects suffer. Its a tale we seen played out many times. Redis is the most recent example.
- Hasnep 1y agoMost apps that integrate with ollama that I've seen just have an OpenAI compatible API parameter which defaults to port 11434 which ollama uses, but can be changed easily. Is there a way to integrate ollama more deeply?
- dcreater 1y agoYes, but I fear the average person will not understand that and assume you need Ollama. That false perception is sufficiently damaging im afraid
- jcelerier 1y agohappy sglang user here :)
- prettyblocks 1y agoLocal inference is becoming completely commoditized imo. These days even docker has a local models you can launch with a single click (or command).
- deleted 1y ago[deleted]
- fud101 1y agoi was trying to remove it but noticed they've hidden the uninstall away. It amounts to doing a rm - which is a joke.