7 ms·

Llama.cpp supports Vulkan. why doesn't Ollama?

217 points by buyucu 2y ago

buyucu 2y ago
llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.
- Kubuxu 2y ago
  Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomment-2599740463 https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
  - buyucu 2y ago
    I was not aware of this context, thanks!
  - andy_ppp 2y ago
    This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.
    - api 2y ago
      This is fascinating. I’ve been using ollama with no knowledge of this because it just works without a ton of knobs I don’t feel like spending the time to mess with. As usual, the real work seems to be appropriated by people who do the last little bit — put an acceptable user experience and some polish on it — and they take all the money and credit. It’s shitty but it also happens because the vast majority of devs, especially in the FOSS world, do not understand or appreciate user experience. It is bar none the most important thing in the success of most things in computing. My rule is: every step a user has to do to install or set up something halves adoption. So if 100 people enter and there are two steps, 25 complete the process. For a long time Apple was the most valuable corporation on Earth on the basis of user experience alone. Apple doesn’t invent much. They polish it, and that’s where like 99% of the value is as far as the market is concerned. The reason is that computers are very confusing and hard to use. Computer people, which most of us are, don’t see that because it’s second nature to us. But even for computer people you get to the point where you’re busy and don’t have time to nerd out on every single thing you use, so it even matters to computer people in the end.
      - anewhnaccount2 2y ago
        I think it's understood, but someone needs to work on the actual infrastructure.
      - openrisk 2y ago
        The problem is that whatever esoteric motivations technical people have to join the FOSS movement (scratching an itch, seeking fame, saving the world, doing what everybody else is doing etc.), does not translate well to the domain of designing user experiences. People with the education and talent to have an impact here have neither incentives nor practical means to "FOSS-it". You could Creative Commons some artwork (and there are beautiful examples) but thats about it. The art and science of making software usable thus remains a proprietary pursuit. Indeed if that bottleneck could somehow be relaxed, adoption of FOSS software would skyrocket because the technical core is so good and keeps getting better.
    - jdright 2y ago
      Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is my personal experience with Python in general, and 10x worse when I need to figure out all compatible packages with specifc ROCm support for my GPU. This is madness, even C and C++ setup and build is easier than this Python hell.
      - exe34 2y ago
        I get the impression the important stuff is done in a container rather than on the host system, so having python/pip might be all you need.
        ecurtin 2y ago
        This is true.
      - cge 2y ago
        RamaLama's use of Python is different: it appears to just be using Python for scripting its container management. It doesn't need ROCm to work with Python or anything else. It has no difficult dependencies or anything else: I just installed it with `uv tool install ramalama` and it worked fine. I'd agree that Python packaging is generally bad, and that within an LLM context it's a disastrous mess (especially for ROCm), but that doesn't appear to be how RamaLama is using it at all.
        ecurtin 2y ago
        @cge you have this right, the main python script has no dependancies, it just uses python3 stdlib stuff. So if you have a python3 executable on your system you are good to go. All the stuff with dependancies runs in a container. On macOS, using no containers works well also, as we basically just install brew llama.cpp There's really no major python dependancy problems people have been running this on many Linux distros, macOS, etc. We deliberately don't use python libraries because of the packaging problems.
  - bearjaws 2y ago
    It's hilarious that docker guys are trying to take another OSS and monetize it. Hey if it worked once?...
- n144q 2y ago
  Thanks, just yesterday I discovered that Ollama could not use iGPU on my AMD machine, and was going through a long issue for solutions/workarounds (https://github.com/ollama/ollama/issues/2637 https://github.com/ollama/ollama/issues/2637). Existing instructions are based on Linux, and some people found it utterly surprising that anyone wants to run LLMs on Windows (really?). While I would have no trouble installing Linux and compile from source, I wasn't ready to do that to my main, daily-use computer. Great to see this. PS. Have you got feedback on whether this works on Windows? If not, I can try to create a build today.
- zozbot234 2y ago
  The PR has been legitimately out-of-date and unmergeable for many months. It was forward-ported a few weeks ago, and is now still awaiting formal review and merging. (To be sure, Vulkan support in Ollama will likely stay experimental for some time even if the existing PR is merged, and many setups will need manual adjustment of the number of GPU layers and such. It's far from 100% foolproof even in the best-case scenario!) For that matter, some people are still having issues building and running it, as seen from the latest comments on the linked GitHub page. It's not clear that it's even in a fully reviewable state just yet.
  - buyucu 2y ago
    this pr was reviewable multiple times, rebased multiple times. all because ollama team kept ignoring it. it has been open for almost 7 months now without a single comment from the ollama folks.
  - ecurtin 2y ago
    It's gets out of date with conflicts, etc. Because it's ignored, if this was the upstream project of Ollama, llama.cpp the maintainers would have got this merged months ago.
- 9cb14c1ec0 2y ago
  The PR at issue here blocks iGPUs. My fork of the PR changes removes that: https://github.com/9cb14c1ec0/ollama-vulkan https://github.com/9cb14c1ec0/ollama-vulkan I successfully ran Phi4 on my AMD Ryzen 7 PRO 5850U iGPU with it.
  - buyucu 2y ago
    this is great! I think pufferfish is taking PRs to his fork as well.
quibono 2y ago
Is Ollama just the porcelain around llama.cpp? Or is there more to it than that?
- buyucu 2y ago
  yes, it's a convenience wrapper around llama.cpp
- diggan 2y ago
  They also decided to rehost the model files in their own (closed) library/repository + store the files split into layers on disk, so you cannot easily reuse model-files between applications. I think the point is that models can share layers, I'm not sure how much space you actually save, I just know that if you use both LM Studio + Ollama you cannot share models but if you use LM Studio + llama.cpp you can share the same files between them, no need to download duplicate model weights.
- ac29 2y ago
  The main feature IMO is the model library. llama.cpp on its own does not come with any built in way to download and manage models.
turnsout 2y ago
This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?
- paradite 2y ago
  For starters: - It doesn't have a website - It doesn't have a download page, you have to build it yourself
  - woadwarrior01 2y ago
    > - It doesn't have a download page, you have to build it yourself I'd wager that anyone capable enough to run a command line tool like Ollama should also be able to download prebuilt binaries from the llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp https://formulae.brew.sh/formula/llama.cpp
    - a12k 2y ago
      I am very technically inclined and use Ollama (in a VM, but still) because of all the steps and non-obviousness of how to run Llama.cpp. This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking.
      - woadwarrior01 2y ago
        > This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking. No this isn't. There are plenty of end user GUI apps that make it far easier than Ollama to download and run local LLMs (disclaimer: I build one of them). That's an entirely different market. IMO, the intersection between the set of people who use a command line tool, and the set of people who are incapable of running `brew install llama.cpp` (or its Windows or Linux equivalents) is exceedingly small.
cedws 2y ago
The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.
- andy_ppp 2y ago
  It would be extremely unsurprising if Nvidia was funding this embrace and extend behind the scenes.
  - parineum 2y ago
    It would be pretty surprising to their shareholders if Nvidia was hiding where it was spending it's money.
- buyucu 2y ago
  I did not assume the worst when submitting the post, but that is also my suspicion. The whole thing is very dodgy.
  - bloomingkales 2y ago
    Are closer to the metal AI developers an under tracked bottle neck? AMD and Intel can barely get off the ground due to lagging software developers.
    - jvanderbot 2y ago
      This is where I want to work. But I feel like an AI swe is more likely to go "down" than an AI company is likely to hire me, a guy who loves optimizing pipelines for parallelism.
    - zozbot234 2y ago
      Metal is an Apple thing, not Intel or AMD. (And Ollama supports that.)
      - 2y ago
paradite 2y ago
Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone who built apps for Windows, Linux, macOS, iOS and Android, it is not trivial to ensure your new features or updates work on all platforms, and you have to deal with deprecations.
- buyucu 2y ago
  ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.
  - paradite 2y ago
    Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.
    - buyucu 2y ago
      check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.
      - fkyoureadthedoc 2y ago
        If it's "just" a friendly front and, why doesn't llama.cpp just drop one themselves? Do they actually care about the situation, or are random people just mad on their behalf?
      - paradite 2y ago
        At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.
the_mitsuhiko 2y ago
Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.
- buyucu 2y ago
  I totally agree that ollama needs competition. They have been doing very sketchy things lately. I wish llama.cpp had an alternative wrapper client like ollama.
- prabir 2y ago
  There is https://cortex.so/ https://cortex.so/ that I’m looking forward too.
  - adastra22 2y ago
    Hey thanks, I didn't know about cortex and this looks perfect.
- 7thpower 2y ago
  Can you please explain why you think they may be operating in bad faith?
  - diggan 2y ago
    Not parent, but same feeling. First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library. Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture. Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "Misinformation" and rewrites your messages. You can ask a honest question, it gets deleted and you get blocked for a day. Just today I asked why the R1 models they're shipping that are the distilled ones, doesn't have "distilled" in the name, or even any way of knowing which tag is which model, and got the answer "if you don't like how things are done on Ollama, you can run your own object registry" which doesn't exactly inspire confidence. Another thing I noticed after a while is that there are bunch of people with zero knowledge of terminals that want to run Ollama, even though Ollama is a project for developers (since you do need to know how to run a terminal). Just making the messaging clearer would help a lot in this regarding, but somehow the Ollama team thinks thats gatekeeping and it's better to teach people basic terminal operations.
mschwaig 2y ago
Ollama tries to appeal to a lowest common denominator user base, who does not want to worry about stuff like configuration and quants, or which binary to download. I think they want their project to be smart enough to just 'figure out what to do' on behalf of the user. That appeals to a lot of people, but I think them stuffing all backends into one binary and auto-detecting at runtime which to use and is actually a step too far towards simplicity. What they did to support both CUDA and ROCm using the same binary looked quite cursed last time I checked (because they needed to link or invoke two different builds of llama.cpp of course). I have only glanced at that PR, but I'm guessing that this plays a role in how many backends they can reasonably try to support. In nixpkgs it's a huge pain that we configure quite deliberately what we want Ollama to do at build time, and then Ollama runs off and does whatever anyways, and users have to look at log output and performance regressions to know what it's actually doing, every time they update their heuristics for detecting ROCm. It's brittle as hell.
- buyucu 2y ago
  I disagree with this, but it's a reasonable argument. The problem is that the Ollama team has basically ignored the PR, instead of engaging the community. The least they can do is to explain their reasoning. This PR is #1 on their repo based on multiple metrics (comments, iterations, what have you)
llm_trw 2y ago
Can someone explain what the point of ollama is? Every time I look at it, it seems like it's a worse llama.cpp that removes options to make things "easier".
- ianpurton 2y ago
  It's very easy to install and add models.
- portaouflop 2y ago
  It’s to make things easier for casual users. With ollama I type brew install ollama and then ollama get something, and I have it already running. With llama.ccp it’s seems i have to build it first, then manually download models somewhere - this is an instant turnoff, i maybe have 5 minutes of my life to waste on this
- danielbln 2y ago
  Not to be snide, but removing options to make things easier has been wildly successful in a variety of project/products.
- baq 2y ago
  yeah that's literally the point. you're listing something that you think is a disadvantage and some people think exactly the opposite.
- pornel 2y ago
  Ollama : llama.cpp :: Dropbox : rsync
  - diggan 2y ago
    Not sure this is a good analogy. LM Studio is closer to Dropbox as both takes X and makes it easier for users who don't necessarily are very technical. Ollama is a developer-oriented tool (used via terminal + a daemon), so wouldn't compare it to what Dropbox is/did for file syncing.
- michaelt 2y ago
a12k 2y ago
Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs, which imo increases the sketchiness.
- nialv7 2y ago
  It's fully open source. I mean yes it uses llama.cpp without giving it credit. But why run it in a VM?
  - a12k 2y ago
    It severely over-permissions itself on my Mac.
    - deleted 2y ago
      [deleted]
    - super_mario 2y ago
      Can you please elaborate? How are you running ollama? I just build it from source and have written a shell script to start/stop it. It runs under my local user account (I should probably have its own user) and is of course not exposed outside localhost.
    - celeritascelery 2y ago
      I have never had it request any permissions
    - _w1tm 2y ago
      Just install it from Brew and run the service in a separate terminal tab.
  - instagary 2y ago
    Isn't there a clause in MIT that says you're required to give credit? Also, I didn't know a YC company which started it: https://www.ycombinator.com/companies/ollama https://www.ycombinator.com/companies/ollama.
benxh 2y ago
My biggest gripe with Ollama is the badly named models, e.g. under deepseek-r1, it defaults to the distill models.
- buyucu 2y ago
  I agree they should rename them. But defaulting to a 671b model is also evil.
  - rfoo 2y ago
    No. If you can't run it and most people can never run the model on their laptop, it's fine, let people know the fact, instead of giving them illusion.
    - Mashimo 2y ago
      Letting people download 400GB just to find that out is also .. not optimal. But yes, I have been "yelled" at on reddit for telling people you need vram in the hundreds of GB.
      - diggan 2y ago
        > Letting people download 400GB just to find that out is also .. not optimal. Letting people download any amount of bytes just to find out they got something else isn't optimal. So what to do? Highlight the differences when you reference them so people understand. Tweets like these: https://x.com/ollama/status/1881427522002506009 https://x.com/ollama/status/1881427522002506009 > DeepSeek's first-generation reasoning models are achieving performance comparable to OpenAI's o1 across math, code, and reasoning tasks! Give it a try! 7B distilled: ollama run deepseek-r1:7b Are really misleading. Reading the first part, you think the second part is that model that gives "performance comparable to OpenAI's o1" but it's not, it's a distilled model with way worse performance. Yes, they do say it's the distilled model, but I hope I'm not alone in seeing how people less careful would confuse the two. If they're doing this on purpose, I'd leave a very bad taste in my mouth. If they're doing this accidentally, it also gives me reason to pause and re-evaluate what they're doing.
your_challenger 2y ago
I don't know why one would use Ollama instead of llama.cpp. llama.cpp is so easy to use and the maintainer is pretty famous and active in the community.
- buyucu 2y ago
  Llama.cpp dropped support for multimodal vlms. That is why I am using ollama. I would happily switch back if I could.
  - Gracana 2y ago
    llama.cpp readme still lists multimodal models.. Qwen2-VL and others. Is that inaccurate, or something different? [edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8010 https://github.com/ggerganov/llama.cpp/issues/8010
    - buyucu 2y ago
      it's a grey zone but vlms are effectively not being developed anymore.
trash_cat 2y ago
I use Ollama because I am a casual user and can't be bothered to read the docs on how to setup llama.cpp. I just want to run a simple llm locally. Why would I care about Vulkan?
- buyucu 2y ago
  with vulkan it runs much much faster on consumer hardware, especially opn igpus like intel or amd.
  - zozbot234 2y ago
    Well, it definitely runs faster on external dGPU's. With iGPU's and possibly future NPU's, the pre-processing/"thinking" phase is much faster (because that one is compute-bound) but text generation tends to be faster on CPU because it makes better use of available memory bandwidth (which is the relevant constraint there). iGPU's and NPU's will still be a win wrt. energy use, however.
  - bdhcuidbebe 2y ago
    For Intel, OpenVINO should be the preferred route. I dont follow AMD, but Vulkan is just the common denominator here.
    - buyucu 2y ago
      If you support Vulkan, you support almost every GPU out there in the consumer market across all hardware vendors. It's an amazing fallback option. I agree they should also support OpenVINO, but compared to Vulkan OpenVINO is a tiny market.
      - bdhcuidbebe 2y ago
        I made an argument for performance, not for compatibility. If you run your local llm in the least performant way possible on tour overly expensive GPU, then you are not making value of your purchase. Vulkan is a fallback option is all. I even see people running on their CPU because some apps dont support their hardware and llama.cpp made it even possible. It is still a really bad idea. Its just goes to show there’s still much to do.
2-3-7-43-1807 2y ago
can someone please give a quick summary of the criticism towards ollama? as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. i found ollama through hn and last time i didn't notice any lack of trust or suspected sketchiness ... so what changed?
- buyucu 2y ago
  ollama has been advertising llama.cpp features as their own, which I find very dishonest.
- denverllc 2y ago
  IMO ggerganov is a 10x programmer in the same way Fabrice Bellard is: doing the actual hard infrastructure work that most developers would not be able to do in a reasonable amount of time and at a high performance. In contrast, the ollama dev team is doing useful work (creating an easy interface) but otherwise mostly piggybacking off the already existing infrastructure
- woadwarrior01 2y ago
  > as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. That's completely off the mark. https://www.ycombinator.com/companies/ollama https://www.ycombinator.com/companies/ollama
  - 2-3-7-43-1807 2y ago
    seems like i confused llamafile with ollama ... this whole llm biotope is a huge mess
deleted 2y ago
[deleted]
lolinder 2y ago
So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, why isn't llama.cpp as easy to use? Making local LLMs accessible to the masses is an essential job right now—it's important to normalize owning your data as much as it can be normalized. For all of its faults, Ollama does that, and it does it far better than any alternative. Maybe wait to trash it for being "just" a wrapper until someone actually creates a viable alternative.
- buyucu 2y ago
  supporting vulkan will help ollama reach the masses who don't have dedicated gpus from nvidia. this is such a low hanging fruit that it's silly how they are acting.
  - lolinder 2y ago
    As has been pointed out in this thread in a comment that you replied to (so I know you saw it) [0], Ollama goes to a lot of contortions to support multiple llama.cpp backends. Yes, their solution is a bit of a hack, but it means that the effort to adding a new back end is substantial. And again, they're doing those contortions to make it easy for people. Making it easy involves trade-offs. Yes, Ollama has flaws. They could communicate better about why they're ignoring PRs. All I'm saying is let's not pretend they're not doing anything complicated or difficult when no one has been able to recreate what they're doing. [0] https://news.ycombinator.com/item?id=42886933 https://news.ycombinator.com/item?id=42886933
    - buyucu 2y ago
      This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.
Havoc 2y ago
ollama was good initially in that it made LLMs more accessible for non-technical people while everyone was figuring things out. Lately they seem to be contributing mostly confusion to the conversation. The #1 model the entire world is talking about is literally mislabeled their side. There is no such thing as R1-1.5b. Quantization without telling users also confuses noobs as to what is possible. Setting up an api different from the thing they're wrapping adds chaos. And claiming each feature added llama.cpp as something "ollama now supports" is exceedingly questionable especially when combined with the very sparse acknowledgement that it's a wrapper at all. Whole thing just doesn't have good vibes
- dingocat 2y ago
  What do you mean there is no such thing as R1-1.5b? DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0]. [0] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948
  - trissi1996 2y ago
    Which is not the same model, it's not R1 it's R1-Distill-Qwen-1.5B....
    - dingocat 2y ago
      A distinction they make clear and write extensively about on the model page, yes?
      - macki0 2y ago
        wheres that made clear in "ollama run deepseek-r1” the command to download/run the model?
        dingocat 2y ago
        Which you have to go to the model page to find.
deleted 2y ago
[deleted]
av_conk 2y ago
I tried using ollama because I couldn't get ROCm working on my system with llama-cpp. Ollama bundles the ROCm libraries for you. I got around 50 tokens per second with that setup. I tried llama-cpp with the Vulkan backend and doubled the amount of tokens per second. I was under the impression ROCm is superior to Vulkan, so I was confused about the result. In any case, I've stuck with llama-cpp.
- buyucu 2y ago
  It depends on your GPU. Vulkan is well-supported by essentially all GPUs. AMD support ROCm well for their datacenter GPUs, but support for consumer hardware has not been as good.
deleted 2y ago
[deleted]
wkat4242 2y ago
That's a weird thing about Ollama yes. It took very long for them to support KV cache quantisation too (which drastically reduces the amount of VRAM needed for context!). Even though the underlying llama.cpp had offered it for ages. And they had it handed to them on a platter, someone had developed everything and submitted a patch. The developer of that patch even was about to give up as he had to constantly keep it up to date with upstream even though he was constantly being ignored. So he had no idea if it would ever be merged. They just seem to be really hesitant to offer new features. Eventually it was merged and it made a huge difference to people with low VRAM cards.
arvigeus 2y ago
Finally a response from the maintainers: https://github.com/ollama/ollama/pull/5059#issuecomment-2628002106 https://github.com/ollama/ollama/pull/5059#issuecomment-2628... I am the least qualified person to comment on that, but honestly their response made me to raise an eyebrow.
colorant 2y ago
There is ipex-llm support for Ollama on Intel GPU (https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/ollama_quickstart.md https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...)

217 points by buyucu 2y ago

llama.cpp has supported vulkan for more than a year now. For more than 6 months now there has been an open PR to add vulkan backend support for Ollama. However, Ollama team has not even looked at it or commented on it. Vulkan backends are existential for running LLMs on consumer hardware (iGPUs especially). It's sad to see Ollama miss this opportunity.

Kubuxu 2y ago
Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomment-2599740463 https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
- buyucu 2y ago
  I was not aware of this context, thanks!
- andy_ppp 2y ago
  This is great, I did not know about RamaLama and I'll be using and recommending that in future and if I see people using Ollama in instructions I'll recommend they move to RamaLama in the future. Cheers.
  - api 2y ago
    This is fascinating. I’ve been using ollama with no knowledge of this because it just works without a ton of knobs I don’t feel like spending the time to mess with. As usual, the real work seems to be appropriated by people who do the last little bit — put an acceptable user experience and some polish on it — and they take all the money and credit. It’s shitty but it also happens because the vast majority of devs, especially in the FOSS world, do not understand or appreciate user experience. It is bar none the most important thing in the success of most things in computing. My rule is: every step a user has to do to install or set up something halves adoption. So if 100 people enter and there are two steps, 25 complete the process. For a long time Apple was the most valuable corporation on Earth on the basis of user experience alone. Apple doesn’t invent much. They polish it, and that’s where like 99% of the value is as far as the market is concerned. The reason is that computers are very confusing and hard to use. Computer people, which most of us are, don’t see that because it’s second nature to us. But even for computer people you get to the point where you’re busy and don’t have time to nerd out on every single thing you use, so it even matters to computer people in the end.
    - anewhnaccount2 2y ago
      I think it's understood, but someone needs to work on the actual infrastructure.
    - openrisk 2y ago
      The problem is that whatever esoteric motivations technical people have to join the FOSS movement (scratching an itch, seeking fame, saving the world, doing what everybody else is doing etc.), does not translate well to the domain of designing user experiences. People with the education and talent to have an impact here have neither incentives nor practical means to "FOSS-it". You could Creative Commons some artwork (and there are beautiful examples) but thats about it. The art and science of making software usable thus remains a proprietary pursuit. Indeed if that bottleneck could somehow be relaxed, adoption of FOSS software would skyrocket because the technical core is so good and keeps getting better.
  - jdright 2y ago
    Yeah, I would love an actual alternative to Ollama, but RamaLama is not it unfortunately. As the other commenter said, onboarding is important. I just want one operation install and it needs to work and the simple fact RamaLama is written in Python, assures it will never be that easy, and this is even more true with LLM stuff when using AMD gpu. I know there will be people that disagree with this, that's ok. This is my personal experience with Python in general, and 10x worse when I need to figure out all compatible packages with specifc ROCm support for my GPU. This is madness, even C and C++ setup and build is easier than this Python hell.
    - exe34 2y ago
      I get the impression the important stuff is done in a container rather than on the host system, so having python/pip might be all you need.
      - ecurtin 2y ago
        This is true.
    - cge 2y ago
      RamaLama's use of Python is different: it appears to just be using Python for scripting its container management. It doesn't need ROCm to work with Python or anything else. It has no difficult dependencies or anything else: I just installed it with `uv tool install ramalama` and it worked fine. I'd agree that Python packaging is generally bad, and that within an LLM context it's a disastrous mess (especially for ROCm), but that doesn't appear to be how RamaLama is using it at all.
      - ecurtin 2y ago
        @cge you have this right, the main python script has no dependancies, it just uses python3 stdlib stuff. So if you have a python3 executable on your system you are good to go. All the stuff with dependancies runs in a container. On macOS, using no containers works well also, as we basically just install brew llama.cpp There's really no major python dependancy problems people have been running this on many Linux distros, macOS, etc. We deliberately don't use python libraries because of the packaging problems.
- bearjaws 2y ago
  It's hilarious that docker guys are trying to take another OSS and monetize it. Hey if it worked once?...
n144q 2y ago
Thanks, just yesterday I discovered that Ollama could not use iGPU on my AMD machine, and was going through a long issue for solutions/workarounds (https://github.com/ollama/ollama/issues/2637 https://github.com/ollama/ollama/issues/2637). Existing instructions are based on Linux, and some people found it utterly surprising that anyone wants to run LLMs on Windows (really?). While I would have no trouble installing Linux and compile from source, I wasn't ready to do that to my main, daily-use computer. Great to see this. PS. Have you got feedback on whether this works on Windows? If not, I can try to create a build today.
zozbot234 2y ago
The PR has been legitimately out-of-date and unmergeable for many months. It was forward-ported a few weeks ago, and is now still awaiting formal review and merging. (To be sure, Vulkan support in Ollama will likely stay experimental for some time even if the existing PR is merged, and many setups will need manual adjustment of the number of GPU layers and such. It's far from 100% foolproof even in the best-case scenario!) For that matter, some people are still having issues building and running it, as seen from the latest comments on the linked GitHub page. It's not clear that it's even in a fully reviewable state just yet.
- buyucu 2y ago
  this pr was reviewable multiple times, rebased multiple times. all because ollama team kept ignoring it. it has been open for almost 7 months now without a single comment from the ollama folks.
- ecurtin 2y ago
  It's gets out of date with conflicts, etc. Because it's ignored, if this was the upstream project of Ollama, llama.cpp the maintainers would have got this merged months ago.
9cb14c1ec0 2y ago
The PR at issue here blocks iGPUs. My fork of the PR changes removes that: https://github.com/9cb14c1ec0/ollama-vulkan https://github.com/9cb14c1ec0/ollama-vulkan I successfully ran Phi4 on my AMD Ryzen 7 PRO 5850U iGPU with it.
- buyucu 2y ago
  this is great! I think pufferfish is taking PRs to his fork as well.

quibono 2y ago

Is Ollama just the porcelain around llama.cpp? Or is there more to it than that?

buyucu 2y ago
yes, it's a convenience wrapper around llama.cpp
diggan 2y ago
They also decided to rehost the model files in their own (closed) library/repository + store the files split into layers on disk, so you cannot easily reuse model-files between applications. I think the point is that models can share layers, I'm not sure how much space you actually save, I just know that if you use both LM Studio + Ollama you cannot share models but if you use LM Studio + llama.cpp you can share the same files between them, no need to download duplicate model weights.
ac29 2y ago
The main feature IMO is the model library. llama.cpp on its own does not come with any built in way to download and manage models.

turnsout 2y ago

This is going to sound like a troll, but it's an honest question: Why do people use Ollama over llama.cpp? llama.cpp has added a ton of features, is about as user-friendly as Ollama, and is higher-performance. Is there some key differentiator for Ollama that I'm missing?

paradite 2y ago
For starters: - It doesn't have a website - It doesn't have a download page, you have to build it yourself
- woadwarrior01 2y ago
  > - It doesn't have a download page, you have to build it yourself I'd wager that anyone capable enough to run a command line tool like Ollama should also be able to download prebuilt binaries from the llama.cpp releases page[1]. Also, prebuilt binaries are available on things like homebrew[2]. [1]: https://github.com/ggerganov/llama.cpp/releases https://github.com/ggerganov/llama.cpp/releases [2]: https://formulae.brew.sh/formula/llama.cpp https://formulae.brew.sh/formula/llama.cpp
  - a12k 2y ago
    I am very technically inclined and use Ollama (in a VM, but still) because of all the steps and non-obviousness of how to run Llama.cpp. This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking.
    - woadwarrior01 2y ago
      > This framing feels a bit like the “Dropbox won’t succeed because rsync is easy” thinking. No this isn't. There are plenty of end user GUI apps that make it far easier than Ollama to download and run local LLMs (disclaimer: I build one of them). That's an entirely different market. IMO, the intersection between the set of people who use a command line tool, and the set of people who are incapable of running `brew install llama.cpp` (or its Windows or Linux equivalents) is exceedingly small.

cedws 2y ago

The way Ollama has basically been laundering llama.cpp’s features as its own felt dodgy, this appears to confirm there’s something underhanded going on.

andy_ppp 2y ago
It would be extremely unsurprising if Nvidia was funding this embrace and extend behind the scenes.
- parineum 2y ago
  It would be pretty surprising to their shareholders if Nvidia was hiding where it was spending it's money.
buyucu 2y ago
I did not assume the worst when submitting the post, but that is also my suspicion. The whole thing is very dodgy.
- bloomingkales 2y ago
  Are closer to the metal AI developers an under tracked bottle neck? AMD and Intel can barely get off the ground due to lagging software developers.
  - jvanderbot 2y ago
    This is where I want to work. But I feel like an AI swe is more likely to go "down" than an AI company is likely to hire me, a guy who loves optimizing pipelines for parallelism.
  - zozbot234 2y ago
    Metal is an Apple thing, not Intel or AMD. (And Ollama supports that.)
    - 2y ago

paradite 2y ago

Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone who built apps for Windows, Linux, macOS, iOS and Android, it is not trivial to ensure your new features or updates work on all platforms, and you have to deal with deprecations.

buyucu 2y ago
ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.
- paradite 2y ago
  Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.
  - buyucu 2y ago
    check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.
    - fkyoureadthedoc 2y ago
      If it's "just" a friendly front and, why doesn't llama.cpp just drop one themselves? Do they actually care about the situation, or are random people just mad on their behalf?
    - paradite 2y ago
      At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.

the_mitsuhiko 2y ago

Ollama needs competition. I’m not sure what drives the people that maintain it but some of their actions imply that there are ulterior motives at play that do not have the benefit of their users in mind. However such projects require a lot of time and effort and it’s not clear if this project can be forked and kept alive.

buyucu 2y ago
I totally agree that ollama needs competition. They have been doing very sketchy things lately. I wish llama.cpp had an alternative wrapper client like ollama.
prabir 2y ago
There is https://cortex.so/ https://cortex.so/ that I’m looking forward too.
- adastra22 2y ago
  Hey thanks, I didn't know about cortex and this looks perfect.
7thpower 2y ago
Can you please explain why you think they may be operating in bad faith?
- diggan 2y ago
  Not parent, but same feeling. First I got the feeling because of how they store things on disk and try to get all models rehosted in their own closed library. Second time I got the feeling is when it's not obvious at all about what their motives are, and that it's a for-profit venture. Third time is trying to discuss things in their Discord and the moderators there constantly shut down a lot of conversation citing "Misinformation" and rewrites your messages. You can ask a honest question, it gets deleted and you get blocked for a day. Just today I asked why the R1 models they're shipping that are the distilled ones, doesn't have "distilled" in the name, or even any way of knowing which tag is which model, and got the answer "if you don't like how things are done on Ollama, you can run your own object registry" which doesn't exactly inspire confidence. Another thing I noticed after a while is that there are bunch of people with zero knowledge of terminals that want to run Ollama, even though Ollama is a project for developers (since you do need to know how to run a terminal). Just making the messaging clearer would help a lot in this regarding, but somehow the Ollama team thinks thats gatekeeping and it's better to teach people basic terminal operations.

mschwaig 2y ago

Ollama tries to appeal to a lowest common denominator user base, who does not want to worry about stuff like configuration and quants, or which binary to download. I think they want their project to be smart enough to just 'figure out what to do' on behalf of the user. That appeals to a lot of people, but I think them stuffing all backends into one binary and auto-detecting at runtime which to use and is actually a step too far towards simplicity. What they did to support both CUDA and ROCm using the same binary looked quite cursed last time I checked (because they needed to link or invoke two different builds of llama.cpp of course). I have only glanced at that PR, but I'm guessing that this plays a role in how many backends they can reasonably try to support. In nixpkgs it's a huge pain that we configure quite deliberately what we want Ollama to do at build time, and then Ollama runs off and does whatever anyways, and users have to look at log output and performance regressions to know what it's actually doing, every time they update their heuristics for detecting ROCm. It's brittle as hell.

buyucu 2y ago
I disagree with this, but it's a reasonable argument. The problem is that the Ollama team has basically ignored the PR, instead of engaging the community. The least they can do is to explain their reasoning. This PR is #1 on their repo based on multiple metrics (comments, iterations, what have you)

llm_trw 2y ago

Can someone explain what the point of ollama is? Every time I look at it, it seems like it's a worse llama.cpp that removes options to make things "easier".

ianpurton 2y ago
It's very easy to install and add models.
portaouflop 2y ago
It’s to make things easier for casual users. With ollama I type brew install ollama and then ollama get something, and I have it already running. With llama.ccp it’s seems i have to build it first, then manually download models somewhere - this is an instant turnoff, i maybe have 5 minutes of my life to waste on this
danielbln 2y ago
Not to be snide, but removing options to make things easier has been wildly successful in a variety of project/products.
baq 2y ago
yeah that's literally the point. you're listing something that you think is a disadvantage and some people think exactly the opposite.
pornel 2y ago
Ollama : llama.cpp :: Dropbox : rsync
- diggan 2y ago
  Not sure this is a good analogy. LM Studio is closer to Dropbox as both takes X and makes it easier for users who don't necessarily are very technical. Ollama is a developer-oriented tool (used via terminal + a daemon), so wouldn't compare it to what Dropbox is/did for file syncing.
michaelt 2y ago

a12k 2y ago

Ollama is sketchy enough that I run it in a VM. Which is odd because it would probably take less effort to just run Llama.cpp directly, but VMs are pretty easy so just went that route. When I see people bring up the sketchiness most of the time the creator responds with the equivalent of shrugs, which imo increases the sketchiness.

nialv7 2y ago
It's fully open source. I mean yes it uses llama.cpp without giving it credit. But why run it in a VM?
- a12k 2y ago
  It severely over-permissions itself on my Mac.
  - deleted 2y ago
    [deleted]
  - super_mario 2y ago
    Can you please elaborate? How are you running ollama? I just build it from source and have written a shell script to start/stop it. It runs under my local user account (I should probably have its own user) and is of course not exposed outside localhost.
  - celeritascelery 2y ago
    I have never had it request any permissions
  - _w1tm 2y ago
    Just install it from Brew and run the service in a separate terminal tab.
- instagary 2y ago
  Isn't there a clause in MIT that says you're required to give credit? Also, I didn't know a YC company which started it: https://www.ycombinator.com/companies/ollama https://www.ycombinator.com/companies/ollama.

benxh 2y ago

My biggest gripe with Ollama is the badly named models, e.g. under deepseek-r1, it defaults to the distill models.

buyucu 2y ago
I agree they should rename them. But defaulting to a 671b model is also evil.
- rfoo 2y ago
  No. If you can't run it and most people can never run the model on their laptop, it's fine, let people know the fact, instead of giving them illusion.
  - Mashimo 2y ago
    Letting people download 400GB just to find that out is also .. not optimal. But yes, I have been "yelled" at on reddit for telling people you need vram in the hundreds of GB.
    - diggan 2y ago
      > Letting people download 400GB just to find that out is also .. not optimal. Letting people download any amount of bytes just to find out they got something else isn't optimal. So what to do? Highlight the differences when you reference them so people understand. Tweets like these: https://x.com/ollama/status/1881427522002506009 https://x.com/ollama/status/1881427522002506009 > DeepSeek's first-generation reasoning models are achieving performance comparable to OpenAI's o1 across math, code, and reasoning tasks! Give it a try! 7B distilled: ollama run deepseek-r1:7b Are really misleading. Reading the first part, you think the second part is that model that gives "performance comparable to OpenAI's o1" but it's not, it's a distilled model with way worse performance. Yes, they do say it's the distilled model, but I hope I'm not alone in seeing how people less careful would confuse the two. If they're doing this on purpose, I'd leave a very bad taste in my mouth. If they're doing this accidentally, it also gives me reason to pause and re-evaluate what they're doing.

your_challenger 2y ago

I don't know why one would use Ollama instead of llama.cpp. llama.cpp is so easy to use and the maintainer is pretty famous and active in the community.

buyucu 2y ago
Llama.cpp dropped support for multimodal vlms. That is why I am using ollama. I would happily switch back if I could.
- Gracana 2y ago
  llama.cpp readme still lists multimodal models.. Qwen2-VL and others. Is that inaccurate, or something different? [edit] Oh I see, here's an issue about it: https://github.com/ggerganov/llama.cpp/issues/8010 https://github.com/ggerganov/llama.cpp/issues/8010
  - buyucu 2y ago
    it's a grey zone but vlms are effectively not being developed anymore.

trash_cat 2y ago

I use Ollama because I am a casual user and can't be bothered to read the docs on how to setup llama.cpp. I just want to run a simple llm locally. Why would I care about Vulkan?

buyucu 2y ago
with vulkan it runs much much faster on consumer hardware, especially opn igpus like intel or amd.
- zozbot234 2y ago
  Well, it definitely runs faster on external dGPU's. With iGPU's and possibly future NPU's, the pre-processing/"thinking" phase is much faster (because that one is compute-bound) but text generation tends to be faster on CPU because it makes better use of available memory bandwidth (which is the relevant constraint there). iGPU's and NPU's will still be a win wrt. energy use, however.
- bdhcuidbebe 2y ago
  For Intel, OpenVINO should be the preferred route. I dont follow AMD, but Vulkan is just the common denominator here.
  - buyucu 2y ago
    If you support Vulkan, you support almost every GPU out there in the consumer market across all hardware vendors. It's an amazing fallback option. I agree they should also support OpenVINO, but compared to Vulkan OpenVINO is a tiny market.
    - bdhcuidbebe 2y ago
      I made an argument for performance, not for compatibility. If you run your local llm in the least performant way possible on tour overly expensive GPU, then you are not making value of your purchase. Vulkan is a fallback option is all. I even see people running on their CPU because some apps dont support their hardware and llama.cpp made it even possible. It is still a really bad idea. Its just goes to show there’s still much to do.

2-3-7-43-1807 2y ago

can someone please give a quick summary of the criticism towards ollama? as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. i found ollama through hn and last time i didn't notice any lack of trust or suspected sketchiness ... so what changed?

buyucu 2y ago
ollama has been advertising llama.cpp features as their own, which I find very dishonest.
denverllc 2y ago
IMO ggerganov is a 10x programmer in the same way Fabrice Bellard is: doing the actual hard infrastructure work that most developers would not be able to do in a reasonable amount of time and at a high performance. In contrast, the ollama dev team is doing useful work (creating an easy interface) but otherwise mostly piggybacking off the already existing infrastructure
woadwarrior01 2y ago
> as far as my intel goes it's a mozilla project shouldered mostly by one 10x programmer. That's completely off the mark. https://www.ycombinator.com/companies/ollama https://www.ycombinator.com/companies/ollama
- 2-3-7-43-1807 2y ago
  seems like i confused llamafile with ollama ... this whole llm biotope is a huge mess

deleted 2y ago

[deleted]

lolinder 2y ago

So many here are trashing on Ollama, saying it's "just" nice porcelain around llama.cpp and it's not doing anything complicated. Okay. Let's stipulate that. So where's the non-sketchy, non-for-profit equivalent? Where's the nice frontend for llama.cpp that makes it trivial for anyone who wants to play around with local LLMs without having to know much about their internals? If Ollama isn't doing anything difficult, why isn't llama.cpp as easy to use? Making local LLMs accessible to the masses is an essential job right now—it's important to normalize owning your data as much as it can be normalized. For all of its faults, Ollama does that, and it does it far better than any alternative. Maybe wait to trash it for being "just" a wrapper until someone actually creates a viable alternative.

buyucu 2y ago
supporting vulkan will help ollama reach the masses who don't have dedicated gpus from nvidia. this is such a low hanging fruit that it's silly how they are acting.
- lolinder 2y ago
  As has been pointed out in this thread in a comment that you replied to (so I know you saw it) [0], Ollama goes to a lot of contortions to support multiple llama.cpp backends. Yes, their solution is a bit of a hack, but it means that the effort to adding a new back end is substantial. And again, they're doing those contortions to make it easy for people. Making it easy involves trade-offs. Yes, Ollama has flaws. They could communicate better about why they're ignoring PRs. All I'm saying is let's not pretend they're not doing anything complicated or difficult when no one has been able to recreate what they're doing. [0] https://news.ycombinator.com/item?id=42886933 https://news.ycombinator.com/item?id=42886933
  - buyucu 2y ago
    This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.

Havoc 2y ago

ollama was good initially in that it made LLMs more accessible for non-technical people while everyone was figuring things out. Lately they seem to be contributing mostly confusion to the conversation. The #1 model the entire world is talking about is literally mislabeled their side. There is no such thing as R1-1.5b. Quantization without telling users also confuses noobs as to what is possible. Setting up an api different from the thing they're wrapping adds chaos. And claiming each feature added llama.cpp as something "ollama now supports" is exceedingly questionable especially when combined with the very sparse acknowledgement that it's a wrapper at all. Whole thing just doesn't have good vibes

dingocat 2y ago
What do you mean there is no such thing as R1-1.5b? DeepSeek released a distilled version based on a 1.5B Qwen model with the full name DeepSeek-R1-Distill-Qwen-1.5B, see chapter 3.2 on page 14 of their research article [0]. [0] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948
- trissi1996 2y ago
  Which is not the same model, it's not R1 it's R1-Distill-Qwen-1.5B....
  - dingocat 2y ago
    A distinction they make clear and write extensively about on the model page, yes?
    - macki0 2y ago
      wheres that made clear in "ollama run deepseek-r1” the command to download/run the model?
      - dingocat 2y ago
        Which you have to go to the model page to find.

deleted 2y ago

[deleted]

av_conk 2y ago

I tried using ollama because I couldn't get ROCm working on my system with llama-cpp. Ollama bundles the ROCm libraries for you. I got around 50 tokens per second with that setup. I tried llama-cpp with the Vulkan backend and doubled the amount of tokens per second. I was under the impression ROCm is superior to Vulkan, so I was confused about the result. In any case, I've stuck with llama-cpp.

buyucu 2y ago
It depends on your GPU. Vulkan is well-supported by essentially all GPUs. AMD support ROCm well for their datacenter GPUs, but support for consumer hardware has not been as good.

deleted 2y ago

[deleted]

wkat4242 2y ago

That's a weird thing about Ollama yes. It took very long for them to support KV cache quantisation too (which drastically reduces the amount of VRAM needed for context!). Even though the underlying llama.cpp had offered it for ages. And they had it handed to them on a platter, someone had developed everything and submitted a patch. The developer of that patch even was about to give up as he had to constantly keep it up to date with upstream even though he was constantly being ignored. So he had no idea if it would ever be merged. They just seem to be really hesitant to offer new features. Eventually it was merged and it made a huge difference to people with low VRAM cards.

arvigeus 2y ago

Finally a response from the maintainers: https://github.com/ollama/ollama/pull/5059#issuecomment-2628002106 https://github.com/ollama/ollama/pull/5059#issuecomment-2628... I am the least qualified person to comment on that, but honestly their response made me to raise an eyebrow.

colorant 2y ago

There is ipex-llm support for Ollama on Intel GPU (https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/ollama_quickstart.md https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...)