3 ms·

supporting vulkan will help ollama reach the masses who don't have dedicated gpus from nvidia. this is such a low hanging fruit that it's silly how they are ac

by buyucu 2y ago

supporting vulkan will help ollama reach the masses who don't have dedicated gpus from nvidia. this is such a low hanging fruit that it's silly how they are acting.

lolinder 2y ago
As has been pointed out in this thread in a comment that you replied to (so I know you saw it) [0], Ollama goes to a lot of contortions to support multiple llama.cpp backends. Yes, their solution is a bit of a hack, but it means that the effort to adding a new back end is substantial. And again, they're doing those contortions to make it easy for people. Making it easy involves trade-offs. Yes, Ollama has flaws. They could communicate better about why they're ignoring PRs. All I'm saying is let's not pretend they're not doing anything complicated or difficult when no one has been able to recreate what they're doing. [0] https://news.ycombinator.com/item?id=42886933 https://news.ycombinator.com/item?id=42886933
- buyucu 2y ago
  This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.
  - lolinder 2y ago
    This is a change from your response to the comment that I linked to, where you said it was a good point. Why the difference? Maybe I should clarify that I'm not saying that the effort to enable a new backend is substantial, I'm saying that my understanding of that comment (the one you acknowledged made a good argument) is that the maintenance burden of having a new backend is substantial.
    - buyucu 2y ago
      I didn't say it was a good point. I said I disagree, but it's a respectable opinion I could imagine someone having.
      - lolinder 2y ago
        Okay, now we're playing semantics. "Reasonable argument" were your words. What changed between then and now to where the same argument is now "incorrect"?
        buyucu 2y ago
        I literally start the sentence with 'I disagree with this'
        lolinder 2y ago
        But you never said why, and you never said it was incorrect, you said it was a reasonable argument and then appealed to the popularity of the PR as the reason why you disagree. But now suddenly what I said is not just an argument you disagree with but is also incorrect. I've been genuinely asking for several turns of conversation at this point why what I said is incorrect. Why is it incorrect that the maintenance burden of maintaining a Vulkan backend would be a sufficient explanation for why they don't want to merge it without having to appeal to some sort of conspiracy with Nvidia?
        buyucu 2y ago
        llama.cpp already supports Vulkan. This is where all the hard work is at. Ollama hardly does anything on top of it to support Vulkan. You just check if the libraries are available, and get the available VRAM. That is all. It is very simple.
  - cmm 2y ago
    that PR doesn't actually work though -- it finds the Vulkan libraries and has some memory accounting logic, but the bits to actually build a Vulkan llama.cpp runner are not there. I'm not sure why its author deems it ready for inclusion. (I mean, the missing work should not be much, but it still has to be done)
    - buyucu 2y ago
      the pr was working 6 months ago and it has been rebased multiple times as the ollama team kept ignoring it and mainline moved. I'm using it right now.

by buyucu 2y ago

supporting vulkan will help ollama reach the masses who don't have dedicated gpus from nvidia. this is such a low hanging fruit that it's silly how they are acting.

lolinder 2y ago

As has been pointed out in this thread in a comment that you replied to (so I know you saw it) [0], Ollama goes to a lot of contortions to support multiple llama.cpp backends. Yes, their solution is a bit of a hack, but it means that the effort to adding a new back end is substantial. And again, they're doing those contortions to make it easy for people. Making it easy involves trade-offs. Yes, Ollama has flaws. They could communicate better about why they're ignoring PRs. All I'm saying is let's not pretend they're not doing anything complicated or difficult when no one has been able to recreate what they're doing. [0] https://news.ycombinator.com/item?id=42886933 https://news.ycombinator.com/item?id=42886933

buyucu 2y ago
This is incorrect. The effort it took to enable Vulkan was relatively minor. The PR is short and to be honest it doesn't do much, because it doesn't need to.
- lolinder 2y ago
  This is a change from your response to the comment that I linked to, where you said it was a good point. Why the difference? Maybe I should clarify that I'm not saying that the effort to enable a new backend is substantial, I'm saying that my understanding of that comment (the one you acknowledged made a good argument) is that the maintenance burden of having a new backend is substantial.
  - buyucu 2y ago
    I didn't say it was a good point. I said I disagree, but it's a respectable opinion I could imagine someone having.
    - lolinder 2y ago
      Okay, now we're playing semantics. "Reasonable argument" were your words. What changed between then and now to where the same argument is now "incorrect"?
      - buyucu 2y ago
        I literally start the sentence with 'I disagree with this'
        lolinder 2y ago
        But you never said why, and you never said it was incorrect, you said it was a reasonable argument and then appealed to the popularity of the PR as the reason why you disagree. But now suddenly what I said is not just an argument you disagree with but is also incorrect. I've been genuinely asking for several turns of conversation at this point why what I said is incorrect. Why is it incorrect that the maintenance burden of maintaining a Vulkan backend would be a sufficient explanation for why they don't want to merge it without having to appeal to some sort of conspiracy with Nvidia?
        buyucu 2y ago
        llama.cpp already supports Vulkan. This is where all the hard work is at. Ollama hardly does anything on top of it to support Vulkan. You just check if the libraries are available, and get the available VRAM. That is all. It is very simple.
- cmm 2y ago
  that PR doesn't actually work though -- it finds the Vulkan libraries and has some memory accounting logic, but the bits to actually build a Vulkan llama.cpp runner are not there. I'm not sure why its author deems it ready for inclusion. (I mean, the missing work should not be much, but it still has to be done)
  - buyucu 2y ago
    the pr was working 6 months ago and it has been rebased multiple times as the ollama team kept ignoring it and mainline moved. I'm using it right now.