3 ms·

Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone

Could it be that supporting multiple platforms open up more support tickets and adds more work to keep the software working on those new platforms? As someone who built apps for Windows, Linux, macOS, iOS and Android, it is not trivial to ensure your new features or updates work on all platforms, and you have to deal with deprecations.

buyucu 2y ago
ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.
- paradite 2y ago
  Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.
  - buyucu 2y ago
    check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.
    - fkyoureadthedoc 2y ago
      If it's "just" a friendly front and, why doesn't llama.cpp just drop one themselves? Do they actually care about the situation, or are random people just mad on their behalf?
    - paradite 2y ago
      At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.
      - zozbot234 2y ago
        The Vulkan-specific flags are needed (1) to set up the llama.cpp build options when building Ollama w/ Vulkan support - which apparently is still a challenge with the current PR, if the latest comments on the GitHub page are accurate; also (2) to pick how many model layers should be run on the GPU, depending on available GPU memory. Llama.cpp doesn't do that for you, you have to set that option yourself or just tell it to move "everything", which often fails with an error. (Finding the right amount is actually a trial-and-error process which depends on the model, quantization and also varies depending on how much context you have in the current conversation. If you have too many layers loaded and too little GPU memory, a large context can result in unpredictable breakage.)
        paradite 2y ago
        Thanks a lot for the explanation. If I can ask one more question, why don't Ollama use binaries of pre-built llama.cpp with Vulkan support directly?
- zozbot234 2y ago
  This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on GPU. This is not totally foolproof and will likely always need manual adjustment in some cases.
  - buyucu 2y ago
    the work involved is tiny compared to the work llama.cpp did to get vulkan up and running. this is not rocket science.
    - exe34 2y ago
      This sounds like it should be trivial to reproduce and extend - I look forward to trying out your repo!
      - buyucu 2y ago
        the owner of that PR has already forked ollama. try it out. I did and it works great.
        paradite 2y ago
        I guess git and GitHub are working as intended then. This is not a sarcastic comment. I'm genuinely happy that this was the outcome.
geerlingguy 2y ago
They already support ROCm, which probably introduces 10x more support requests than Vulkan would!

buyucu 2y ago

ollama is not doing anything. llama cpp does all that work. ollama is just a small wrapper on top.

paradite 2y ago
Ok assuming what you said is correct, why wouldn't Ollama then be able to support Vulkan by default out of the box? Sorry I'm not sure what's the relationship exactly between the two projects. This is a genuine questions, not a troll question.
- buyucu 2y ago
  check the PR, it's a very short one. It's not more complicated than setting a compile time flag. I have no idea why they have been ignoring it. Ollama is just a friendly front end for llama.cpp. It doesn't have to do any of those things you mentioned. Llama.cpp does all that.
  - fkyoureadthedoc 2y ago
    If it's "just" a friendly front and, why doesn't llama.cpp just drop one themselves? Do they actually care about the situation, or are random people just mad on their behalf?
  - paradite 2y ago
    At the risk of being pedantic (I don't know much about C++ and I'm genuinely curious), if Ollama is really just a wrapper around Llama.cpp, why would it need the Vulkan specific flags? Shouldn't it just call Llama.cpp and let Llama.cpp handle the flags internally within Llama.cpp? I'm thinking from an abstraction layer perspective.
    - zozbot234 2y ago
      The Vulkan-specific flags are needed (1) to set up the llama.cpp build options when building Ollama w/ Vulkan support - which apparently is still a challenge with the current PR, if the latest comments on the GitHub page are accurate; also (2) to pick how many model layers should be run on the GPU, depending on available GPU memory. Llama.cpp doesn't do that for you, you have to set that option yourself or just tell it to move "everything", which often fails with an error. (Finding the right amount is actually a trial-and-error process which depends on the model, quantization and also varies depending on how much context you have in the current conversation. If you have too many layers loaded and too little GPU memory, a large context can result in unpredictable breakage.)
      - paradite 2y ago
        Thanks a lot for the explanation. If I can ask one more question, why don't Ollama use binaries of pre-built llama.cpp with Vulkan support directly?
zozbot234 2y ago
This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on GPU. This is not totally foolproof and will likely always need manual adjustment in some cases.
- buyucu 2y ago
  the work involved is tiny compared to the work llama.cpp did to get vulkan up and running. this is not rocket science.
  - exe34 2y ago
    This sounds like it should be trivial to reproduce and extend - I look forward to trying out your repo!
    - buyucu 2y ago
      the owner of that PR has already forked ollama. try it out. I did and it works great.
      - paradite 2y ago
        I guess git and GitHub are working as intended then. This is not a sarcastic comment. I'm genuinely happy that this was the outcome.

geerlingguy 2y ago

They already support ROCm, which probably introduces 10x more support requests than Vulkan would!