4 ms·

This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on G

This is not quite correct. Ollama must assess the state of Vulkan support and amount of available memory, then pick the fraction of the model to be hosted on GPU. This is not totally foolproof and will likely always need manual adjustment in some cases.

buyucu 2y ago
the work involved is tiny compared to the work llama.cpp did to get vulkan up and running. this is not rocket science.
- exe34 2y ago
  This sounds like it should be trivial to reproduce and extend - I look forward to trying out your repo!
  - buyucu 2y ago
    the owner of that PR has already forked ollama. try it out. I did and it works great.
    - paradite 2y ago
      I guess git and GitHub are working as intended then. This is not a sarcastic comment. I'm genuinely happy that this was the outcome.