3 ms·

Does ollama support loaders other than llamacpp? I'm using oobabooga with exllama2 to run exl2 quants on a dual NVIDIA gpu, and nothing else seems to beat perfo

by lxe 3y ago

Does ollama support loaders other than llamacpp? I'm using oobabooga with exllama2 to run exl2 quants on a dual NVIDIA gpu, and nothing else seems to beat performance of it.

_ink_ 3y ago
I tried that, but failed to get the GPU split working. Do you have a link on how to do that?
- lxe 3y ago
  Do what exactly? I have no issues with GPU split on oobabooga with either exl2 or gguf.