3 ms·

Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13

• brucethemoose2 3y ago
  The OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.