4 ms·
This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I kn
by refulgentis 3y ago
This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice".
You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
- Jimbabwe 3y agoThanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
- brucethemoose2 3y agoThe OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.
- brucethemoose2 3y agoYeah I feel this way about llama.cpp too. Maybe not that exact wording, but its not the end all be all llama runtime like everyone seems to think it is.