4 ms·

> qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better of

> qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better off running LLMs on your phone. Even an older phone. Or literally whatever device you are reading HN on. But if you insist, the runtime you want in MLC-LLM's Vulkan runtime.

refulgentis 3y ago
This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
- Jimbabwe 3y ago
  Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
  - brucethemoose2 3y ago
    The OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.
- brucethemoose2 3y ago
  Yeah I feel this way about llama.cpp too. Maybe not that exact wording, but its not the end all be all llama runtime like everyone seems to think it is.
Jimbabwe 3y ago
Thanks, I’ll look into it! Especially if the llama.cpp route is a dud, like the other response says it will be. My little qnap clunker handles all the self hosting stuff I throw at it, but I won’t be surprised if it simply has met its match

refulgentis 3y ago

This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec

Jimbabwe 3y ago
Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
- brucethemoose2 3y ago
  The OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.
brucethemoose2 3y ago
Yeah I feel this way about llama.cpp too. Maybe not that exact wording, but its not the end all be all llama runtime like everyone seems to think it is.

Jimbabwe 3y ago

Thanks, I’ll look into it! Especially if the llama.cpp route is a dud, like the other response says it will be. My little qnap clunker handles all the self hosting stuff I throw at it, but I won’t be surprised if it simply has met its match