6 ms·

There’s probably a better place to ask this highly specific technical question, but I’m avoiding Reddit these days so just throwing it out I guess. I’ve been tr

There’s probably a better place to ask this highly specific technical question, but I’m avoiding Reddit these days so just throwing it out I guess. I’ve been trying to run these in a container but it’s verrrry slow, I believe, because of the lack of gpu help. All the instructions I find are for nvidia gpus and my server is a qnap tvs-473e with an embedded amd cpu/gpu (I know, I know). The only good news is that I’ve upgraded the ram to 32gb, and I have a 1TB ssd. Any idea of how I can get my own self-hosted LLM/chat service on this funky hardware? The nvidia/docker option requires installing the nvidia runtime alongside docker, but I can’t find an amd equivalent. Thanks. Sorry for the wall of text nobody cares about.

brucethemoose2 3y ago
> qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better off running LLMs on your phone. Even an older phone. Or literally whatever device you are reading HN on. But if you insist, the runtime you want in MLC-LLM's Vulkan runtime.
- refulgentis 3y ago
  This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
  - Jimbabwe 3y ago
    Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
    - brucethemoose2 3y ago
      The OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.
  - brucethemoose2 3y ago
    Yeah I feel this way about llama.cpp too. Maybe not that exact wording, but its not the end all be all llama runtime like everyone seems to think it is.
- Jimbabwe 3y ago
  Thanks, I’ll look into it! Especially if the llama.cpp route is a dud, like the other response says it will be. My little qnap clunker handles all the self hosting stuff I throw at it, but I won’t be surprised if it simply has met its match
curious_cat_163 3y ago
You'll want to try llama.cpp [1]. The set of models that it can support is expanding [2]. Folks have also written services [3] that wrap around it. [1] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python
- Jimbabwe 3y ago
  Thanks! I was just following the thread about their recent addition of the OpenCl support and was on the verge of trying it out last weekend. I’ll definitely continue once I’m home again!
  - atcalan 3y ago
    It runs up to 13b models nicely on a 16g orange pi 5 fyi.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
loxias 3y ago
> my server is a qnap tvs-473e with an embedded amd cpu/gpu That's your problem. I googled and it looks like one of these all-in-one appliances like a drobo or whatever's popular these days. That's not a server. (At least, I wouldn't call it a server. It's an all-in-one appliance, or toy, depending on perspective) And yegods, that price... Spend $500, get an actual computer, not some priced up appliance, and you'll have a much better time. Regardless of if you spend it on more CPU or more GPU. You can get a used computer off ebay for $100 and shove a $400 graphics card in it. Or maybe get a ryzen 7 7700x, I'm looking at a mobo+cpu combo with that for $500 right now. Finally, to make sure this response does contain a answer to what you asked: ;-) if you can run this stuff in a container on your appliance already, but it's very slow, congrats! I'd call that a win. I looked up the chip, the RX-421BD, it's of similar power as an Athelon circa 2017. I think my router might have more compute power. You _do_ have those 512 shader cores, given effort, you could try and get them to do something useful. But I wouldn't assume it's possible (well, maybe you don't mind writing your own shaders ;-)). Just because the chip has "some gpu" doesn't mean it has "the right kind of gpu you'd need to hijack for lots of matrix multiplies, without writing the assembly yourself". Sorry this isn't more helpful, but it's the truth.
- KeplerBoy 3y ago
  Something to add: if your NAS-Server is advertised with it's dual 4K capable HDMI ports, you know you're looking at a consumer toy targeted at people who hoard (pirated) media and not for serious stuff.
  - loxias 3y ago
    Hah! I hadn't even noticed that, I just saw "I could build that for 1/4th the cost"+"wtf, only 4 drives?" Stuff like this prompts a dual response in me. :) It always gives me a strong urge to educate "it's not that hard, and it's fun" to build it yourself. AND it always makes me kick myself for not commercializing the expandable media servers I started building in the early 2000s, for me, for the dorm, for my friends, i.e. exactly the people you identify. :)
- Jimbabwe 3y ago
holoduke 3y ago
Save yourself some time and buy a 4090 if you really want to be high tier (consumer range) you will have a much faster experience. Not only with text. Also stable diffusion etc
- brucethemoose2 3y ago
  Or better yet, a 3090 for very similar performance at half the price.
  - atcalan 3y ago
    Why not both? Or two 3090s for that sweet, sweet nvlink for fine tuning?

brucethemoose2 3y ago

> qnap tvs-473e Specs say this runs an AMD RX-421BD. This is a 2015 AMD CPU with 2 bulldozer cores and a tiny IGP. ...To be blunt, you would be much better off running LLMs on your phone. Even an older phone. Or literally whatever device you are reading HN on. But if you insist, the runtime you want in MLC-LLM's Vulkan runtime.

refulgentis 3y ago
This. Sibling llama.cpp comment is standard "I know llama.cpp, I assume that's 80% of the universe instead of .8%, and I assume that's all anyone needs. So I know just enough to be dangerous with ppl looking for advice". You'll see it over and over again when you're looking for help, be careful, it's 100% a blind alley in your case. It's very likely you'll be disappointed by MLC as well, simultaneously it's your only real option. You definitely won't hit 1 tkn/sec, and honestly, id bet 0.1 tkn / sec
- Jimbabwe 3y ago
  Thanks. Yeah I’ve noticed that llama.cpp is definitely the recommendation du jour, but this thread gave me hope: https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_opencl_is_merged_amd_gpus_now_work_with https://www.reddit.com/r/LocalLLaMA/comments/13p8zq2/update_... Sorry for reddit link.
  - brucethemoose2 3y ago
    The OpenCL backend is not that great. Its kinda slow on GPUs, completely unusable on IGPs, and lags behind the CUDA, Metal and rOCM backends feature wise. To be blunt, there isn't much interest in support outside of apple/nvidia. There is a WIP Vulkan backend, but (last I checked) progress is slow and its not optimzed for IGPs either. MLC-LLM is much more promising once its features get fleshed out, as it "inherits" support for many devices from its Apache TVM backend.
- brucethemoose2 3y ago
  Yeah I feel this way about llama.cpp too. Maybe not that exact wording, but its not the end all be all llama runtime like everyone seems to think it is.
Jimbabwe 3y ago
Thanks, I’ll look into it! Especially if the llama.cpp route is a dud, like the other response says it will be. My little qnap clunker handles all the self hosting stuff I throw at it, but I won’t be surprised if it simply has met its match

curious_cat_163 3y ago

You'll want to try llama.cpp [1]. The set of models that it can support is expanding [2]. Folks have also written services [3] that wrap around it. [1] https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python

Jimbabwe 3y ago
Thanks! I was just following the thread about their recent addition of the OpenCl support and was on the verge of trying it out last weekend. I’ll definitely continue once I’m home again!
- atcalan 3y ago
  It runs up to 13b models nicely on a 16g orange pi 5 fyi.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

loxias 3y ago

> my server is a qnap tvs-473e with an embedded amd cpu/gpu That's your problem. I googled and it looks like one of these all-in-one appliances like a drobo or whatever's popular these days. That's not a server. (At least, I wouldn't call it a server. It's an all-in-one appliance, or toy, depending on perspective) And yegods, that price... Spend $500, get an actual computer, not some priced up appliance, and you'll have a much better time. Regardless of if you spend it on more CPU or more GPU. You can get a used computer off ebay for $100 and shove a $400 graphics card in it. Or maybe get a ryzen 7 7700x, I'm looking at a mobo+cpu combo with that for $500 right now. Finally, to make sure this response does contain a answer to what you asked: ;-) if you can run this stuff in a container on your appliance already, but it's very slow, congrats! I'd call that a win. I looked up the chip, the RX-421BD, it's of similar power as an Athelon circa 2017. I think my router might have more compute power. You _do_ have those 512 shader cores, given effort, you could try and get them to do something useful. But I wouldn't assume it's possible (well, maybe you don't mind writing your own shaders ;-)). Just because the chip has "some gpu" doesn't mean it has "the right kind of gpu you'd need to hijack for lots of matrix multiplies, without writing the assembly yourself". Sorry this isn't more helpful, but it's the truth.

KeplerBoy 3y ago
Something to add: if your NAS-Server is advertised with it's dual 4K capable HDMI ports, you know you're looking at a consumer toy targeted at people who hoard (pirated) media and not for serious stuff.
- loxias 3y ago
  Hah! I hadn't even noticed that, I just saw "I could build that for 1/4th the cost"+"wtf, only 4 drives?" Stuff like this prompts a dual response in me. :) It always gives me a strong urge to educate "it's not that hard, and it's fun" to build it yourself. AND it always makes me kick myself for not commercializing the expandable media servers I started building in the early 2000s, for me, for the dorm, for my friends, i.e. exactly the people you identify. :)
Jimbabwe 3y ago

holoduke 3y ago

Save yourself some time and buy a 4090 if you really want to be high tier (consumer range) you will have a much faster experience. Not only with text. Also stable diffusion etc

brucethemoose2 3y ago
Or better yet, a 3090 for very similar performance at half the price.
- atcalan 3y ago
  Why not both? Or two 3090s for that sweet, sweet nvlink for fine tuning?