17 ms·

Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.

Const-me 2y ago
The integrated GPU of the 5700G uses old architecture from 2017, this one: https://en.wikipedia.org/wiki/Radeon_RX_Vega_series https://en.wikipedia.org/wiki/Radeon_RX_Vega_series Pretty sure it does not support ROCm. BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.
- fazkan 2y ago
  @Const-me according to this it should work, https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216
fazkan 2y ago
haven't tested it, but it should according to https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216 You just need to update the version check here https://github.com/slashml/amd_inference/blob/4b9ec069c4b2accc764eaa677eb43e2d1d5ff1f3/src/amd_setup.py#L11 https://github.com/slashml/amd_inference/blob/4b9ec069c4b2ac... feel free to open an issue, with the requirements and we will test it.
Chordless 2y ago
I'm able to run Ollama and llama.cpp on my Ryzen 4600G APU following this guide: https://agieverywhere.com/apuguide/AMDAPU/APU_Linux https://agieverywhere.com/apuguide/AMDAPU/APU_Linux Your APU should be similar, just faster. There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU: HSA_OVERRIDE_GFX_VERSION=9.0.0 HSA_ENABLE_SDMA=0 Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.

17 ms·

Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.

by phkahler 2y ago

Const-me 2y ago
The integrated GPU of the 5700G uses old architecture from 2017, this one: https://en.wikipedia.org/wiki/Radeon_RX_Vega_series https://en.wikipedia.org/wiki/Radeon_RX_Vega_series Pretty sure it does not support ROCm. BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/MistralChat https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.
- fazkan 2y ago
  @Const-me according to this it should work, https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216
fazkan 2y ago
haven't tested it, but it should according to https://github.com/ROCm/ROCm/issues/2216 https://github.com/ROCm/ROCm/issues/2216 You just need to update the version check here https://github.com/slashml/amd_inference/blob/4b9ec069c4b2accc764eaa677eb43e2d1d5ff1f3/src/amd_setup.py#L11 https://github.com/slashml/amd_inference/blob/4b9ec069c4b2ac... feel free to open an issue, with the requirements and we will test it.
Chordless 2y ago
I'm able to run Ollama and llama.cpp on my Ryzen 4600G APU following this guide: https://agieverywhere.com/apuguide/AMDAPU/APU_Linux https://agieverywhere.com/apuguide/AMDAPU/APU_Linux Your APU should be similar, just faster. There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU: HSA_OVERRIDE_GFX_VERSION=9.0.0 HSA_ENABLE_SDMA=0 Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.