3 ms·
I've given up trying to locally use LLMs on AMD
by dankwizard 1y ago
I've given up trying to locally use LLMs on AMD
- lhl 1y agoBasically anything llama.cpp (Vulkan backend) should work out of the box w/o much fuss (LM Studio, Ollama, etc). The HIP backend can have a big prefill speed boost on some architectures (high-end RDNA3 for example). For everything else, I keep notes here: https://llm-tracker.info/howto/AMD-GPUs https://llm-tracker.info/howto/AMD-GPUs