llama.cpp, ExLlama, and MLC LLM have all had ROCm inferencing for months (here are a bunch of setup instructions I've written up, for Linux and Windows: https:/

llama.cpp, ExLlama, and MLC LLM have all had ROCm inferencing for months (here are a bunch of setup instructions I've written up, for Linux and Windows: https://llm-tracker.info/books/howto-guides/page/amd-gpus https://llm-tracker.info/books/howto-guides/page/amd-gpus ) - but I don't think that's the problem (and wouldn't drive lots of volume or having downstream impact in any case). The bigger problem is on the training/research support. Eg, here's no official support for AMD GPUs for bitsandbytes, and no support at all for FlashAttention/FA2 (nothing that 100K in hardware/grants to Dettmers or Dao's labs wouldn't fix I suspect). The real elephant though is that AMD still having the disconnect that lack of support for consumer cards and home/academic devs in general has been disastrous (while Nvidia supports CUDA on basically every single GPU they've made since 2010) - just last week there was this mindblowing thread where it turns out an AMD employee is paying out of pocket for AMD GPUs to support build/CI for drivers on Debian. I mean, WTF, that's stupidity that's beyond embarrassing and gets into negligence terriroty IMO: https://news.ycombinator.com/item?id=37665784 https://news.ycombinator.com/item?id=37665784