4 ms·
It might work on rocm? I am not sure about the status of GPTQ on rocm.
by brucethemoose2 3y ago
It might work on rocm? I am not sure about the status of GPTQ on rocm.
- lhl 3y agoGPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.