4 ms·
GPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.
by lhl 3y ago
GPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.