4 ms·
I mean the 35GB version of Falcon is maybe not something you'd want to use in production Also ironically, this version of Falcon will require CUDA.
by zwaps 3y ago
I mean the 35GB version of Falcon is maybe not something you'd want to use in production
Also ironically, this version of Falcon will require CUDA.
- brucethemoose2 3y agoIt might work on rocm? I am not sure about the status of GPTQ on rocm.
- lhl 3y agoGPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.