3 ms·
Yes, but would be massive overkill. Falcon 40B takes ~35GB of VRAM to load now, and probably less in the future with better quant from llama.cpp and such. http
by brucethemoose2 3y ago
Yes, but would be massive overkill. Falcon 40B takes ~35GB of VRAM to load now, and probably less in the future with better quant from llama.cpp and such.
https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ
Large context size is becoming less of an issue now too.
But maybe it would be good for batched inference?
- ilaksh 3y agoI was thinking maybe RunPod or Modal could use it to handle multiple containers per CPU, possibly including some that are much lighter like stable diffusion.
- brucethemoose2 3y agoIf its multiple llm client requests, batched inference would be massively more efficient. And for multiple services... Probably just better to run multiple cheap instances and/or load dynamically? The MI300 is super expensive.
- ilaksh 3y agoHow expensive is it compared to A100?
- brucethemoose2 3y agoThat is an excellent question. In theory its more expensive to produce than an H100, but in practice... shrug. But I was thinking 3090 or lesser tesla/Quadro instances would be sufficient.
- zwaps 3y agoI mean the 35GB version of Falcon is maybe not something you'd want to use in production Also ironically, this version of Falcon will require CUDA.
- brucethemoose2 3y agoIt might work on rocm? I am not sure about the status of GPTQ on rocm.
- lhl 3y agoGPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.