5 ms·
4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/fa
by mikeravkine 3y ago
4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ
Inference is very slow right now but it works!
- takeiteasyy 3y agoThis will require a cpp port first to run on e.g. apple silicon?
- wsgeorge 3y agoNo, but to benefit from the ggml ecosystem, yes. Someone's taking a stab at it: https://github.com/nikisalli/falcon.cpp https://github.com/nikisalli/falcon.cpp
- rovr138 3y agoThis is using huggingface. It's python. There's no reason it wouldn't run on arm. https://huggingface.co/tiiuae/falcon-40b https://huggingface.co/tiiuae/falcon-40b I'll also add that the fact something is in C++ doesn't mean it will run on arm or that it can be compiled in it.
- bongobingo1 3y agoThe hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
- airgapstopgap 3y agoMulti-Query Attention, used here, should make 40B inference viable on systems where even 33B LLaMA with Multi-Head is basically unusable, so sometimes improvements still come from software optimization (it's no free lunch though).
- BonoboIO 3y agoWill be interesting to see if someone comes up with an ASIC or FPGA.
- groby_b 3y agoWhy would you think there's an ASIC/FPGA design that significantly improves over GPUs specifically targeted at running large models already? Where's the win? The fundamental limit for hardware acceleration are number of gates you can squeeze on a die, right now. (Or, alternatively. memory bandwidth)
- code_witch_sam 3y agoanalog chips like what MythicAI is developing seem like the next obvious leap forward for deploying inferences broadly. ASIC/FPGA wouldn't be much different than a GPU. ASIC seems like a brittle solution https://www.youtube.com/watch?v=GVsUOuSjvcg https://www.youtube.com/watch?v=GVsUOuSjvcg
- wrs 3y agoThe minimum size of a model with equivalent performance is unknown (like so much else about LLMs), so requirements could be reduced.
- Tepix 3y agoI think there is still lots room for improvement to reduce hardware requirements, such as 3-bit quantization or pruning weights from sparse models. But if you're willing to spend $1500 on two used RTX 3090, it's the sweet spot in terms of the ability to run large models right now. Everything beyond that is much more expensive.
- dzhiurgis 3y agois there a model version where it can be deployed using huggingface's tool?
- sireat 3y agoHow is it possible to run this model on 2x 4090s ? I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/nvidia-kills-off-nvlink-on-rtx-4090 https://www.windowscentral.com/hardware/computers-desktops/n...
- Timshel 3y agoProbably this : https://huggingface.co/docs/accelerate/usage_guides/big_modeling https://huggingface.co/docs/accelerate/usage_guides/big_mode...
- MacsHeadroom 3y agoWithout NVLink different layers are loaded onto individual cards and only one card often has to wait for the other card, slowing down generation. It's still faster than CPU offloading. You can even mix and match GPUs like a 4090 + 3060.
- brianjking 3y agoHow can it be deployed on a Huggingface Space or Colab notebook?