3 ms·
For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is
by leetharris 3y ago
For falcon 40b you probably need an A100 40gb or so.
Every model is drastically different.
If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
- frodowtf 3y agoDo you know any "standard" way or measures to determine the approximate hardware requirements of a model?
- tyre 3y agoYou want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD. If you want to train a model, that’s a different story.
- MacsHeadroom 3y agoModel size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that. For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
- Aerroon 3y agoTo add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way. Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.