3 ms·

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For e

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

leetharris 3y ago
For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.
- frodowtf 3y ago
  Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?
  - tyre 3y ago
    You want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD. If you want to train a model, that’s a different story.
  - MacsHeadroom 3y ago
    Model size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that. For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
- Aerroon 3y ago
  To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way. Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.
  - PcChip 3y ago
    Let's say I wanted to use one of their quantized models with this OpenLLM project. How would I do that?
    - Aerroon 3y ago
      Sorry, I don't know. I suspect that it's not possible (yet?). OpenLLM lists a bunch of models in the Github Readme. I think the best way would be to use those for now.
luckystarr 3y ago
From the HuggingFace page: > You will need at least 85-100GB of memory to swiftly run inference with Falcon-40B. So it may be possible with less (swap around method), though not as efficiently and also slower.
philoko 3y ago
For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.
- aarnphm 3y ago
  Currently on main, 8bit and 4bit quant is supported One can simply do ```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4``` Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization

leetharris 3y ago

For falcon 40b you probably need an A100 40gb or so. Every model is drastically different. If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

frodowtf 3y ago
Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?
- tyre 3y ago
  You want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD. If you want to train a model, that’s a different story.
- MacsHeadroom 3y ago
  Model size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that. For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)
Aerroon 3y ago
To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML The number after q determines how many bits the weights are. Eg q4 means that is 4-bit. If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way. Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.
- PcChip 3y ago
  Let's say I wanted to use one of their quantized models with this OpenLLM project. How would I do that?
  - Aerroon 3y ago
    Sorry, I don't know. I suspect that it's not possible (yet?). OpenLLM lists a bunch of models in the Github Readme. I think the best way would be to use those for now.

luckystarr 3y ago

From the HuggingFace page: > You will need at least 85-100GB of memory to swiftly run inference with Falcon-40B. So it may be possible with less (swap around method), though not as efficiently and also slower.

philoko 3y ago

For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.

aarnphm 3y ago
Currently on main, 8bit and 4bit quant is supported One can simply do ```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4``` Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization