4 ms·
Do you know which model size can be run with a 3090?
by epups 3y ago
Do you know which model size can be run with a 3090?
- ianpurton 3y agoAssuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion. Note: I have no practical experience of this, just reading around.
- MacsHeadroom 3y ago24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.
- redox99 3y ago3090 can handle the ~30B models quantized to 4 bits.