4 ms·
Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billio
by ianpurton 3y ago
Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.
Note: I have no practical experience of this, just reading around.
- MacsHeadroom 3y ago24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.