4 ms·
Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?
by frodowtf 3y ago
Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?
- tyre 3y agoYou want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD. If you want to train a model, that’s a different story.
- MacsHeadroom 3y agoModel size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that. For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)