Very rougly you can consider the Bs of a model as GBs of memory then it depends on the quantization level. Say for an 8B model: - FP16: 2x 8GB = 16GB - Q8: 1x

Very rougly you can consider the Bs of a model as GBs of memory then it depends on the quantization level. Say for an 8B model: - FP16: 2x 8GB = 16GB - Q8: 1x 8GB - Q4: 0.5x 8GB = 4GB It doesn't 100% neatly map like this but this gives you a rough measure. In top of this you need some more memory depending on the context length and some other stuff. Rationale for the calculation above: A model is basically a billions of variables with a floating number value. So the size of a model roughly maps to number of variables (weights) x word-precision of each variable (4, 8, 16bits..) You don't have to quantize all layers to the same precision this is why sometimes you see fractional quantizations like 1.58bits.