3 ms·
Unless its coincidence, the article seems to be underestimating what it likely takes, since 264GB is just exactly what it takes to load the weights of a 132B mo
by dragonwriter 3y ago
Unless its coincidence, the article seems to be underestimating what it likely takes, since 264GB is just exactly what it takes to load the weights of a 132B model (which the article says this is) in 16-bit, which is how models are normally released, but additional memory is usually needed varying with usable context.
But that basic requirement can be cut down significantly by quantization; 4-bit usually has only modest quality loss and drops to about 1/4 of the memory requirements, or about 66GB plus context. Which while also not typical consumer grade (especially for running on GPU), is perhaps within reach of doing CPU or mixed inference on a not-completely-unreasonable personal machine, and with an MoE architecture it might not even be too slow doing CPU or mixed inference.
- galaxyLogic 3y agoI wonder how the introduction of Neural Processing Units in processors by Interl and AMD might affect the performance equation.