3 ms·
It's great that Databricks is releasing an open model that matches or beats what was state of the art of the art just a year ago. This is excellent for competi
by loudmax 3y ago
It's great that Databricks is releasing an open model that matches or beats what was state of the art of the art just a year ago. This is excellent for competition, and it's amazing how far we've come in such a short time.
But to put something in perspective:
> The model requires ~264GB of RAM
So, open weights doesn't mean it's anywhere close to being able to run on consumer grade hardware.
- dragonwriter 3y agoUnless its coincidence, the article seems to be underestimating what it likely takes, since 264GB is just exactly what it takes to load the weights of a 132B model (which the article says this is) in 16-bit, which is how models are normally released, but additional memory is usually needed varying with usable context. But that basic requirement can be cut down significantly by quantization; 4-bit usually has only modest quality loss and drops to about 1/4 of the memory requirements, or about 66GB plus context. Which while also not typical consumer grade (especially for running on GPU), is perhaps within reach of doing CPU or mixed inference on a not-completely-unreasonable personal machine, and with an MoE architecture it might not even be too slow doing CPU or mixed inference.
- galaxyLogic 3y agoI wonder how the introduction of Neural Processing Units in processors by Interl and AMD might affect the performance equation.