4 ms·
The very opening line says > I'm glad you're happy with the fact that LLaMA 30B (a 20gb file) can be evaluated with only 4gb of memory usage! The line you quo
by 4bpp 3y ago
The very opening line says
> I'm glad you're happy with the fact that LLaMA 30B (a 20gb file) can be evaluated with only 4gb of memory usage!
The line you quoted occurs in a context where it is also implied that the low memory usage is a fact, and there might only be a bug insofar as that the model is being evaluated incorrectly. This is what is entailed by the assertion that it "is" sparse: that is, a big fraction of the parameters are not actually required to perform inference on the model.
- wpietri 3y agoI think you are making a lot of soup from very little meat. I read those links the same way mtlynch read them. I think you're looking for a perfection of phrasing that is much more suited to peer-reviewed academic papers than random tweets and GitHub comments taken from the middle of exploring something. Seeing your initial comment and knowing little about the situation, I was entirely prepared to share your skepticism. But at this point I'm much more skeptical of you.