3 ms·
I don't like the cynicism here. * Yes, performance numbers would be helpful. * No, this is not new, and for most applications, not practical. The basic point
by blagie 3y ago
I don't like the cynicism here.
* Yes, performance numbers would be helpful.
* No, this is not new, and for most applications, not practical.
The basic point is that yesterday, I couldn't run a large model locally. Today, I can, in about one pip install and 20 lines of code. Yay!
There are a lot of things which are convenient to be able to test locally. YES it takes overnight, but if you're trying to confirm if there's a business case, that's often good enough. For a single compute (or even just a few), this is also simpler than spinning up a cloud machine.
- brucethemoose2 3y agoYou can essentially already do this with llama.cpp and ssd swap, much more quickly. Technically it only does the prompt processing and a few layers on the GPU (if that), but honestly that is better, just to avoid all the transfers over the GPU bus. And you can do it in MLC, in your IGP, if you have enough CPU RAM to fit the model. Running a 70B very slowly is nothing new. To be blunt, this strategy is a bad idea in the face of newer implementations.