3 ms·
Not in my experience. For an ML hobbyist, I don't have particularly strong hardware - around 40 GB of system RAM and a very small GPU with 8GB of RAM. I can run
by selfhoster11 3y ago
Not in my experience. For an ML hobbyist, I don't have particularly strong hardware - around 40 GB of system RAM and a very small GPU with 8GB of RAM. I can run all sizes of models, up to and including 70B (albeit slowly). 13 is definitely the most usable on this particular hardware, given the trade-off between model capabilities and generation speed.