3 ms·
Aren't small local models worse efficiency-wise? It means that every person must have a powerful enough machine to power a small model, and we are very, very fa
by gbalduzzi 4mo ago
Aren't small local models worse efficiency-wise? It means that every person must have a powerful enough machine to power a small model, and we are very, very far away from that.
The best solution, from an efficiency point of view, is to use smaller models on datacenters, requiring much less of them.
- pbmonster 4mo agoThere's an efficiency sweet spot where hardware that people have anyway gets a higher percentage of load. MacBooks have a lot of memory and a lot of FLOPs. They mostly sit unused all day. Yes, the excess energy use will be higher than a GPU in a datacenter doing the same work, but you have to generate an absurd amount of tokens before the dollar-efficiency catches up with the MacBook.
- gbalduzzi 4mo agoYou need to have a 3k dollar machine available though, I think you are overestimating how many people have access to it