3 ms·
Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly gene
by mrtksn 3y ago
Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output.
Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run at acceptable speed?
- wmf 3y agoThe thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.
- mrtksn 3y agoCare to explain? Are they using 10x energy for 10x speed improvements?
- wmf 3y agoThey're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.
- mrtksn 3y ago100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.
- tome 3y agoYes, we pipeline requests so multiple users are being handled by the same hardware at one time.
- mrtksn 3y agoThanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?
- tome 3y agoThe industry as a whole is a very long way away from that. The power requirements are too high for mobile.
- pptr 3y agoI think the limitation is chip size/cost. SRAM is a lot less dense than RAM. According to Google this is typically used for registers and caches, which are only megabytes large.