4 ms·
Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mo
by mrtksn 3y ago
Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip?
I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as every device can become smart. Remember how some British researchers made a self driving car using an LLM? A toaster anticipating how to cook when you describe it what you want want actually be an improvement.
- tome 3y agoI don't think we've put a GroqChip in a mobile device yet. Interesting idea!
- wmf 3y agoI assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.
- mrtksn 3y agoWell, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output. Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run at acceptable speed?
- wmf 3y agoThe thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.
- mrtksn 3y agoCare to explain? Are they using 10x energy for 10x speed improvements?
- wmf 3y agoThey're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.
- mrtksn 3y ago100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.
- tome 3y agoYes, we pipeline requests so multiple users are being handled by the same hardware at one time.
- mrtksn 3y agoThanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?
- tome 3y agoThe industry as a whole is a very long way away from that. The power requirements are too high for mobile.
- pptr 3y agoI think the limitation is chip size/cost. SRAM is a lot less dense than RAM. According to Google this is typically used for registers and caches, which are only megabytes large.
- frozenport 3y agoYeah just offload the compute onto the cloud.
- mrtksn 3y agoIts too unreliable, too restricted and too not-private. ChatGPT stopped processing images for me, trying to get help but support doesn't appear to be very fast, they asked for more info but not heard back since. Its too restricted, can't do anything on hard topics. It doesn't work when you try to work out exploits or dangers in a system for example. Its not private, they say they don't train on API requests but companies steer clear when it comes to send sensitive data.
- frozenport 3y agoThe model being too restrictive does seem to be a good point. Do you think there are less restrictive models hosted on poe.com?
- SahAssar 3y ago> Remember how some British researchers made a self driving car using an LLM? No? Do you mean actual, full self driving on normal roads in traffic?
- mrtksn 3y agoYes, IIRC they reason on the car actions using LLMs. They still use image processing but once you identify the objects in the scene, the LLM interprets and decides what to do with the car. I'm not sure which one was it though(Ghost Autonomy maybe?).
- SahAssar 3y agoDo you have a source? Because that actually, properly working would be headline global news and would value the company in the billions.
- mrtksn 3y agoIt was discussed here on HN, that's how I know about it. I found a few things when searched around but not sure which one was the one I recall. Anyway, here is a video from one: https://www.youtube.com/watch?v=C2rbym6bXM0 https://www.youtube.com/watch?v=C2rbym6bXM0 Here is a paper discussing something similar: https://arxiv.org/abs/2307.07162 https://arxiv.org/abs/2307.07162
- SahAssar 3y agoThe description for that video says > Ghost Autonomy’s MLLM-based capabilities are currently in development. These video and image examples show MLLM-based analysis of driving scenes captured from Ghost vehicles driving in both autonomous and conventional mode. MLLM-based reasoning is not yet being returned to the car to impact actual driving maneuvers. So the model discussed is not doing any driving whatsoever. This is not self-driving at any level.
- 3y ago