7 ms·
Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?
by s1k3s 4y ago
Isn't GPT so complex that it requires hundreds of GB of ram to be used? How's it going to run on iphone?
- endisneigh 4y agoyes, there are billions of parameters necessary. but large language models only came out about 5 years ago. I'm confident 5 years from now the parameters necessary to get gpt-4 performance will be decreased orders of magnitude. at the very least, even if that's not the case, inference will be drastically less gpu heavy by then I suspect.
- s1k3s 4y agoWait, so there's a way to make a model as smart as GPT but with less parameters? Isn't that why it's so good?
- endisneigh 4y agothat's a complicated question to answer. what I'd say is that more parameters makes the model more robust, but there are diminishing returns. optimizations are under way
- month13 4y agoThis is an older paper, but DeepMind alleges in their Chinchilla paper that far better performance can be extracted with fewer parameters; quote "We find that current large language models are significantly under-trained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant." It's difficult to evaluate a LLM's performance as it's all qualitative, but Meta's LLaMA has been doing quite well, at even 13B parameters.
- astrange 4y agoChinchilla is aimed at finding a cost-performance tradeoff as well, not the optimal amount of training. If cost is no barrier because it'll be used forever, then probably there's no amount of training that's good enough.
- monocasa 4y agoThe rumor I've heard is that GPT4 didn't meaningfully increase the parameter count versus GPT3.5, but instead focused on training and structural improvements.
- jazzyjackson 4y agoYou can train a small model to behave like the large model at a subset of tasks.
- chatmasta 4y agoThere will also be hardware improvements (as always) and ASIC chips specifically designed for running this kind of model. For example, see this "Optical Transformers" paper [0] and its HN discussion [1] from last month. [0] https://arxiv.org/abs/2302.10360 https://arxiv.org/abs/2302.10360 [1] https://news.ycombinator.com/item?id=34905210 https://news.ycombinator.com/item?id=34905210
- tlrobinson 4y agoI could also imagine a sort of two-tier approach, where the on-device model can handle the majority of queries, but recognize when it should pass the query on to a larger model running in the cloud.
- dachryn 4y agodont underestimate how many of those parameters are actually necessary to support multiple languages. If you focus on english only, this can easily reduce the paramters 5fold
- spacebanana7 4y agoCould you explain how supporting multiple languages increases the parameter count so much? I'm genuinely curious. LLMs seem to be comfortable with hundreds of programming languages, DSLs and application specific syntaxes so how does supporting a couple more natural languages become so expensive? I see how more training data would be needed, but I don't understand how that maps to a greater parameter count.
- garblegarble 4y agoI'm similarly skeptical, but that said I'm running 30B parameter LLMs on my 32GB M1 Macbook Pro every day now. The trick is quantising them down to 4 (or even 3) bit, it's possible to massively reduce the memory requirements. Have a look at[1] The devs working on llama.cpp have been discussing ways to further reduce the memory requirements by mmapping the large weights files (I thought LLMs mutated the weights as they run inference, but they clearly know more than me about the internals), bringing it within reach of phone memory. So, iPhones are not as far off the computational capacity to run these models as you'd think. Memory (and to a greater extent, battery and cooling) are the limiting factors. iPads even less so, given they run M1 chips and have much larger batteries & much more RAM https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- qumpis 4y agoOfftopic, but for what purpose are you running llms locally (especially everyday)? My understanding was that the prompting requires to make them work at all was too great.
- garblegarble 4y agoA little bit of research, a little bit of actual useful tasks - I'm interested in summarisation, which alpaca is decent at (even compared to existing summarisation-specific models I've tried) My other motivation is making sure I understand what offline LLMs can do... while I use GPT-3 and 4 extensively, I don't want to send something over the wire if I don't have to (e.g. if I can summarise e-mails locally, I'd rather do that than send them to OpenAI). It's also surprisingly good at defining things if I'm somewhere with no internet connectivity and want to look something up (although obviously that's not really what it's good at & hallucination risks abound)
- johnthuss 4y agoWhat prompt are you using for summarization? I’ve tried several variations without consistent results.
- pornel 4y agoHundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds. It's also likely that memory requirements will be shrunk through software and ML improvements.
- AgentOrange1234 4y agoI don’t think RAM on phones has been increasing exponentially? For good reason — it uses a lot of power.
- pornel 4y agoIt has been increasing. In iPhones went from 128MB to 6GB, and there are Android phones with 18GB now. RAM hasn't been increasing as steeply as it could, but if there's a strong use-case for it, it may happen. Also consider that Apple is in control of the whole chipset and software, so they could implement things like turning the extra RAM on only during ML computation.
- datpiff 4y ago> Hundreds of GBs of RAM in a phone is just ~6 years away if Moore's law holds. It's not expected to. The consensus seems to be ~2025 https://arxiv.org/abs/1511.05956 https://arxiv.org/abs/1511.05956
- bonestamp2 4y agoNot to mention storage. The model is estimated to be 500-600GB. That's a lot of storage to tie up on a phone.