8 ms·
Qualcomm works with Meta to enable on-device AI applications using Llama 2
- smoldesu 3y ago> The ability to run generative AI models like Llama 2 on devices such as smartphones, PCs, VR/AR headsets Maybe it's an upcoming feature for the Quest 3? To that end, I've been pretty amazed by how far quantization has come. Some early llama-2 quantizations[0] have gotten down to ~2.8gb, though I haven't tested it to see how it performs yet. Still though, we're now talking about models that can comfortably run on pretty low-end hardware. It will be interesting to see where llama crops up with so many options for inferencing hardware. [0] https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/tree/main https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/tree/ma...
- m3kw9 3y agoChatGPT 3.5 is the base level people expect LLMs to be, it would be 2-3 generation(3-4 years) of hardware before we can reach that. Anything below is just going to get bad reviews
- bigyikes 3y agoA small model might be useful for e.g. NPC interactions in a Quest game
- fit2rule 3y ago[dead]
- madars 3y agoBut for what applications? Sure, for answering free-form questions I expect GPT-3.5+ quality. I don't think GPT-3.5 is necessary to provide auto-complete in your email client.
- Legend2440 3y ago3-4 years to run it on your phone seems generous, barring algorithmic breakthroughs. If I can run 100B+ models on my high-end desktop in 3-4 years I will be very happy.
- nullc 3y agoWhat do you consider a high end desktop now?
- holoduke 3y agoHe means a videocard with 512gb of memory or more.
- delecti 3y agoIs 512gb a typo? The current biggest consumer card has 24GB, so we're probably 15 years from a 512GB card (judging from the increase of 4Gb to 24GB between 2012 and 2022).
- flangola7 3y agoThat's why 3-4 years would be impressive
- Legend2440 3y agoWith 4-bit quantization the requirements are more like 64gb. I expect we'll see more unified memory designs like Apple's 128GB M1 Ultra.
- zacmps 3y agoI doubt it to be honest, desktop GPUs use too much power (and hence produce too much heat) to be integrated in that fashion, and any kind of shared memory will be too high latency.
- 3y ago
- kernal 3y agoIsn't that the same LLM that doesn't know how many e's are in "ketchup"? Nice.
- Legend2440 3y agoAren't you the same user that doesn't understand word-level tokenization?
- kernal 3y agoYou must have me confused with an LLM bot that does know how many e's are in "ketchup".
- brucethemoose2 3y agoLlama 65B finetunes already exceed it in some niches, like roleplaying or specific (coding and spoken) languages
- vorpalhex 3y agoHow would it work to get a model that needs 8Gb+ of vram currently into some chiplet form factor? Is there an obvious way of translating this more directly to hardware?
- malux85 3y agoIf your filesystem is fast enough, then you can dynamically load and unload chunks of it as you use it
- simonw 3y agoI don't think that works for LLMs. My understanding is that every single token produced by the model requires running floating point operations against all x-billion parameters, so the entire thing needs to be loaded into memory at all times for it to work.
- malux85 3y agoNope, you can use mmap to virtually map it to memory, and then you don’t have to hold the whole thing in RAM at once. I have spent the last 4-5 weeks working on this and optimising it You can see some information here: https://justine.lol/mmap/ https://justine.lol/mmap/
- simonw 3y agoHow does that work? Does it mean that for every token generated it has to page areas of disk into RAM and then back out again?
- not2b 3y agoYou'd have to have enough RAM to hold the whole model in memory or performance will be awful, mmap is just a way to get faster startup (if the mmap'd file looks exactly like the in-memory representation) and easier sharing (the mmap region can be shared read-only memory that multiple processes use).
- oneplane 3y agoI doubt Qualcomm will be able to increase their performance ahead of Nvidia decreasing their energy requirements.
- pjmlp 3y agoSince NVidia hardly ships on Android phones, it doesn't really matter.
- oneplane 3y agoThis isn't just about mobile telephones, or android for that matter. It's about on-device TPUs or AI accelerators or parallel compute units.
- pjmlp 3y agoWhich basically means Android devices, in what concerns mainstream consumers, given the state of the PC market. If on the other hand we look at embedded market, it is also not something where NVidia has any kind of strong market presence, with the exception of them trying to be in a good position on autonomous driving market.
- oneplane 3y agoThis isn't about mainstream consumers either. This is about a technological development where the balance between performance and energy consumption is attacked from both ends, where one vendor has the performance and the other vendor has the energy consumption but neither have both (yet).
- deleted 3y ago[deleted]
- behnamoh 3y agoMeanwhile Apple is stubbornly insisting on its own ways, and awkwardly silent during this whole AI revolution. I gave up using Siri years ago due to its glaring stupidity compared with Google Assistant and Alexa. While Apple keeps making money from overpriced hardware, the competitors are working on actually being pioneers in AI. It makes me sad to see so much computational power in my iPhone and iPad getting wasted on silly subpar iOS apps.
- huslage 3y agoApple has not been silent. The whole iOS 17 release has AI stuff sprinkled all through it. They just aren't speaking the same language as everyone else. They are also hiring generative AI folks like crazy for things like Siri. They want to own the on-device space and are well on the way to revolutionizing their operating systems and hardware to do it.
- ryandrake 3y agoThat seems right. Apple leads out with "We're doing feature X, Y, and Z" and downplays the particular technology behind those features. Nobody really cares if X, Y, and Z are powered by AI, as long as they're awesome. Contrast that with the rest of the industry who lead out with "We're doing AI!" and then they connect it to this and that feature. I'd argue that consumers care about having great features, not about what particular technology powers it. My phone could run on unicorn poop, and I don't care as long as it does what I want it to do.
- pazimzadeh 3y agoYeah...Apple doesn't tease products before they announce them.
- code51 3y agoI slowly started to think Apple knows there will be a golden hour to jump on a mature, economic LLM. Until then, they are probably watching without a massive investment into LLMs if it's not drastically changing their products.
- 3y ago
- simonw 3y agoIf you have a modern iPhone you can try running an LLM directly on it today using the MLC iPhone app: https://mlc.ai/mlc-llm/#iphone https://mlc.ai/mlc-llm/#iphone It can run Vicuna-7B which is a pretty impressive model. (They have an Android app too but I haven't tried that yet).
- haunter 3y ago>the MLC iPhone app Seems like US only
- brucethemoose2 3y agoMLC is great, but unfortunately is a demo more than anything. Other front ends aren't really integrating MLC yet.
- seydor 3y agoFB should release a phone, they already have quest OS
- samwillis 3y agoAs I said in the other Llama 2 thread[0], this is Meta doing a traditional textbook "commoditise your compliment". It's a preemptive attack with this particular announcement on Apple and Google. They don't what them own LLMs on portable devices, they want it to be a commodity. [0]: https://news.ycombinator.com/item?id=36775642 https://news.ycombinator.com/item?id=36775642
- baby 3y agoThe move from Meta is pretty obvious, they just want to attract the best AI researchers, they know that this generation of AI is not it, and they know that the best AI researchers want to work on open sourced stuff.
- scarface_74 3y agoWhat could possibly go wrong https://www.cnet.com/tech/mobile/heres-why-the-facebook-phone-flopped/ https://www.cnet.com/tech/mobile/heres-why-the-facebook-phon... Next up: they should make a social media app where each user can customize their profile page and have their favorite music start playing. It would be called YourSpace
- transcriptase 3y agoFacebook hardware in my devices that claims it’s there to protect my privacy. What’s the catch? Firmware based “anonymous” telemetry?
- serf 3y agopersonally I hope that the facebook altruism as of late has been a market effect from their needing to compete with other mega-corps, and that it comes with no catch or ill-will because it's simply there to be 'the other'. it's probably a naive hope, but I hope it's right. (i've never trusted FB further than I could throw their company, so this is a big leap of faith for me.)
- not2b 3y agoNot Facebook hardware in this case, it would be their model running on QCOM hardware. If people can write apps in which the model is used locally and no information goes to the net (to Meta, Google, or anyone else) that's a huge privacy win. Presumably Meta would make money by licensing it to Qualcomm and getting some amount per device. The key issue would be whether local-only apps can be written, or if Qualcomm and Meta would insist on seeing what the user is up to and would try to monetize that.
- ssss11 3y agoHmm I wonder what sort of privacy impacts there are in a future of having Meta (or Google et al) AI running on chips on your phone when the parent company has so much info on you and blatantly flaunts privacy laws.
- Nellyz 3y ago[dead]