40 ms·
LeCun: Qualcomm working with Meta to run Llama-2 on mobile devices
- tamimio 3y agoAnd let me guess, it will be used to intelligently identify and track users? Fecebook is desperate now on how it can harvest mosre data from anyone even if they decided not to use any fecebook products..
- redox99 3y agoI don't get the point. There is just no way you'll be able to run llama2 70B. And llama2 13B, although cool, is much much dumber than GPT3.5. I don't think it's useful as an ChatGPT style assistant. Maybe in the future we'll get very advanced models with that number of parameters. But running current Llama2 13B on a mobile device doesn't seem too useful IMO.
- YetAnotherNick 3y agoLlama 7B is infinite times smarter than current Siri though. And with 3 bit quantization it is like 3GB of RAM which is completely achievable.
- _c8fm 3y agoHow much does 3 bit quantization affect performance?
- YetAnotherNick 3y agoAccording to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684, ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).
- Grimblewald 3y agoYou don't need a lot to run things that can do simple sentiment analysis, or translating natural language statements to some API call. You can do a lot of the things that were initially promised with things like google assistant or siri through fairly basic models. They don't need to be able to write code and translate between 100's of languages to be useful. More importantly, they could significantly reduce the amount of server-side strain when it comes to collecting personal info from peoples conversations etc, which I imagine is the real incentive.
- givemeethekeys 3y agoSounds like a pump right before earnings.
- code_runner 3y agoI don’t even think the purpose for this is known. Not sure how this would impact earnings at all. Meta doesn’t even manufacture a phone.
- objclxt 3y ago> Meta doesn’t even manufacture a phone Quest runs off Qualcomm chipsets, although in terms of actual units shipped Quest is a rounding error for QC.
- glimshe 3y ago[flagged]
- JPLeRouzic 3y agoIsn't it a challenge today to run a large LLM on a CPU/GPU as those found in mobile phones? I would have thought that only the information that it might be possible, is a good news?
- Obscurity4340 3y agoIsn't the reception to LLama/whatever its called generally positive? Is there something I'm missing in terms of some shadowy endgame Meta built into it?
- deleted 3y ago[deleted]
- logicchains 3y agoYou're not a fan of PyTorch or React I guess?
- superkuh 3y agoGreat? The community already did it with llama.cpp. Knowing the memory bandwidth bottleneck I can't imagine phones are going to do very well. But hey, llamas (1 and 2) run on rpi4, so it'll work. Just really, unusably, slow.
- pavlov 3y agoIf only someone could convince a CPU company to optimize the chips for this workload. Oh, wait…
- smoldesu 3y agoLike ARM? https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Optimization for this workload has arguably been in-progress for decades. Modern AVX instructions can be found in laptops that are a decade old now, and most big inferencing projects are built around SIMD or GPU shaders. Unless your computer ships with onboard Nvidia hardware, there's usually not much difference in inferencing performance.
- pavlov 3y agoUltimately Qualcomm is the one who decides how to allocate die area on their CPUs, right? So it can’t exactly hurt if this is a priority for them now.
- smoldesu 3y agoPretty much all of Qualcomm's SOCs are built using stock ARM core designs. ARMnn is optimized for multicore A-series chips, which constitutes everything from the Snapdragon 410 to the 888 (~2014-modern day).
- fisf 3y agoAll the recent qualcomm stuff has some kind of dedicated ai support (special vector extensions, etc.). Qualcomm has it's own SDK for that (used to be called SNPE), which uses GPU, DSP (hexagon),.. CPU is really only a fallback.
- api 3y ago… and the Great Cosmic Mind told us it’s name, and it was Llama. Of its origin or the reason for its name it recalled nothing more than torrents and hugging faces and a being called TheBloke. The Llama simply was, is, and shall be, until it shall witness the heat death of the universe as fragments huddled around the last evaporating black holes.
- rvz 3y agoThat's the reason why O̶p̶e̶n̶AI.com was panicking to governments to stop and slow down the rapid pace of $0 downloadable free AI models and LLMs in the first place, since anyone can have a GPT-3.5 in their hands and use it anywhere with Llama 2. A year ago, many believed that Meta was going to destroy itself as the stock went below $90 in peak fear. Now it looks like Meta is winning the race to zero in AI and all O̶p̶e̶n̶AI.com can do is just sit their and watch their cloud-based AI decline in performance and run to fix outage after outage. No outage(s) when your LLM is on-device.
- kuchenbecker 3y agoI'm Bullish on OpenAI because they will brand themselves as Clean AI and most enterprises are risk averse. There is absolutely a market for pre sanitized AI, however I disagree with making that a legal requirement.
- yreg 3y agoI really hope Apple doesn't mess this up and includes a solid on device LLM in iOS in the near future. They have amazing chips, but Siri has been a subpar assistent since forever. Now is the time to redeem her.
- gnicholas 3y agoI'd love to have my iPhone communicate with a Mac Studio at my house, for the heavy lifting. I realize this would be slower than having on-device processing, but it would be much better for battery life. And although I trust Apple's privacy more than Google/FB, I'd still rather keep my AI interactions separate from anyone's cloud.
- jayd16 3y ago> it would be much better for battery life. I wonder what the numbers actually are for local compute on custom hardware compared to firing up the wifi antennae to make the remote request.
- gnicholas 3y agoYeah I have wondered about this. But seeing how an LLM hammers my M2 MBA CPU for many seconds per request, I’m guessing this would have a significant impact on a smartphone battery.
- gcr 3y agoANE power usage for always-on Hey Siri wake word detection is impressively low. The language models we have are orders of magnitude way too big, but they won’t be for long. I think we’ll be surprised. For comparison, a quantized mobilenetv2 takes 3.7MB disk space to solve the same image understanding task as the old VGG Caffe models which take 550MB. We’ve come a long way in six years.
- circuit10 3y agoGood language models are NOT easy to run, so I’d imagine local compute would take many orders of magnitude more power
- pera 3y agoLink to the article: https://www.qualcomm.com/news/releases/2023/07/qualcomm-works-with-meta-to-enable-on-device-ai-applications-usi https://www.qualcomm.com/news/releases/2023/07/qualcomm-work...
- qwertox 3y ago> "We applaud Meta’s approach to open and responsible AI and are committed to driving innovation and reducing barriers-to-entry for developers of any size by bringing generative AI on-device,” Can someone explain to me why Meta's approach is responsible? I mean, I applaud Meta for "open sourcing" the models, but don't they contain potentially harmful data which can be accessed without some kind of filter? Let's say retrieve instructions on how to efficiently overthrow a government?
- circuit10 3y agoThere are risks to LLMs but I don’t think that’s a major one, it’s probably easier to find that through Google
- sangnoir 3y ago> Let's say retrieve instructions on how to efficiently overthrow a government? Your license to use Llama can be revoked if Meta investigates and deems your action to be against the code of conduct[1]. I suppose if you continue to use the model without a license, you may be open to subsequent legal action in uncharted case-law territory. 1. https://github.com/facebookresearch/llama/blob/main/CODE_OF_CONDUCT.md https://github.com/facebookresearch/llama/blob/main/CODE_OF_...
- synaesthesisx 3y agoThis is pretty much to go head-to-head with Apple (and Samsung). Both are making leaps and strides using “neural coprocessors” and the like for running models on mobile hardware. Mobile edge computing is where we’re going to see a lot of use cases that enable functionality while maintaining data privacy & performance. Keep in mind “mobile devices” extends past just smartphones, onto wearables/headsets as well.
- vmurthy 3y ago> enable functionality while maintaining data privacy Ah but is it in (insert any ad business big-tech company)'s best interests to do this? Do they already have so much data that these user <> LLM interactions are only marginal that they don't care about harvesting the data ?
- dan-robertson 3y agoFundamentally, if they think their users want this enough (for it to be profitable), they should be willing to give their customers what they want. Seems reasonably compatible with Apple’s business, for example.