18 ms·
Transformer architecture optimized for Apple Silicon
- great_psy 4y agoMaybe apple will have a bigger effect on ai adoption than any other company. Local inference is huge for anything that requires even a little bit of privacy.
- rcarr 4y agoConvenience will be the biggest factor. Whoever makes it easier for the end consumer to get what they want wins. It's why ChatGPT made such a big splash in comparison to all the other AI models which were also impressive. Local inference may play a part in that if it's quicker but if the last twenty years are anything to go by it will be convenience rather than privacy which is the deciding factor. And Apple do tend to make the UX slicker than other companies so I wouldn't put it past them ending up the biggest player.
- brookst 4y ago> it will be convenience rather than privacy which is the deciding factor This. Nobody really cares about local processing for privacy. Even those that claim to often don't mean it. Remember the total freakout over Apple's proposed local, privacy-preserving processing to detect CSAM before uploading it to iCloud? The consensus seemed to be that secret, opaque, and un-auditable cloud-based scanning was much preferable.
- sedatk 4y agoLocal CSAM scanning wasn't an open book either. Also, it wasn't mutually exclusive with cloud scanning.
- FloatArtifact 4y agoHowever, there was enough of an outcrying by the community and my experts that it was pulled. Therefore end users do care about privacy when they understand the implications. After all, it's hard to care when you don't have power as an end user too act on those feelings.
- brookst 4y agoDo you think the current regime of Google/etc doing ad hoc scans on cloud storage, with no transparency about who is requesting scans for what, is more private?
- brookst 4y agoIt was pretty open — auditable and cryptographically proven sources for all triggering hashes, auditable updates. And it was mutually exclusive with cloud scanning, as it was part of E2EE.
- sedatk 4y agoAbout mutual exclusivity, you may be right. I probably misremember that specific detail, but that was before Apple enabled E2EE on iCloud. It was removed as a feature back then when they suggested local CSAM scanning. "It was pretty open" Not really. The scanning code wasn't open source. Apple didn't explain how their proprietary hashing algorithm worked, let alone providing any source code. It was just the promise of Apple about how the whole thing would work. If that's sufficient, then iCloud scanning is as open as local scanning. Auditability of DB updates doesn't mean much either. Even if a third party organization detected "new additions" to the CSAM DB, what would be the next step? There would be no way to verify if those hashes actually correspond to CSAM. A dictatorship, for example, would just say "yes it's CSAM, trust us", and you'd have no option but to trust their word. Even in the US, there is no way to verify if CSAM DB fully corresponds to actual CSAM. It's just NCMEC's word. We simply don't know.
- jazzyjackson 4y agoAlso for anyone not in a cellular sweetspot.
- totoglazer 4y ago(2022)
- hannofcart 4y agoAs someone entirely at sea with the rapid pace of development in this sphere: 1. Is this a new LLM from Apple? 2. Is this a way to optimize running LLMs like Llama locally on M1 macs? 3. Something else altogether?
- uoaei 4y ago2. A Transformer is a core building block of LLMs. > [T]he device spec for this reference implementation is M1 or newer chips for the Mac and A14 and newer chips for the iPhone and iPad
- jeffbee 4y agoIt's none of those things. It is tweaks of other existing code to run better on apple's hardware. This other article is far more informative than the repo: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra...
- sheepscreek 4y ago#2. A way to optimize running LLMs locally on Apple Silicon (including iPhones) I am just a little better informed. As I understand it, their code improves model performance and memory consumption using PyTorch and Huggingface libraries.
- singularity2001 4y agoit's python code so not for iphone (?)
- sheepscreek 4y agoSorry - I should have been more clear. Their examples compared the A* CPU performance and the repo includes Swift only code samples. But they’ve also made it possible to use them with traditional tooling (torch, huggingface). Hope that helps explain it.
- simonh 4y ago
- iamsanteri 4y agoThe race is on and ecosystems are moving fast.
- endisneigh 4y agoi'd say within 5 years apple will have optimized apple silicon and their tech, along with language model improvements, such that you will be able to get gpt-4 level performance in the iPhone 19 with inference happening entirely locally. openai is doing great work and is serious competition, but I think many underestimate big tech. once they're properly motivated they'll catch up quick. I think we can agree that openai is a sufficient motivator.
- bottlepalm 4y agoMaybe we should launch 100 of them out into space in different directions. Very low mass means we should be able to push it to a pretty high velocity.
- LASR 4y ago> into space in different directions Different directions - this is the fundamental challenge. It's hard to know what these directions are when we're flying blind at the cutting edge of technology. We're all human. I suspect out of the 100, you'll have 95 of them going into the same as before areas. Maybe 5 truly understand the above point and explicitly strike down ideas that have been done before. GPT-3 has been live for a while now. The industry has hundreds of story generator apps specialized for various kinds of content generation. Very few are really thinking about AGI or chain-of-thought reasoning etc as an example. So I don't think launching 100 of them into different directions would work. Maybe 5 of them with both the expertise and direction to pursue seemingly impossible goals.
- simonh 4y agoI know some people don’t like iPhones but this is taking it a bit far.
- ll_mama 4y ago5 years? Bit of a long stretch with how much focus is going into AI. I’d be surprised if it’s not possible within two iterations of the new iPhone.
- passwordoops 4y agoWeird I just read this tweet [0] arguing Apple will be launching their own secure and private LLM that runs on device (edge compute). https://twitter.com/LinusEkenstam/status/1638999208911949845?s=20 https://twitter.com/LinusEkenstam/status/1638999208911949845...
- deleted 4y ago[deleted]
- tinyhouse 4y agoThis is great. I cannot wait to try it on my laptop as I like to do dev locally. But I don't understand the development part - besides on device, how would you deploy this on a server let's say given Apple serves is not something cloud providers provide.
- leetrout 4y agoPhones not servers.
- eternalban 4y agoEdge not center.
- smoldesu 4y agoThere are acceleration options for ARM, either Microsoft's ONNX or proprietary "Ampere-optimized" libtorch are both good options in my experience.
- au8er 4y agoWhile the github contains the code, the article describing the optimisations are here: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra.... TL;DR: execution of pytorch models on apple's neural engine and standard data-oriented optimisations (changing matrix layout, chunking to optimise temporal cache locality, and minimising redundant memory copies)
- hahnchen 4y agoyou should put [June 2022] in the title
- singularity2001 4y agoalso no updates since then. does any library follow this reference implementation? llama.cpp surely is more optimized?
- JJJollyjim 4y agollama.cpp runs on the CPU, not the ANE or GPU.
- btown 4y agoI find it fascinating that this was released after the June 2022 launch of the M2 chipset and line of products, and yet Apple had no desire to show relative performance of M2 vs. M1 here - even in the simultaneous announcement here: https://machinelearning.apple.com/research/neural-engine-transformers https://machinelearning.apple.com/research/neural-engine-tra... It's fascinating to me that at least one of two things is true: either (a) Apple has lost its ability to coordinate "hype" between its teams, or (b) the difference between comparable levels e.g. the M1 Max vs. the M2 Max are so negligible that they don't look good in an announcement like this. Has anyone run inference for LLMs or other transformer models on comparable M1 and M2 Macs? Are there good benchmarks for this specific workload?
- coldtea 4y ago>I find it fascinating that this was released after the June 2022 launch of the M2 chipset and line of products And this was put out in Aug 2022. It's very propable that the team worked on it and tested it just on M1, and M2 was kept under wraps in different teams working on it until the announcement. So they just wrote the annoucement on the CPUs they worked on - and since it's not for a commercial product, Apple didn't care to optimize marketing anyway.
- windowsrookie 4y agoIt's my understanding the the performance difference is very small between the M1 and M2. So small that Apple is still selling new M1 Macs today. The M2 added more GPU cores and an advanced media decoder.
- pbronez 4y agoInteresting. Maybe they just wanted to publish quickly and didn’t take the time to benchmark across? Seems like a miss. M2 NPU is supposed to be 44% better than M1 https://www.cpu-monkey.com/en/article/apple_m2_vs_apple_m1__what_are_the_differences https://www.cpu-monkey.com/en/article/apple_m2_vs_apple_m1__...
- acdha 4y agoAs a newbie to this space, I see this mentioning PyTorch. I was looking at Whisper earlier today and, somewhat impressively, was reminded that my M1 Pro has a CPU fan. Is it realistic to think it would be a modest amount of work to install this in my local venv and use the NPU instead?
- cavisne 4y agoFor whisper whisper.cpp is probably a better bet, it also doesn’t use the NPU but it performs much better than the pytorch implementation.
- option 4y agoIn 2 years, LLM similar to today’s capability of ChatGPT will be running locally on newer phones and macs.
- deleted 4y ago[deleted]
- singularity2001 4y agoyes, but by then ChatGPT will have advanced by two years.
- qgin 4y agoIn regards to LLMs there is a collision between Apple's extremely good chip design capabilities and Apple's insistence that Siri never says anything that isn't 100% scripted and 100% certain to not bad. Up until now, they've chosen to limit Siri functionality rather than leave anything to chance. LLMs will absolutely be able to run locally, but whether Apple will be able to stop worrying and love the model remains to be seen.
- jonplackett 4y agoIt would seem to be more risky if the model is out on the device and you can’t correct any screw ups without doing an update. They’ll have to do something about Siri soon though. Even my 5 year old daughter told me Siri is ‘a bit thick’. And that’s just compared to Alexa never I’d ChatGPT level
- blueboo 4y agothey're waiting for the overton window to shift wrt acceptance of imperfect LLM responses... ...it's shifting...
- michelb 4y agoPretty sure Siri is getting a complete overhaul since it's a pita to add anything to it right now.
- celestialcheese 4y agoReminded me of one of the cooler uses of old iphones.[1] These old phones are going to continue being useful long after their initial life as a phone. As long as apple doesn't act like apple and lock everything down. 1 - https://findthatmeme.com/blog/2023/01/08/image-stacks-and-iphone-racks-building-an-internet-scale-meme-search-engine-Qzrz7V6T.html https://findthatmeme.com/blog/2023/01/08/image-stacks-and-ip...
- lr1970 4y agoThe github repo has only 5 commits in it and the last one is August 9, 2022. It looks abandoned.
- anentropic 4y agoIt's a reference implementation, other people are supposed to pick it up and apply the techniques to their own work I would love to see some links to any where it has been used? So far the most promising codebases for running LLMs on Mac have been the 'cpp' reimplementations, which ditch Pytorch and run on the CPU, using other tricks to fit the model into available RAM.
- MaysonL 4y agoNote that the most recent commit to this repo is 7 months ago.
- ronyfadel 4y agoThe bottleneck with compute at the edge is (and will be) model size (both app download time and storage space on device). Stable Diffusion sits at about 2GB for fp16, Whisper Medium at 1.53GB, LLAMA is 120GB. Sure, Apple can ship an optimized model (<2-4GB) as part of the OS, but what if a capable app maker wants to ship one? Users will not be happy with an app sized at >1GB.
- bloudermilk 4y agoI would gladly install a 1GB+ app on my 512GB iPhone that never gets above 50GB because of iCloud’s remote storage optimizations.
- ronyfadel 4y agoThat is not even taking into consideration that shipping your weights to the client is akin to giving your product away.
- aix1 4y agoWait, how is that fundamentally different from shipping binary code? Isn't that also akin to "giving your product away"?
- ronyfadel 4y agoYou can easily pirate an ML model, host it and provide a backend service that uses it, and no one would suspect anything. I don't think you can pirate e.g. the Facebook app binary and make it your own.
- KuzMenachem 4y agoYou can put some obstacles (https://developer.apple.com/documentation/coreml/generating_a_model_encryption_key https://developer.apple.com/documentation/coreml/generating_...) in the way though.
- CGamesPlay 4y agoResearch developments are already showing that our models are woefully inefficient in their current state (compare the performance of GPT-3 140B against Alpaca 30B). Not only will hardware get better, the minimum model sizes for good inference will become smaller in the future.
- sciencesama 4y agohow to run this on the mac ?
- pffft8888 4y agoIs it attention free?
- pfoof 4y agoSo, is Stable Diffusion working finally on TPU or not? DiffusionBee uses GPU and running this https://github.com/apple/ml-stable-diffusion https://github.com/apple/ml-stable-diffusion with CPU_AND_NE just segfaults
- leodriesch 4y agoAmazing AI by Sindre Sorhus uses Apples port of Stable Diffusion to the neural engine. https://sindresorhus.com/amazing-ai https://sindresorhus.com/amazing-ai
- pfoof 4y agoThanks, will check that out!
- polskibus 4y agoAdd 2022 to the title
- stefandeml 4y agodid anyone use this and adapted it make it work for llama or Alpaca?
- stefandeml 4y agodid anyone use this and adapted it make it work to run the llama or alpaca model?
- callen43 4y agoIs there also something available to make use of the ANE during _training_? E.g fine-tuning BERT on an M1 Mac in a couple of hours? (This here only applies to inference, right?)
- machinekob 4y agoIts FP16/Int8 inference only (cause you can only access it via apple frameworks that dosent support training). Also its only used if your data is small enough (4mb cache) it wont be useful for big transformers/ big images processing in a while.
- Torkel 4y ago2022? Latest commit seems to be "on Aug 9, 2022"
- waffletower 4y agoNot sure why Apple is throwing this code over the fence, (or catapulting it over the balustrade -- to continue the castle metaphor used here) and not also selling server form factor Apple silicon based devices for data centers. They left out an "issues" pane on the github repo, so it is intended to be quite the unidirectional act. I am not sure that Apple will remain proportionally huge with respect to other big tech while they squander their development on boutique mass market products, while ignoring the vast growth they could achieve if they expanded into first class cloud computing markets.
- apatheticonion 4y agoAs much as I love the progress in AI (also see Microsoft's recent Office Copilot) - I seriously think that government needs to step in to regulate the fair use of training data. Though this is painted by my personal beliefs, in order to maximise innovation, I believe it's the role of government to implement regulations that support a competitive commercial environment. Without this, monopolies will form around hard to obtain resources, innovation will stagnate and consumers will be subject to exploitation. Currently, data is mostly acquired without user consent and is accessible retroactively. Companies own that data, they can trade it and they can use it however they want to. You as a consumer have no say in this and it's virtually impossible to live a normal life without being the subject of analysis. While it's incredible that companies can produce undeniably valuable products like Copilot - ultimately - they will profit from these products. The irony is they built them from data sourced from you, likely from something you paid for (MS Word, etc). The key ingredient in these products is training data. If you wanted to compete with them, no matter how capable you are as an engineer, you could never make Copilot without the same scale of data Microsoft has gathered. I don't know what kind of regulation would even out the playing field, but I wouldn't mind being compensated for my role in creating these highly profitable products.