21 ms·
What we know about the Apple Neural Engine
- ls612 4y agoDoes anyone know if the neural engine on the new M1/M2 Max is directly hooked up to the unified memory the way the GPU is?
- wmf 4y agoDefine directly I guess.
- ls612 4y agoMy understanding is that the CPU and GPU both have DMA to the memory at some incredible speed since it’s all on the same chip. Does the ANE have that same DMA speed and latency?
- jamiek88 4y agoI believe so as it’s used by adobe amongst others, this was from a convo with an adobe engineer gushing about the UMA/DMA and what an improvement it was from the fans whirring jet engine end of Intel era. I can’t find any documentation about it though just everyone working under that assumption.
- sebzim4500 4y agoCan this really be everything publically known about the ANE? Sounds hard to believe, I would have thought someone would have reverse engineered something about it by now.
- detrites 4y agoMy question too. This semi-answer on the page seems to contradict itself (source: https://github.com/hollance/neural-engine/blob/master/docs/programming-ane.md https://github.com/hollance/neural-engine/blob/master/docs/p... ): "> Can I program the ANE directly? Unfortunately not. You can only use the Neural Engine through Core ML at the moment. There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks. (Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)" The last part links to this bunch of headers: https://github.com/nst/iOS-Runtime-Headers/tree/master/PrivateFrameworks/AppleNeuralEngine.framework https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva... So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?
- saagarjha 4y agoCorrect. (It is also unlikely that Apple exposes the Neural Engine directly.)
- SomeHacker44 4y agoSee other commenter above about GeoHot's analysis which is much more in depth.
- djoldman 4y agogeohot's findings: https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane
- detrites 4y agoOk, this is much more like what I expected from the OP. Anyone disappointed, here be full details on everything.
- ljlolel 4y agohttps://news.ycombinator.com/item?id=35302833 https://news.ycombinator.com/item?id=35302833
- gymbeaux 4y agoIs this the same geohot that got persecuted by Sony for jailbreaking the PS3?
- deleted 4y ago[deleted]
- ncr100 4y agoYes and spent years building a self driving kit and AI training background for many regular cars.
- ah- 4y agoThere's also basic ANE support for Asahi now: https://github.com/eiln/ane https://github.com/eiln/ane https://github.com/eiln/anecc https://github.com/eiln/anecc https://github.com/AsahiLinux/m1n1/pull/296/files https://github.com/AsahiLinux/m1n1/pull/296/files
- rowanG077 4y agoThat's misleading. It's much more apt to say it's being worked on. This is not available in any Asahi release at this time.
- callamdelaney 4y agoThere is an initial commit for ane support from yesterday. That means it's likely a long way off being deployed to asahi-edge.
- jeffrallen 4y agoÂne means donkey in French. Just sayin'.
- marginalia_nu 4y agoAlright, then, Apple Semantic System.
- mrweasel 4y agoWhich shortened also means donkey, brilliant.
- eastbound 4y agoNo those initials were already taken by Atlassian Software Systems. They seem to have lodged the paperwork with that name in 2002, and to have dropped it later on (they rather went with TEAM when going IPO in 2015), but back in 2010 when I applied, there was a book (collection of news articles) in the waiting room for candidates titled “Atlassian Software Systems”. Great guys. https://youtu.be/VfyUbuFoiBU https://youtu.be/VfyUbuFoiBU
- marginalia_nu 4y agoCan't make this shit up.
- dylan604 4y agoDon't forget the Advance SubStation Alpha subtitle format
- bitL 4y agoIt's really terrible that Apple markets this as the next big thing but forgets to include detailed documentation so people have to experiment and figure out what works...
- barkingcat 4y agoApple didn’t “forget” they never want to ever release apple proprietary docs. It’s their competitiveness advantage/ moat.
- smoldesu 4y agoIt's a damn shame, too. If we got the APFS documentation we were promised I feel like we'd be one step closer to world peace.
- lonelyasacloud 4y agoPart Apple's docs haven't been great for a while, part that's just how they roll, and part trying (like most everyone) to figure out what their strategy is going to be in a post GPT4 world [0]. [0] Persist with their own models running locally, how much to integrate with rest of the OS and maintain privacy moral ground, that sort of thing.
- cynicalsecurity 4y agoProprietary software, dude. It really sucks.
- saagarjha 4y agoPeople don't have to do anything. You use CoreML to program it.
- grupthink 4y agoIt's not that simple. If you have a model that actually does something useful (e.g. not just doing matmul & conv2d) your model will fail to run on ANE and, instead, the device will move it over to CPU/GPU and turn your iPhone into a heater. I literally had to continually wipe down my iPhone with a wet towel to keep it from overheating so I could build, ct.convert, run, and debug a model I was working on. Apple doesn't document how to keep operations on ANE. A model created by coremltools may run on either CPU, GPU, ANE, but you don't get to choose. And, if you don't know what you're doing and naively build a model, you will likely run on CPU/GPU only. If your batch is too large, too small, if you need to transpose tensors, if you need to expand mismatched tensors to matmul them together, if your model has an IF branch, or a loop, if you breathe the wrong way, your model silently falls off ANE. But you don't know what caused it. You have to open Netron and guess. Also, it may run on ANE on one device, but not another. There's no documentation from Apple about any of this. So, no, you do not simply "use CoreML to program it".
- thedonkeycometh 4y ago[dead]
- m3kw9 4y agoCan LLM run in it?
- egman_ekki 4y agomaybe https://github.com/apple/ml-ane-transformers https://github.com/apple/ml-ane-transformers
- simonw 4y agoSo my phone and my laptop both have the capability to perform 15 trillion operations per second, just in the neural engine? What kind of things are taking advantage of this right now? It's gotta be more than just Face ID right? What's my laptop likely to be doing with that?
- gedy 4y agoI can imagine some thing like Siri running on device much more effectively against local content. The cynic in me doesn’t want to hope too much for cloudless services like this, but one can hope.
- waboremo 4y agoScene analysis in photos, image captions, and machine translations are also done using ANE. CoreML also utilizes it when possible.
- fauigerzigerk 4y agoI don't know for sure, but things like text recognition (Live Text) or object recognition in Photos (Visual Look Up) are obvious candidates. I think neural engine is absolutely key to Apple's strategy. They want people to buy expensive devices and they don't want to process user data on their servers. Users get privacy. Apple gets money. It's a pretty coherent business model.
- flutas 4y ago
- anentropic 4y agoDo we think Apple are going to provide more info and maybe a public API over time? Or they are keeping it obscure for commercial reasons? Or just not very competent/don't care? Seems weird having these amazing chips and only blunt tools
- my123 4y agoCoreML. Directly exposing the ANE wouldn't make much sense, as it's an IP block that changes between generations in incompatible ways.
- brookst 4y agoThis is the answer. CoreML gives you an abstraction over different generations and sizes of underlying NPU. You might not want the abstraction, but love it or hate it, that’s kind of the Apple way. It will be very interesting to see what their next chips look like since we’re getting to the point where HW designs will reflect the rise of the, uh, transformers.
- enzomanico 4y agoYes of cour
- mmaunder 4y agoAnyone done any work on using a model for transcription on the local device using the ANE? I've heard it kills the battery. Having to transcribe voice in the cloud is a serious impediment to end to end encryption for certain applications.
- intalentive 4y agoThis is close: https://github.com/ggerganov/whisper.cpp https://github.com/ggerganov/whisper.cpp
- mmaunder 4y agoThanks!
- Tkuivamaki 4y agoSiri is still pants tho
- cubefox 4y agoWould be interesting to compare its performance to similar AI cores in other SoCs, such as Qualcomm's Snapdragon or Google's Tensor.