8 ms·
The hunt for the M1’s neural engine
- noduerme 5y agoThis is a bizarre result but so.. what's the conclusion? That only a few things like Apple's proprietary image lookup are able to tap into the ANE so far? Or that it's actually just a marketing gimmick? Reading this makes me wonder if it's not just a placeholder for some kind of intrusive system that will neural-hash everything you own, but I'm sure I'm just being paranoid.
- londons_explore 5y ago> That only a few things like Apple's proprietary image lookup are able to tap into the ANE so far? That would seem like the logical conclusion. Perhaps there are hardware bugs/shortcomings that makes it very hard to use for the neural network API. Perhaps the software team is just behind and still building that.
- my123 5y agoUsing CoreML has some catches, like having to use FP16 instead of integer formats notably.
- noduerme 5y agoMy initial understanding was that Adobe was leveraging it in their experimental "AI" Photoshop plugins. Although a majority of those seem to require a live internet connection to work. Some of the newer core Photoshop functionality for intelligent selection is ridiculously fast on an M1 Max, though, which makes me think it's probably using the neural chips.
- sharikous 5y agoIt might be, but there is also another matrix multiplication accelerator that could be responsible
- my123 5y agoThe ANE is accessible via the CoreML framework, it's a high level interface for ML inference. It however turns out that a _lot_ of customer apps today don't use those accelerators at all. (and about the attempt at using BNNS functions, that's not offloaded, it runs on the host CPU cores w/ the AMX tightly bound accelerator)
- galangalalgol 5y agoWould it be possible for someone ro write a onnx runtime utilizing coreml? That would open it up to a lot more applications instantly.
- ianai 5y agoONNX lists CoreML support.
- sharikous 5y agoI have the same anxiety but I don't think it is still there. The Asahi project is putting a deliberate low priority on the ANE but I have seen some other small reverse engineering attempts. I think some use of the ANE outside of Apple APIs will be possible soon.
- bayindirh 5y agoTensorflow has a CoreML enabled version which run on ANE. https://github.com/apple/tensorflow_macos https://github.com/apple/tensorflow_macos
- my123 5y agoAFAIK that doesn't run on ANE but on the GPU. ANE is used for inference only w/ CoreML.
- avianlyric 5y agoI would guess that the ANE has some very specialised hardware (E.g. INT8 or FP16 only), and there’s isn’t a huge amount of it. So many nets either don’t fit completely, or aren’t using the right types of operation to match the ANE. Either can’t run on the ANE, or only have a subset of layers that can run on the ANE. So when running a neural net iOS / macOS needs to make a decision about where to run each net. Even if a net has layers in it that are a perfect match for the ANE, there’s still a trade off from having move the workload back and forth between a CPU core and the ANE (although the unified memory should eliminate a big chunk of this cost). It might be that in the general case it’s not worth the latency hit from using mixed processors when running net that isn’t 100% ANE compatible, or could just be that Apple haven’t got round to implementing the logic needed to gracefully spilt workloads across the ANE and a CPU core. Which would make sense, because they’ve got the time and expertise to ensure all their nets fit within the ANE. Something that’s difficult for 3rd party devs to do, because they don’t have access to detailed ANE docs.
- masklinn 5y ago> Reading this makes me wonder if it's not just a placeholder for some kind of intrusive system that will neural-hash everything you own, but I'm sure I'm just being paranoid. It’s also actively counter-productive: if they wanted to do this sort of tracking, they could just have done what all of their competitors do and send the data straight to their servers. This is hardware (and thus expenses) which is only necessary because of their stance on privacy, and avoiding off-device work.
- mikotodomo 5y agoApple has made huge compromises to try and give some privacy back to the consumer, who lost it all from paying for cheap products where the product is you. And people just ignore this progress because they want to be anti-Apple. It's sad.
- deleted 5y ago[deleted]
- servytor 5y agoIf you are interested in the M1 neural engine, I highly recommend you check out this[0]. [0]: https://github.com/geohot/tinygrad/tree/master/accel/ane https://github.com/geohot/tinygrad/tree/master/accel/ane
- rickdeveloper 5y agoHe live streamed himself writing a lot of that: https://www.youtube.com/watch?v=mwmke957ki4 https://www.youtube.com/watch?v=mwmke957ki4 https://www.youtube.com/watch?v=H6ZpMMDvB1M https://www.youtube.com/watch?v=H6ZpMMDvB1M https://www.youtube.com/watch?v=JAyw7OAcXDE https://www.youtube.com/watch?v=JAyw7OAcXDE https://www.youtube.com/watch?v=Cb2KwcnDKrk https://www.youtube.com/watch?v=Cb2KwcnDKrk
- erwincoumans 5y agoYes, George Hotz (geohot) reverse engineered the neural engine and could make it work for tinygrad, the videos posted in the other reply describe the reverse engineering process. I wonder why Apple didn't provide low-level API's to access the hardware? It may have various restrictions. I recall Apple also didn't provide proper API's to access OpenCL frameworks on iOS, but some people found workarounds to access that as well. Maybe they only integrate with a few limited but important use cases, TensorFlow, Adobe that they can control. Could it be that using the ANE in the wrong way overheats the M1?
- xenadu02 5y agoCoreML is the API to use the ANE.
- erwincoumans 5y agoThanks, that's right there is a high level API. I meant low-level API's, and to clarify changed my post.
- fredoralive 5y agoPossibly just to avoid having programs that rely too much on specific implementation details of the current engine causing issues in the future if they decide to change the hardware design? An obvious comparison is graphics cards where you don't get low level access to the GPU[1], so they can change architecture details across generations. Using a high level API probably makes it easier to implement a software version for hardware that doesn't have the neural engine, like Intel Macs or older A-cores. [1] Although this probably starts a long conversation about various GPU and ML core APIs and quite how low level they get.
- marcan_42 5y agoThe Load Balancer is there because the M1 Max has two independent Neural Engines (unlike the GPU cores, which are load balanced in hardware and the OS sees as a single one even on the Ultra)... but one ANE is inexplicably disabled on all production systems. The M1 Ultra, logically, has four... and only the first in each die is enabled. I was waiting for the Mac Studio to drop to come to a conclusion, since it's plausible one ANE could've been disabled for power reasons on the laptops... but with both secondary ANEs in each die in the M1 Ultra off, and with no reports of anyone seeing the first ANE being disabled instead (which could mean it's a yield thing), I'm going to go ahead and say there was a silicon bug or other issue that made the second ANE inoperable or problematic, and they just decided to fuse it off and leave it as dead silicon on all M1 Max/Ultra systems.
- erwincoumans 5y agoThat is interesting, do you have any references / articles that describe that some ANE's are disabled? Could it be overheating if the ANE's are not used the right way?
- Jcowell 5y agoProbably not but he’s the one the highest divers into the M1 Chip due to his Linux Project so he’s probably the reputable source. Any article written would probably be referencing him.
- erwincoumans 5y agoAh, delightful to have a/the master behind Asahi development/reverse engineering efforts here!
- bschwindHN 5y agoDelightful to see the creator of Bullet Physics here in the comments too!
- bayindirh 5y ago
- endorphine 5y agoFor the totally uninitiated, what's a neural engine in general? What are they used for, and why Apple added this to their products?
- justusthane 5y agoHis previous blog post has more general information about the neural engine: https://eclecticlight.co/2022/03/29/live-text-visual-look-up-face-recognition-ml-and-privacy/ https://eclecticlight.co/2022/03/29/live-text-visual-look-up... Basically it's used by anything involving machine learning: - Speech recognition - Face recognition - Visual lookup (image recognition) - Live Text (OCR) It allows all these functions to be performed efficiently on-device rather than shipping data off to the cloud.
- jlouis 5y agoIt's an area of the chip suited for operations on low precision/range floating point numbers. Neural networks don't generally require high precision in floating point computations, but require a lot of them. This means your data paths can be smaller (16 bit wide rather than 32 bit wide); the consequence of which is you can do far more computation per mm2 die space. The second part of the chain is that you also tailor the operations the die area support to those operations necessary in a typical neural network, further optimizing the chip. The end result is very power-efficient execution of neural networks, which allows you to beat a GPU or CPUs power curve, improving thermals of the core, and in the case of mobile devices, optimizes battery usage.
- acdha 5y agoIt's probably also worth noting that the last part is fairly important to Apple since they have based a lot of their privacy stance around on-device processing of things like Siri commands, photo/video analysis, image recognition (VoiceOver can attempt to automatically describe images for blind people), speech to text dictation, the various audio/video enhancements like Center Stage or the voice emphasis features, etc. and all of that means they're running a lot of networks on battery. Those efficiency wins are almost certainly worth it even if third-party developers don't use it much.
- OskarS 5y agoThis is slightly beside the point, but those stack traces are C++ functions. I was pretty surprised by that (though, granted, I don't know anything about macOS internals). I would have expected either pure C for the kernel, or maybe Objective-C if they wanted a higher level language. They don't really have any C++ in their operating system APIs, right? Like, if you interface with Core <Whatever>, isn't that all C, Objective-C and Swift? Is there a lot of C++ in the macOS kernel/userspace?
- saagarjha 5y agoIOKit in the kernel is a C++ API; the userspace version is C.
- irae 5y agoThey are still a fork of BSD, so it stands to reason that the team working for years on kernel and lower level parts of the OS still uses C++.
- TickleSteve 5y agoMach is not a fork of BSD. The kernel is Mach (Hybrid microkernel) supporting a BSD API.
- astrange 5y agoThere is a lot of FreeBSD in the kernel/libc/userland. BSD doesn't use C++ though, they use C.
- galad87 5y agoI don't know how many, but a good number of frameworks are C++ internally, for example WebKit, the Objective-C runtime itself, IOKit uses a subset of C++, Metal Shading Language is a subset of C++.
- gurkendoktor 5y agoThere is (was?) always a lot of C++ in Core Animation stacktraces, and Core Animation was really foundational to the iPhone UI.
- pedro_hab 5y agoAs a developer I am bit ashamed of this question, but I gotta ask: What consumer apps actually use Neural Engines? I think something like Photoshop, maybe. But wouldn't it just train a model and use it as regular code? I'm interested in AI, but it's a joke to me about startups and jargon more often than not. I feels weird to add this to all chips when I can't see that much usage.
- OberstKrueger 5y agoPixelmator Pro uses it for some of its ML functionality. Image scaling can use it, and it provides a cleaner image when upscaling, removing some compression artifacts and just smoothing it out more naturally. I’ve found it can work well downsizing too, although less of an effect. They also have an ML auto-crop tool and ML denoiser. All of these will hit the Neural Engine pretty good.
- kitsunesoba 5y agoI think there may be more indirect usage than direct usage. Little bits of neural engine usage are peppered through the native APIs. Of course if you’re using third party libraries that don’t rely on macOS APIs this won’t be happening in your app.
- miohtama 5y agoMaybe the neural engine is ideal to scan your local image library to find kiddy porn: https://www.theverge.com/2021/12/15/22837631/apple-csam-detection-child-safety-feature-webpage-removal-delay https://www.theverge.com/2021/12/15/22837631/apple-csam-dete...
- sharkjacobs 5y agoAnything "predictive" probably uses Neural Engine. These are iOS features, but a lot of them apply to MacOS too - Visual Lookup - Animoji - Face ID - recognizing "accidental" palm input while using Apple Pencil - monitoring users' usage habits to optimize device battery life and charging - app recommendations - Siri - curating photos into galleries, selecting "good" photos to show in the photos widget - identifying people's faces in photos - creating "good" photos with input from tiny camera lenses and sensors - portrait mode - language translation - on-device dictation - AR plane detection Core ML API allows third party developers to use Neural Engine to run models
- modeless 5y agoIsn't BNNS documented to run on the CPU? Why would you expect it to use the neural engine? Apple also has Metal Performance Shaders which of course run on the GPU only. The user accessible API for the neural engine is Core ML. Very high level unfortunately. Hmm, it seems like there's also a new API that can use the neural engine sometimes, "ML Compute". But only for inference? https://developer.apple.com/documentation/mlcompute/mlcdevice/3750437-ane https://developer.apple.com/documentation/mlcompute/mlcdevic...