8 ms·
iOS 12 Core ML Benchmarks
- dep_b 8y agoImpressive but with a specialized coprocessor it's not unexpected. It's a bit of a gamble for Apple though, as other ARM SoC's are probably able to bridge the gap in raw single-core and multi-core performance in a year. The whole ML thing is a bit fuzzy for a lot of people even for programmers like me that are still wondering how their regular applications are going to benefit from all this silicon.
- spullara 8y agoApple doesn't use ARM SoCs. They have their own designs from the PA Semi team and are really just using the ARM instruction set at this point. This is one of the reasons there is such a big performance gap between the top android phones vs the top iPhones.
- dep_b 8y agoI guess the same can be said for AMD's x86_64 CPU's? Still it's fair to compare them to Intel x86_64 CPU's. The big difference is that ARM has something like a reference design where Intel doesn't want to share anything if they don't have to.
- rhencke 8y agoIt's actually the other way around, humorously enough. x86_64 is also known as AMD64, and for good reason. AMD created the core 64-bit x86 design, but, more importantly, they have patents to key pieces of the ISA required to legally implement it. So, long story short, it is actually AMD who didn't have to share anything with Intel if they did not want to - Intel's 64-bit x86 CPUs are fundamentally AMD64-compatible CPUs. The most interesting thing is it resulted in a bit of an interesting legal relationship between the two. Intel licenced AMD the required rights to produce x86-compatible processors. AMD, in turn, ended up licensing Intel the IP required for Intel to create x86_64 chips. Sources: https://en.m.wikipedia.org/wiki/X86-64#Licensing https://en.m.wikipedia.org/wiki/X86-64#Licensing https://www.sec.gov/Archives/edgar/data/2488/000119312509236705/dex102.htm https://www.sec.gov/Archives/edgar/data/2488/000119312509236...
- dep_b 8y agoWell AMD might want to sell the 64 part to you, but Intel doesn't want anybody having the x86 part if they can prevent it.
- shard 8y agoI don't think that's the reason. That's a similar situation with Kryo / Mongoose / Denver CPUs that are the respective companies' own designs and are really just using the ARM instruction set.
- spullara 8y agoIt really is the reason. Their CPUs are better than the CPUs that Android phones have access to. You can see this in the benchmarks. Compare Geekbench scores between them: https://browser.geekbench.com/ios-benchmarks https://browser.geekbench.com/ios-benchmarks https://browser.geekbench.com/android-benchmarks https://browser.geekbench.com/android-benchmarks The fastest iPhone CPU is 44% faster than the fastest Android device CPU.
- nacs 8y agoI think because Apple are dogfooding this ML stuff themselves (camera, face detection, filters, Siri, etc), exposing that API publicly isn't that much more risk to them.
- npunt 8y agoOther ARM SoC's haven't been able to bridge the relative gap in raw single-core performance going on 5+ years, and haven't hit multi parity since iDevices went beyond 2 cores. I don't really see a gamble here at all - adding ML blocks is relatively cheap and icing on the cake. Take Geekbench (and any other benchmarks) with a grain of salt of course, but benchmarks & reviews all corroborate the same story. See these GB4 results for instance: Android [1] - 3323 single, 8894 multi iPhone XS [2] - 4794 single, 11151 multi Especially for single-core, the latest, fastest Android phone (Galaxy S9) is a bit slower than the iPhone 7 [3], and every other Android phone (running Qualcomm's latest Snapdragon 845) is equivalent to the iPhone 6s / SE. The delta is just huge. It's actually a structural / incentive problem that keeps Apple at an advantage in this case, so I wouldn't count on the gap being bridged or even reduced anytime soon. Apple has built an incredible flywheel that lets them earn much more $ per chip than Qualcomm or Samsung by bundling them into super high-margin, high-volume iPhones. Everything equal, my wild-ass guess is Apple can probably throw at least 33% more transistors at workloads than the competition, due to being on leading edge nodes, being comfortable with lower yields, running multiple chip teams in parallel for maximum efficiency & manual layout, etc. This means huge caches, fancy pipelines, lots and lots of specialized silicon, etc. They also have arguably the most talented chip team in the industry. And that's just the chip. Because Apple is so vertically integrated, they're better able to optimize the device as a whole. That means more expensive components like faster ram & disk, better integration between IP blocks like CPU/GPU/ML, and more optimized OS/drivers that all play into overall performance and differentiated use cases. It's a business structure that is nearly impossible to replicate and will continue to create a lasting advantage. With regard to how ML is used, it fits the approach Apple is taking to keep things on-device (vs in the cloud) as much as possible. So any use cases where ML is used in the cloud are potential use cases for ML on the device. [1] https://browser.geekbench.com/android-benchmarks https://browser.geekbench.com/android-benchmarks [2] https://browser.geekbench.com/ios_devices/56 https://browser.geekbench.com/ios_devices/56 [3] https://browser.geekbench.com/ios-benchmarks https://browser.geekbench.com/ios-benchmarks
- jamesonthecrow 8y agoYour point about the integration between software and hardware is spot on. Even the Android devices with powerful GPUs or AI accelerators are really difficult to access because Android APIs (even the NNAPI) is really tough to use. Core ML "just works" with the CPU / GPU / Neural Engine.
- jeffbax 8y agoYou clearly haven't really kept up with Apple and ARM for the past 4 years have you. They're cleaning everyone's clock (pun unintended) with no signs of that abating. Last years X is still a generation faster than the fastest Qualcomm/Samsung chip today let alone what generic designs ARM licenses to the cheaper chip companies.
- dep_b 8y agoOh that's why I get all of the downvotes, finally I understand it. But you haven't really kept up with Apple and ARM for the past 4 weeks have you? Performance of the A12 is up something like 15% up - which is still impressive if you compare it to what Intel does on a yearly basis - but a lot less than it used to be for Apple SoC's. In the mean while the latest Qualcomm SoC's (845 is like 35% faster than 835) are performing almost as well as the A11, while the performance gap used to be more than two years. If Qualcomm manages to get another 35% increase next year their SoC will be faster at least until the A13. So: - Qualcomm is catching up - While single core and multicore did not improve as much as it used to YoY for Apple
- brigade 8y agoAlso, none of the benchmarks in the article have anything to do with the CPU.
- floatingatoll 8y agoOriginal author, when you find this, “Not the” is a typo for “Note the”.
- austin_kodra 8y agoAh! Nice catch. Thanks for this. Fixed!
- jamesonthecrow 8y agoGood catch. Also taking recommendations for better autocorrecting keyboards :)
- saagarjha 8y agoIt's interesting to see that ML performance is about equal on iPhone X and older iPhones, considering that iPhone X (and 8) has a neural engine and the other phones do not.
- ghop02 8y agoIt's my understanding that the neural engine API wasn't actually exposed to Core ML until the iPhone XS. The GPU is the important factor for performance of those models
- walterbell 8y agoWill it be exposed on the iPhone XR (the "mass market" model)? Presumably it will be supported on the upcoming iPad Pro and iOS' (mOS?) Arm laptop.
- microtherion 8y agoIIRC the neural engine was only available to first party apps in iOS 11.
- je42 8y agoso on ios 12 iphone x should see some speed up ?
- jamesonthecrow 8y agoThe neural engine is a huge boost, but also remember it's a logarithmic scale so the iPhone X is a faster 5x slower than the 6s.
- aphextron 8y agoI've noticed on my SE that everything is running much much faster with iOS 12. It seems like the app store is actually really usable now. Does anyone have any idea what they did?
- overcast 8y agoThe biggest change was ramping up processor instantly, and then back down to maximize performance. So for launching an app / scrolling, CPU reaches highest state, until that portion is over, and immediately get back to efficiency setting for preserving battery life. Previously CPU time would ramp up slowly.
- baby_wipe 8y agoIt's probably paranoia, but I feel like my 5S battery is draining faster since the update. That being said, the performance upgrade is well worth it.
- overcast 8y agoIt's not, my 6S did the same thing with intensive apps, like Instagram. With IG it's probably ALWAYS ramped up constantly scrolling, playing , downloading. They definitely sacrificed battery life for speed in older phones. My XS lasts like two days though.
- KSS42 8y agoThe article also points out iPad 2018 being faster than iPhone 7/8/X. This is odd. The article states that iPad 2018 uses A10X, but it actually use the A10. Is there a mix up with one of the iPad Pros? I can see an A10X (which has more GPU cores) beating the other iPads.
- MBCook 8y agoIt may simply be that due to a bigger battery and better heat dissipation it can run faster/longer and thus come out ahead. That said I think you’re right about there being a mixup. Perhaps they mean the 2017 iPad Pro 10.5 inch.
- jamesonthecrow 8y agoThats a good point. I mixed up the iPad Pro 2 with the 6th Gen iPad from 2018. The 2018 iPad just squeaked through my threshold for having enough data to be included here, so it's possible that this is just noise. I'll dig into the variance as more data comes in. The article is updated to reflect it. Thanks!
- KSS42 8y agoIn the graph, did you forget to change the label from "iPad 6th Gen (Wifi)" to "iPad Pro (2017)"?
- coder543 8y agoAre there any apps I can use that would let me see the difference in performance? Or even just use the Neural Engine for anything outside of the built-in camera app and the Measure app. EDIT: I see that Heartbeat is mentioned in the article, but I'm also still wondering what else is out there.
- deleted 8y ago[deleted]
- gok 8y agoHonestly I'm kind of surprised how little buzz there is about the A12's neural engine. ~5 tera-ops in your pocket, and it can run your models. For this particular set of math, that's comparable to a top-of-the-line desktop GPU from 3 years ago.