7 ms·
I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more l
by isusmelj 2y ago
I think the results show that just in general the compute is not used well. That the CPU took 8.4ms and GPU took 3.2ms shows a very small gap. I'd expect more like 10x - 20x difference here.
I'd assume that the onnxruntime might be the issue. I think some hardware vendors just release the compute units without shipping proper support yet. Let's see how fast that will change.
Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. To focus on speed you'd need to get rid of the memory bottleneck. Then you end up designing your own ASIC with it's own memory. The NPUs we see in most devices are part of the SoC around the CPU to offload AI computations.
It would be interesting to run this benchmark in a infinite loop for the three devices (CPU, NPU, GPU) and measure power consumption.
I'd expect the NPU to be lowest and also best in terms of "ops/watt"
- AlexandrB 2y ago> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."
- kmeisthax 2y agoYou forget "Because Apple is doing it", too.
- rjsw 2y agoI think other ARM SoC vendors like Rockchip added NPUs before Apple, or at least around the same time.
- acchow 2y agoI was curious so looked it up. Apple's first chip with an NPU was the A11 bionic in Sept 2017. Rockchip's was the RK1808 in Sept 2019.
- GeekyBear 2y agoFace ID was the first tent pole feature that ran on the NPU.
- j16sdiz 2y agoGoogle TPU was introduced around same time as apple. Basically everybody knew it can be something around that time, just don't know exactly how
- Someone 2y agohttps://en.wikipedia.org/wiki/Tensor_Processing_Unit#Products https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Product... shows the first one is from 2015 (publicly announced in 2016). It also shows they have a TDP of 75+W. I can’t find TDP for Apple’s Neural Engine (https://en.wikipedia.org/wiki/Neural_Engine https://en.wikipedia.org/wiki/Neural_Engine), but the first version shipped in the iPhone 8, which has a 7 Wh battery, so these are targeting different markets.
- bdd8f1df777b 2y agoEven if it were true, they wouldn’t have the same influence as Apple has.
- itishappy 2y agoI assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."
- wtallis 2y agoLooking at it slightly differently: putting low-power NPUs into laptop and phone SoCs is how to get on the AI bandwagon in a way that NVIDIA cannot easily disrupt. There are plenty of systems where a NVIDIA discrete GPU cannot fit into the budget (of $ or Watts). So even if NPUs are still somewhat of a solution in search of a problem (aka a killer app or two), they're not necessarily a sign that these manufacturers are acting entirely without strategy.
- brookst 2y agoThe shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”
- MBCook 2y agoIs there demand? Or do they just assume there is? If they shove it in every single product and that’s all anyone advertises, whether consumers know it will help them or not, you don’t get a lot of choice. If you want the latest chip, you’re getting AI stuff. That’s all there is to it.
- Terr_ 2y ago"The math is clear: 100% of our our car sales come from models with our company logo somewhere on the front, which shows incredible customer desire for logos. We should consider offering a new luxury trim level with more of them." "How many models to we have without logos?" "Huh? Why would we do that?"
- Dalewyn 2y agoThere are no nerves in a neural processing unit, so yes: It's 300% bullshit marketing.
- jcgrillo 2y agoMaybe the N secretly stands for NFT.. Like the tesla self driving hardware only smaller and made of silicon.
- brookst 2y agoNeural is an adjective. Adjectives do not require their associated nouns to be present. See also: digital computers have mo fingers at all.
- -mlv 2y agoI always thought 'digital' referred to numbers, not fingers.
- bdd8f1df777b 2y agoThe derivative meaning has been use so widely that it has surpassed its original one in usage. But it doesn’t change the fact that it originally refers to the fingers.
- Spooky23 2y agoMicrosoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.
- cj 2y agoRightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.
- nxobject 2y agoAs a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.
- bugbuddy 2y agoThe problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.
- zdw 2y agoThis is most likely due to corporate malware. Even modern macs can be brought to their knees by something that rhymes with FrowdStrike Calcon and interrupts all IO.
- alisonatwork 2y agoPro tip: turn off malware scanning in your git repos[0]. There is also the new Dev Drive feature in Windows 11 that makes it even easier for developers (and IT admins) to set this kind of thing up via policies[1]. In companies where I worked where the IT team rolled out "security" software to the Mac-based developers, their computers were not noticeably faster than Windows PCs at all, especially given the majority of containers are still linux/amd64, reflecting the actual deployment environment. Meanwhile Windows also runs on ARM anyway, so it's not really something useful to generalize about. [0] https://support.microsoft.com/en-us/topic/how-to-add-a-file-type-or-process-exclusion-to-windows-security-e524cbc2-3975-63c2-f9d1-7c2eb5331e53 https://support.microsoft.com/en-us/topic/how-to-add-a-file-... [1] https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/
- conradev 2y agoThe real consumers of the NPUs are the operating systems themselves. Google’s TPU and Apple’s ANE are used to power OS features like Apple’s Face ID and Google’s image enhancements. We’re seeing these things in traditional PCs now because Microsoft has demanded it so that Microsoft can use it in Windows 11. Any use by third party software is a lower priority
- pclmulqdq 2y agoThe correct way to make a true "NPU" is to 10x your memory bandwidth and feed a regular old multicore CPU with SIMD/vector instructions (and maybe a matrix multiply unit). Most of these small NPUs are actually made for CNNs and other models where "stream data through weights" applies. They have a huge speedup there. When you stream weights across data (any LLM or other large model), you are almost certain to be bound by memory bandwidth.
- bee_rider 2y agoI’m sure we’ll get GPNPU. Low precision matvecs could be fun to play with.
- touisteur 2y agoSHAVE from MOVIDIUS was fun, before Intel bought them out.
- hedgehog 2y agoDid they become un-fun? There are a bunch on the new Intel CPUs.
- touisteur 2y agoMost of the toolchain got hidden behind openvino and there was no hardware released for years. Keembay was 'next year' for years. I have some code for DSP using it that I can't use anymore. Has Intel actually released new shave cores, with an actual dev environment ? I'm curious.
- hedgehog 2y agoThe politics behind the software issues are complex. At least from the public presentation the new SHAVE cores are not much changed besides bigger vector units. I don't know what it would take to make a lower level SDK available again but it sure seems like it would be useful.
- WithinReason 2y agoyeah I'm not sure being 1% utilised helps power consumption
- shermantanktop 2y agoThat’s how we got an explosion of interesting hardware in the early 80s - hardware companies attempting to entice consumers by claiming “blazing 16 bit speeds” or other nonsense. It was a marketing circus but it drove real investments and innovation over time. I’d hope the same could happen here.
- kmeisthax 2y ago> I think some hardware vendors just release the compute units without shipping proper support yet This is Nvidia's moat. Everything has optimized kernels for CUDA, and maybe Apple Accelerate (which is the only way to touch the CPU matrix unit before M4, and the NPU at all). If you want to use anything else, either prepare to upstream patches in your ML framework of choice or prepare to write your own training and inference code.
- noduerme 2y agoI'm not sure why this is a moat. Isn't it just a matter of translation from CUDA to some other instruction set? If AMD or someone else makes cheaper hardware that does the same thing, it doesn't seem like a stretch for them to release a PyTorch patch or whatever.
- david-gpu 2y agoMost of the computations are done inside NVidia proprietary libraries, not open-source CUDA. And if you saw what goes inside those libraries, I think you would agree that it is a substantial moat.
- theGnuMe 2y agoThere are clean room approaches like AMDs and Scale.
- caeril 2y agoGeohot has multiple (and ongoing) rants about the sheer instability of AMD RDNA3 drivers. Lisa Su engaged directly with him on this, and she didn't seem to give a shit about their problems. AMD is not taking ML applications seriously, outside of their marketing hype.
- fvv 2y agoRdna3 is not cdna
- godelski 2y agoThey definitely aren't doing the timing properly, but also what you might think is timing is not what is generally marketed. But I will say, those marketed versions are often easier to compare. One such example is that if you're using GPU then have you actually considered that there's an asynchronous operation as part of your timing? If you're naively doing `time.time()` then what happens is this start = time.time() # cpu records time pred = model(input.cuda()).cuda() # push data and model (if not already there) to GPU memory and start computation. This is asynchronous end = time.time() # cpu records time, regardless of if pred stores data You probably aren't expecting that if you don't know systems and hardware. But python (and really any language) is designed to be smart and compile into more optimized things than what you actually wrote. There's no lock, and so we're not going to block operations for cpu tasks. You might ask why do this? Well no one knows what you actually want to do. And do you want the timer library now checking for accelerators (i.e. GPU) every time it records a time? That's going to mess up your timer! (at best you'd have to do a constructor to say "enable locking for this accelerator") So you gotta do something a bit more nuanced. If you want to actually time GPU tasks, you should look at cuda event timers (in pytorch this is `torch.cuda.Event(enable_timing=True)`. I have another comment with boilerplate) Edit: There's also complicated issues like memory size and shape. They definitely are not being nice to the NPU here on either of those. They (and GPUs!!!) want channels last. They did [1,6,1500,1500] but you'd want [1,1500,1500,6]. There's also the issue of how memory is allocated (and they noted IO being an issue). 1500 is a weird number (as is 6) so they aren't doing any favors to the NPU, and I wouldn't be surprised that this is a surprisingly big hit considering how new these things are And here's my longer comment with more details: https://news.ycombinator.com/item?id=41864828 https://news.ycombinator.com/item?id=41864828
- artemisart 2y agoImportant precision: the async part is absolutely not python specific, but comes from CUDA, indeed for performance, and you will have to use cuda events too in C++ to properly time it. For ONNX the runtimes I know of are synchronous as we don't do each operation individually but whole models at once, there is no need for async, the timings should be correct.
- theresistor 2y ago> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. It's also often about offload. Depending on the use case, the CPU and GPU may be busy with other tasks, so the NPU is free bandwidth that can be used without stealing from the others. Consider AI-powered photo filters: the GPU is probably busy rendering the preview, and the CPU is busy drawing UI and handling user inputs.
- cakoose 2y agoOffload only makes sense if there are other advantages, e.g. speed, power. Without those, wouldn't it be better to use the NPUs silicon budget on more CPU?
- heavyset_go 2y agoMore CPU means siphoning off more of the power budget on mobile devices. The theoretical value of NPUs is power efficiency on a limited budget.
- theresistor 2y agoIf you know that you need to offload matmuls, then building matmul hardware is more area efficient than adding an entire extra CPU. Various intermediate points exist along that spectrum, e.g. Cell's SPUs.
- avianlyric 2y agoNot really. To get extra CPU performance that likely means more cores, or some other general compute silicon. That stuff tends to be quite big, simply because it’s so flexible. NPUs focus on one specific type of computation, matrix multiplication, and usually with low precision integers, because that’s all a neural net needs. That vast reduction in flexibility means you can take lots of shortcuts in your design, allowing you cram more compute into a smaller footprint. If you look at the M1 chip[1], you can see the entire 16-Neural engine has a foot print about the size of 4 performance cores (excluding their caches). It’s not perfect comparison, without numbers on what the performance core can achieve in terms of ops/second vs the Neural Engine. But it seems reasonable to be that the Neural Engine and handily outperform the performance core complex when doing matmul operations. [1] https://www.anandtech.com/show/16226/apple-silicon-m1-a14-deep-dive https://www.anandtech.com/show/16226/apple-silicon-m1-a14-de...
- spookie 2y agoI've been building an app in pure C using onnxruntime, and it outperforms a comparable one done with python by a substancial amount. There are many other gains to be made. (In the end python just calls C, but it's pretty interesting how much performance is lost)
- dacryn 2y agoagree there, but then again using ort in Rust is faster again. You cannot compare python with a onxx executor. I don't know what you used in Python, but if it's pytorch or similar, those are built with flexibility in mind, for optimal performance you want to export those to onxx and use whatever executor that is optimized for your env. onxxruntime is one of them, but definitely not the only one, and given it's from Microsoft, some prefer to avoid it and choose among the many free alternatives.
- rerdavies 2y agoWhy would the two not be entirely comparable? PyTorch may be slower at building the models; but once the model is compiled and loaded on the NPU, there's just not a whole lot of Python involved anymore. A few hundred CPU cycles to push the input data using python; a few hundred CPU cycles to receive the results using python. And everything in-between gets executed on the NPU.
- noduerme 2y agoI really wish Python wasn't the language controlling all the C code. You need a controller, in a scripting language that's easy to modify, but it's a rather hideous choice. It would be like choosing to build the world's largest social network in PHP or something. lol.
- robertlagrant 2y ago> it's a rather hideous choice Why?
- johnny22 2y ago
- lincpa 2y ago[dead]
- fennecfoxy 2y agoI think it's definitely possibly now (or very soon) for an LLM to write native GPU/NPU code to get itself to run on different hardware.
- hulitu 2y ago> The whole point of the NPU is rather to focus on low power consumption You know which chip has the lowest power consumption ? The one which is turned off. /s