8 ms·
> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption. I ha
by AlexandrB 2y ago
> Also, people often mistake the reason for an NPU is "speed". That's not correct. The whole point of the NPU is rather to focus on low power consumption.
I have a sneaking suspicion that the real real reason for an NPU is marketing. "Oh look, NVDA is worth $3.3T - let's make sure we stick some AI stuff in our products too."
- kmeisthax 2y agoYou forget "Because Apple is doing it", too.
- rjsw 2y agoI think other ARM SoC vendors like Rockchip added NPUs before Apple, or at least around the same time.
- acchow 2y agoI was curious so looked it up. Apple's first chip with an NPU was the A11 bionic in Sept 2017. Rockchip's was the RK1808 in Sept 2019.
- GeekyBear 2y agoFace ID was the first tent pole feature that ran on the NPU.
- j16sdiz 2y agoGoogle TPU was introduced around same time as apple. Basically everybody knew it can be something around that time, just don't know exactly how
- Someone 2y agohttps://en.wikipedia.org/wiki/Tensor_Processing_Unit#Products https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Product... shows the first one is from 2015 (publicly announced in 2016). It also shows they have a TDP of 75+W. I can’t find TDP for Apple’s Neural Engine (https://en.wikipedia.org/wiki/Neural_Engine https://en.wikipedia.org/wiki/Neural_Engine), but the first version shipped in the iPhone 8, which has a 7 Wh battery, so these are targeting different markets.
- bdd8f1df777b 2y agoEven if it were true, they wouldn’t have the same influence as Apple has.
- itishappy 2y agoI assume you're both right. I'm sure NPUs exist to fill a very real niche, but I'm also sure they're being shoehorned in everywhere regardless of product fit because "AI big right now."
- wtallis 2y agoLooking at it slightly differently: putting low-power NPUs into laptop and phone SoCs is how to get on the AI bandwagon in a way that NVIDIA cannot easily disrupt. There are plenty of systems where a NVIDIA discrete GPU cannot fit into the budget (of $ or Watts). So even if NPUs are still somewhat of a solution in search of a problem (aka a killer app or two), they're not necessarily a sign that these manufacturers are acting entirely without strategy.
- brookst 2y agoThe shoehorning only works if there is buyer demand. As a company, if customers are willing to pay a premium for a NPU, or if they are unwilling to buy a product without one, it is not your place to say “hey we don’t really believe in the AI hype so we’re going to sell products people don’t want to prove a point”
- MBCook 2y agoIs there demand? Or do they just assume there is? If they shove it in every single product and that’s all anyone advertises, whether consumers know it will help them or not, you don’t get a lot of choice. If you want the latest chip, you’re getting AI stuff. That’s all there is to it.
- Terr_ 2y ago"The math is clear: 100% of our our car sales come from models with our company logo somewhere on the front, which shows incredible customer desire for logos. We should consider offering a new luxury trim level with more of them." "How many models to we have without logos?" "Huh? Why would we do that?"
- Dalewyn 2y agoThere are no nerves in a neural processing unit, so yes: It's 300% bullshit marketing.
- jcgrillo 2y agoMaybe the N secretly stands for NFT.. Like the tesla self driving hardware only smaller and made of silicon.
- brookst 2y agoNeural is an adjective. Adjectives do not require their associated nouns to be present. See also: digital computers have mo fingers at all.
- -mlv 2y agoI always thought 'digital' referred to numbers, not fingers.
- bdd8f1df777b 2y agoThe derivative meaning has been use so widely that it has surpassed its original one in usage. But it doesn’t change the fact that it originally refers to the fingers.
- Spooky23 2y agoMicrosoft needs to throw something in the gap to slow down MacBook attrition. The M processors changed the game. My teams support 250k users. I went from 50 MacBooks in 2020 to over 10,000 today. I added zero staff - we manage them like iPhones.
- cj 2y agoRightly so. The M processor really did completely eliminate all sense of “lag” for basic computing (web browsing, restarting your computer, etc). Everything happens nearly instantly, even on the first generation M1 processor. The experience of “waiting for something to load” went away. Not to mention these machines easily last 5-10 years.
- nxobject 2y agoAs a very happy M1 Max user (should've shelled out for 64GB of RAM, though, for local LLMs!), I don't look forward to seeing how the Google Workspace/Notions/etc. of the world somehow reintroduce lag back in.
- bugbuddy 2y agoThe problem for Intel and AMD is they are stuck with an OS that ships with a lag-inducing Anti-malware suite. I just did a simple git log and it took 2000% longer than usual because the Antivirus was triggered to scan and run a simulation on each machine instruction and byte of data accessed. The commit log window stayed blank waiting to load long enough for me to complete another tiny project. It always ruin my day.
- zdw 2y agoThis is most likely due to corporate malware. Even modern macs can be brought to their knees by something that rhymes with FrowdStrike Calcon and interrupts all IO.
- alisonatwork 2y agoPro tip: turn off malware scanning in your git repos[0]. There is also the new Dev Drive feature in Windows 11 that makes it even easier for developers (and IT admins) to set this kind of thing up via policies[1]. In companies where I worked where the IT team rolled out "security" software to the Mac-based developers, their computers were not noticeably faster than Windows PCs at all, especially given the majority of containers are still linux/amd64, reflecting the actual deployment environment. Meanwhile Windows also runs on ARM anyway, so it's not really something useful to generalize about. [0] https://support.microsoft.com/en-us/topic/how-to-add-a-file-type-or-process-exclusion-to-windows-security-e524cbc2-3975-63c2-f9d1-7c2eb5331e53 https://support.microsoft.com/en-us/topic/how-to-add-a-file-... [1] https://learn.microsoft.com/en-us/windows/dev-drive/ https://learn.microsoft.com/en-us/windows/dev-drive/
- conradev 2y agoThe real consumers of the NPUs are the operating systems themselves. Google’s TPU and Apple’s ANE are used to power OS features like Apple’s Face ID and Google’s image enhancements. We’re seeing these things in traditional PCs now because Microsoft has demanded it so that Microsoft can use it in Windows 11. Any use by third party software is a lower priority
- pclmulqdq 2y agoThe correct way to make a true "NPU" is to 10x your memory bandwidth and feed a regular old multicore CPU with SIMD/vector instructions (and maybe a matrix multiply unit). Most of these small NPUs are actually made for CNNs and other models where "stream data through weights" applies. They have a huge speedup there. When you stream weights across data (any LLM or other large model), you are almost certain to be bound by memory bandwidth.
- bee_rider 2y agoI’m sure we’ll get GPNPU. Low precision matvecs could be fun to play with.
- touisteur 2y agoSHAVE from MOVIDIUS was fun, before Intel bought them out.
- hedgehog 2y agoDid they become un-fun? There are a bunch on the new Intel CPUs.
- touisteur 2y agoMost of the toolchain got hidden behind openvino and there was no hardware released for years. Keembay was 'next year' for years. I have some code for DSP using it that I can't use anymore. Has Intel actually released new shave cores, with an actual dev environment ? I'm curious.
- hedgehog 2y agoThe politics behind the software issues are complex. At least from the public presentation the new SHAVE cores are not much changed besides bigger vector units. I don't know what it would take to make a lower level SDK available again but it sure seems like it would be useful.
- WithinReason 2y agoyeah I'm not sure being 1% utilised helps power consumption
- shermantanktop 2y agoThat’s how we got an explosion of interesting hardware in the early 80s - hardware companies attempting to entice consumers by claiming “blazing 16 bit speeds” or other nonsense. It was a marketing circus but it drove real investments and innovation over time. I’d hope the same could happen here.