5 ms·

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of

These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. I can't find a die analysis of the Snapdragon X which isolates the NPU specifically but AMDs equivalent with the same ~50 TOPS performance target can be seen here, and takes up about as much area as three high performance CPU cores: https://www.techpowerup.com/325035/amd-strix-point-silicon-pictured-and-annotated#g325035-2 https://www.techpowerup.com/325035/amd-strix-point-silicon-p...

Kon-Peki 2y ago
Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much. Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU cores there - that would increase the power density too much and cause thermal issues that can only be solved with permanent throttling. [1] https://en.wikipedia.org/wiki/Dark_silicon https://en.wikipedia.org/wiki/Dark_silicon
- IshKebab 2y ago
  If they aren't being used it would be better to dedicate the space to more SRAM.
  - a2l3aQ 2y ago
    The point is parts of the CPU have to be off or throttled down when other components are under load to maintain TDP, adding cache that would almost certainly be being used defeats the point of that.
    - jsheard 2y ago
      Doesn't SRAM have much lower power density than logic with the same area though? Hence why AMD can get away with physically stacking cache on top of more cache in their X3D parts, without the bottom layer melting.
      - Kon-Peki 2y ago
        Yes, cache has a much lower power density and could have been a candidate for that space. But I wasn’t on the design team and have no basis for second-guessing them. I’m just saying that cramming more performance CPU cores onto this die isn’t a realistic option.
      - wtallis 2y ago
        The SRAM that AMD is stacking also has the benefit of being last-level cache, so it doesn't need to run at anywhere near the frequency and voltage that eg. L1 cache operates at.
    - IshKebab 2y ago
      Cache doesn't use nearly as much power as active computation; that was my point.
  - VHRanger 2y ago
    SRAM is extremely hot, it's the very opposite of dark silicon
- jcgrillo 2y ago
  Question--what's to be lost by making your features sufficiently not dense to allow them to cool at full tilt?
  - AlotOfReading 2y ago
    Messes with timing, among other things. A lot of those structures are relatively fixed blocks that are designed for specific sizes. Signals take more time to propagate longer distances, and longer conductors have worse properties. Dense and hot is faster and more broadly useful.
    - jcgrillo 2y ago
      Interesting, so does that mean we're basically out of runway without aggressive cooling?
      - joha4270 2y ago
        No. Every successive semiconductor node uses less power than the previous per transistor at the same clock speed. Its just that we then immediately use this headroom to pack more transistors closer and run them faster, so every chip keeps running into power limits, even if they continually do more with said power.
  - positr0n 2y ago
    Good discussion on how at multi GHz clock speeds, the speed of light is actually limiting on some circuit design choices: https://news.ycombinator.com/item?id=12384596 https://news.ycombinator.com/item?id=12384596
ezst 2y ago
I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by looking at the sales and how few are willing to pay a premium for "AI") how overblown and unnecessary this is.
- DrillShopper 2y ago
  Corporatized gains in the market from hype Socialized losses in increased carbon emissions, upheaval from job loss, and higher prices on hardware. The more they say the future will be better the more that it looks like the status quo.
- renewiltord 2y ago
  I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money because they won't optimize die size. Disgusting.
  - nl 2y ago
    Is this a parody? Why would anyone care about die size? And if you do why not get one of the many low power laptops with Atoms etc that do have small die size?
    - throwaway48476 2y ago
      Maybe through a game of telephone they confused die size and node size?
    - thfuran 2y ago
      Yes, they're making fun of the comment they replied to.
JohnFen 2y ago
> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put to?
- jsheard 2y ago
  > But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.
  - AlexAndScripts 2y ago
    It almost makes me want to find some use for them on my Linux box (not that is has an NPU), but I truly can't think of anything. Too small to run a meaningful LLM, and I'd want that in bursts anyway, I hate voice controls (at least with the current tech), and Recall sounds thoroughly useless. Could you do mediocre machine translation on it, perhaps? Local github copilot? An LLM that is purely used to build an abstract index of my notes in the background? Actually, could they be used to make better AI in games? That'd be neat. A shooter character with some kind of organic tactics, or a Civilisation/Stellaris AI that doesn't suck.
    - ywvcbk 2y ago
      > box Presumably you have a GPU? If so there is nothing an NPU can do that a discrete GPU can’t (and it would be much slower than a recent GPU). The real benefits are power efficiency and cost since they are built into the SoC which are not necessarily that useful on a desktop PC.
    - Miraste 2y ago
      In short: no. Current-gen NPUs are so slow they can't do anything useful. AMD and Intel have 2nd-gen ones that came out a few weeks ago, and by spec they may able to run local translation and small LLMs (haven't seen benchmarks yet), but for now they are laptop-only.
kllrnohj 2y ago
Snapdragon X still has a full 12 cores (all same cores, it's homogeneous) and the Strix Point is also 12 cores but in a 4+8 configuration but with the "little" cores not sacrificing that much (nothing like the little cores in ARM's designs which might as well not even exist, they are a complete waste of silicon). Consumer software doesn't scale to that, so what are you going to do with more transistors allocated to the CPU? It's not unlike why Apple puts so many video engines in their SoCs - they don't actually have much else to do with the transistor budget they can afford. Making single thread performance better isn't limited by transistor count anymore and software is bad at multithreading.
- wmf 2y ago
  GPU "infinity" cache would increase 3D performance and there's a rumor that AMD removed it to make room for the NPU. They're not out of ideas for features to put on the chip.

Kon-Peki 2y ago

Modern chips have to dedicate a certain percentage of the die to dark silicon [1] (or else they melt/throttle to uselessness), and these kinds of components count towards that amount. So the point of these components is to be used, but not to be used too much. Instead of an NPU, they could have used those transistors and die space for any number of things. But they wouldn't have put additional high performance CPU cores there - that would increase the power density too much and cause thermal issues that can only be solved with permanent throttling. [1] https://en.wikipedia.org/wiki/Dark_silicon https://en.wikipedia.org/wiki/Dark_silicon

IshKebab 2y ago
If they aren't being used it would be better to dedicate the space to more SRAM.
- a2l3aQ 2y ago
  The point is parts of the CPU have to be off or throttled down when other components are under load to maintain TDP, adding cache that would almost certainly be being used defeats the point of that.
  - jsheard 2y ago
    Doesn't SRAM have much lower power density than logic with the same area though? Hence why AMD can get away with physically stacking cache on top of more cache in their X3D parts, without the bottom layer melting.
    - Kon-Peki 2y ago
      Yes, cache has a much lower power density and could have been a candidate for that space. But I wasn’t on the design team and have no basis for second-guessing them. I’m just saying that cramming more performance CPU cores onto this die isn’t a realistic option.
    - wtallis 2y ago
      The SRAM that AMD is stacking also has the benefit of being last-level cache, so it doesn't need to run at anywhere near the frequency and voltage that eg. L1 cache operates at.
  - IshKebab 2y ago
    Cache doesn't use nearly as much power as active computation; that was my point.
- VHRanger 2y ago
  SRAM is extremely hot, it's the very opposite of dark silicon
jcgrillo 2y ago
Question--what's to be lost by making your features sufficiently not dense to allow them to cool at full tilt?
- AlotOfReading 2y ago
  Messes with timing, among other things. A lot of those structures are relatively fixed blocks that are designed for specific sizes. Signals take more time to propagate longer distances, and longer conductors have worse properties. Dense and hot is faster and more broadly useful.
  - jcgrillo 2y ago
    Interesting, so does that mean we're basically out of runway without aggressive cooling?
    - joha4270 2y ago
      No. Every successive semiconductor node uses less power than the previous per transistor at the same clock speed. Its just that we then immediately use this headroom to pack more transistors closer and run them faster, so every chip keeps running into power limits, even if they continually do more with said power.
- positr0n 2y ago
  Good discussion on how at multi GHz clock speeds, the speed of light is actually limiting on some circuit design choices: https://news.ycombinator.com/item?id=12384596 https://news.ycombinator.com/item?id=12384596

ezst 2y ago

I can't wait for the LLM fad to be over so we get some sanity (and efficiency) back. I personally have no use for this extra hardware ("GenAI" doesn't help me in any way nor supports any work-related tasks). Worse, most people have no use for that (and recent surveys even show predominant hostility towards AI creep). We shouldn't be paying extra for that, it should be opt-in, and then it would become clear (by looking at the sales and how few are willing to pay a premium for "AI") how overblown and unnecessary this is.

DrillShopper 2y ago
Corporatized gains in the market from hype Socialized losses in increased carbon emissions, upheaval from job loss, and higher prices on hardware. The more they say the future will be better the more that it looks like the status quo.
renewiltord 2y ago
I was telling someone this and they gave me link to a laptop with higher battery life and better performance than my own, but I kept explaining to them that the feature I cared most about was die size. They couldn't understand it so I just had to leave them alone. Non-technical people don't get it. Die size is what I care about. It's a critical feature and so many mainstream companies are missing out on my money because they won't optimize die size. Disgusting.
- nl 2y ago
  Is this a parody? Why would anyone care about die size? And if you do why not get one of the many low power laptops with Atoms etc that do have small die size?
  - throwaway48476 2y ago
    Maybe through a game of telephone they confused die size and node size?
  - thfuran 2y ago
    Yes, they're making fun of the comment they replied to.

JohnFen 2y ago

> These NPUs are tying up a substantial amount of silicon area so it would be a real shame if they end up not being used for much. This has been my thinking. Today you have to go out of your way to buy a system with an NPU, so I don't have any. But tomorrow, will they just be included by default? That seems like a waste for those of us who aren't going to be running models. I wonder what other uses they could be put to?

jsheard 2y ago
> But tomorrow, will they just be included by default? That's already the way things are going due to Microsoft decreeing that Copilot+ is the future of Windows, so AMD and Intel are both putting NPUs which meet the Copilot+ performance standard into every consumer part they make going forwards to secure OEM sales.
- AlexAndScripts 2y ago
  It almost makes me want to find some use for them on my Linux box (not that is has an NPU), but I truly can't think of anything. Too small to run a meaningful LLM, and I'd want that in bursts anyway, I hate voice controls (at least with the current tech), and Recall sounds thoroughly useless. Could you do mediocre machine translation on it, perhaps? Local github copilot? An LLM that is purely used to build an abstract index of my notes in the background? Actually, could they be used to make better AI in games? That'd be neat. A shooter character with some kind of organic tactics, or a Civilisation/Stellaris AI that doesn't suck.
  - ywvcbk 2y ago
    > box Presumably you have a GPU? If so there is nothing an NPU can do that a discrete GPU can’t (and it would be much slower than a recent GPU). The real benefits are power efficiency and cost since they are built into the SoC which are not necessarily that useful on a desktop PC.
  - Miraste 2y ago
    In short: no. Current-gen NPUs are so slow they can't do anything useful. AMD and Intel have 2nd-gen ones that came out a few weeks ago, and by spec they may able to run local translation and small LLMs (haven't seen benchmarks yet), but for now they are laptop-only.

kllrnohj 2y ago

Snapdragon X still has a full 12 cores (all same cores, it's homogeneous) and the Strix Point is also 12 cores but in a 4+8 configuration but with the "little" cores not sacrificing that much (nothing like the little cores in ARM's designs which might as well not even exist, they are a complete waste of silicon). Consumer software doesn't scale to that, so what are you going to do with more transistors allocated to the CPU? It's not unlike why Apple puts so many video engines in their SoCs - they don't actually have much else to do with the transistor budget they can afford. Making single thread performance better isn't limited by transistor count anymore and software is bad at multithreading.

wmf 2y ago
GPU "infinity" cache would increase 3D performance and there's a rumor that AMD removed it to make room for the NPU. They're not out of ideas for features to put on the chip.