3 ms·

For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI.

For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI. It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.

orbital-decay 7mo ago
Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).
- wtallis 7mo ago
  > Also similar to GPU + CPU on the same die, yet here we are. I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way. Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.
fodkodrasz 7mo ago
Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use. That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated. Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.
- alphager 7mo ago
  Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish. This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
  - fisf 7mo ago
    Those use cases are already actively implemented, or being implemented in current software. The 5-10 year estimate is wildly wrong.
wtallis 7mo ago
> Can't imagine a single task in my workflow that would benefit from AI. You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks. > It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction. Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.
- throwa356262 7mo ago
  Is everyone a content creator these days? Besides, most of what you mentioned doesn't run on NPU anyway. They are usually standard GPU workload.
  - wtallis 7mo ago
    None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos. And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)

orbital-decay 7mo ago

Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).

wtallis 7mo ago
> Also similar to GPU + CPU on the same die, yet here we are. I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way. Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.

fodkodrasz 7mo ago

Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use. That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated. Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.

alphager 7mo ago
Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish. This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
- fisf 7mo ago
  Those use cases are already actively implemented, or being implemented in current software. The 5-10 year estimate is wildly wrong.

wtallis 7mo ago

> Can't imagine a single task in my workflow that would benefit from AI. You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks. > It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction. Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.

throwa356262 7mo ago
Is everyone a content creator these days? Besides, most of what you mentioned doesn't run on NPU anyway. They are usually standard GPU workload.
- wtallis 7mo ago
  None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos. And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)