4 ms·

I guess we will need to see the benchmarks. I’m personally a bit skeptical that you can get decent AI performance out of a CPU even with a built in accelerator

I guess we will need to see the benchmarks. I’m personally a bit skeptical that you can get decent AI performance out of a CPU even with a built in accelerator, but maybe Intel pulled a rabbit out of their hat.

kevincox 3y ago
Why not. If you have an accelerator you have specific hardware for the feature. The is really no fundamental reason it can't be as good as a GPU, if GPU is the best architecture it would just be a GPU. The main question is did they allocate enough space and power budget to the accelerator. Or is it anemic even for basic tasks.
- VHRanger 3y ago
  > The is really no fundamental reason it can't be as good as a GPU, if GPU is the best architecture it would just be a GPU. There is a reason, and that reason is called memory bandwidth.
- smolder 3y ago
  For one thing, the memory on GPUs is generally much faster, with a wider bus than what typical CPUs have. The die sizes can be quite large, too. Putting a full sized CPU in the same package as the accelerator means making some sacrifices.
- hot_gril 3y ago
  GPUs are huge, TPUs etc are also quite large, and CPUs are tiny. I'm no expert on any of these, but intuitively you're losing something cramming that functionality into a way smaller chip. Probably something to do with bandwidth (where big helps) vs latency (where small helps).
  - coffeebeqn 3y ago
    The GPU chip itself isn’t that big if you take the heat sink off. But it has power delivery and memory etc on the board
    - hot_gril 3y ago
      Even the GPU chip itself is still much larger than the CPU chip.
- cyanydeez 3y ago
  software is the fundamental component of AI. ITS basically this month the And accelerators are supported by llm
- coffeebeqn 3y ago
  How useful is an integrated GPU at any AI workloads? People seem to generally use things like 3090 or 4090
MrBuddyCasino 3y ago
Intel marketing emphasizes AI (and iGPU) after it became clear that their Meteor Lake generation would not meet its efficiency targets and wouldn’t significantly improve upon previous generations.
- c0n5pir4cy 3y ago
  Even worse for Intel, AMD announced their 8040 series with both better efficiency and with an NPU which (with what little details we've had so far) can possibly hit much higher TOPs.
  - cyanydeez 3y ago
    the 7000 processors have NPU on the higher end.
llm_nerd 3y ago
We need to define what the benchmark for decent is. There are a lot of smaller models/uses that for a variety of reasons it is beneficial to run locally. audio transcription, a lot of image analysis, and so on, much of it doesn't require a 250W, $2000 GPU. You aren't going to run an LLM on this, but loads of stuff will work great.
SkyMarshal 3y ago
One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. For example, currently most GPUs have a max of 16Gb of NVRAM, but usually 8GB or 12GB (except for more expensive workstation cards and similar [1]). But if you can spec a system with 32GB, 64GB, 96GB, or 128GB or more system memory with sufficient bandwidth to the CPU, then the accelerator can run a broader range of bigger models locally. Afaik Apple M2 Pro and M3 Pro/Max systems are the only ones with that capability right now. [1]:https://www.nvidia.com/en-us/design-visualization/desktop-graphics/ https://www.nvidia.com/en-us/design-visualization/desktop-gr...
- DeathArrow 3y ago
  >One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?
  - michaelt 3y ago
    > Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
  - SkyMarshal 3y ago
    Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...
deleted 3y ago
[deleted]
binkHN 3y ago
Intel Meteor Lake Analysis - Core Ultra 7 155H only convinces with GPU performance https://www.notebookcheck.net/Intel-Meteor-Lake-Analysis-Core-Ultra-7-155H-only-convinces-with-GPU-performance.783320.0.html https://www.notebookcheck.net/Intel-Meteor-Lake-Analysis-Cor...
lowbloodsugar 3y ago
It’s not the CPu it’s the memory bandwidth, isn’t it?

kevincox 3y ago

Why not. If you have an accelerator you have specific hardware for the feature. The is really no fundamental reason it can't be as good as a GPU, if GPU is the best architecture it would just be a GPU. The main question is did they allocate enough space and power budget to the accelerator. Or is it anemic even for basic tasks.

VHRanger 3y ago
> The is really no fundamental reason it can't be as good as a GPU, if GPU is the best architecture it would just be a GPU. There is a reason, and that reason is called memory bandwidth.
smolder 3y ago
For one thing, the memory on GPUs is generally much faster, with a wider bus than what typical CPUs have. The die sizes can be quite large, too. Putting a full sized CPU in the same package as the accelerator means making some sacrifices.
hot_gril 3y ago
GPUs are huge, TPUs etc are also quite large, and CPUs are tiny. I'm no expert on any of these, but intuitively you're losing something cramming that functionality into a way smaller chip. Probably something to do with bandwidth (where big helps) vs latency (where small helps).
- coffeebeqn 3y ago
  The GPU chip itself isn’t that big if you take the heat sink off. But it has power delivery and memory etc on the board
  - hot_gril 3y ago
    Even the GPU chip itself is still much larger than the CPU chip.
cyanydeez 3y ago
software is the fundamental component of AI. ITS basically this month the And accelerators are supported by llm
coffeebeqn 3y ago
How useful is an integrated GPU at any AI workloads? People seem to generally use things like 3090 or 4090

MrBuddyCasino 3y ago

Intel marketing emphasizes AI (and iGPU) after it became clear that their Meteor Lake generation would not meet its efficiency targets and wouldn’t significantly improve upon previous generations.

c0n5pir4cy 3y ago
Even worse for Intel, AMD announced their 8040 series with both better efficiency and with an NPU which (with what little details we've had so far) can possibly hit much higher TOPs.
- cyanydeez 3y ago
  the 7000 processors have NPU on the higher end.

llm_nerd 3y ago

We need to define what the benchmark for decent is. There are a lot of smaller models/uses that for a variety of reasons it is beneficial to run locally. audio transcription, a lot of image analysis, and so on, much of it doesn't require a 250W, $2000 GPU. You aren't going to run an LLM on this, but loads of stuff will work great.

SkyMarshal 3y ago

One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. For example, currently most GPUs have a max of 16Gb of NVRAM, but usually 8GB or 12GB (except for more expensive workstation cards and similar [1]). But if you can spec a system with 32GB, 64GB, 96GB, or 128GB or more system memory with sufficient bandwidth to the CPU, then the accelerator can run a broader range of bigger models locally. Afaik Apple M2 Pro and M3 Pro/Max systems are the only ones with that capability right now. [1]:https://www.nvidia.com/en-us/design-visualization/desktop-graphics/ https://www.nvidia.com/en-us/design-visualization/desktop-gr...

DeathArrow 3y ago
>One benefit of an AI accelerator on CPU die is that the accelerator gets access to system memory. If you build the system with lots of high-bandwidth system memory, as Apple does with their M3 Max systems, then the system can run more capable AI models locally. Should Nvidia be afraid of Apple? Is OpenAI going to ditch their A100 for Macbook Airs?
- michaelt 3y ago
  > Should Nvidia be afraid of Apple? Depends on what you think about how technologies become successful. Some people would say the foundation of nvidia's current dominance is their mass-market gaming GPUs. While every other High Performance Computing technology wanted you to call their enterprise sales team and pay a big markup, any academic or open source developer could get their hands on an nvidia GPU - and that lead to a huge academic and open source ecosystem. In recent years, nvidia has decided if you want more than 24GB of vram, you've got to call their enterprise sales team and pay a big markup. And a lot of people working on LLMs want more than 24GB of vram. On the other hand, nvidia do have a lot of momentum. And of course cloud options are available these days.
- SkyMarshal 3y ago
  Not Apple since they’re not a player in datacenter AI, only local laptop/desktop AI. But AMD yes, since AMD is developing similar architecture APU systems that will roll out in 2024. Intel is also developing them, but unclear if their NPU and GPU performance will be comparable. https://www.amd.com/en/products/accelerators/instinct/mi300.html https://www.amd.com/en/products/accelerators/instinct/mi300.... https://www.techradar.com/computing/cpu/intel-launches-core-ultra-processors-to-power-up-your-next-laptop-with-ai https://www.techradar.com/computing/cpu/intel-launches-core-... There is also some tech that lets you put some model layers on the Nvidia GPU’s local NVRAM and the rest in system RAM. It’s slower though. Some discussion of that here: https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_model_to_locally_run_in_a_low_end_gpu_with_4/ https://www.reddit.com/r/LocalLLaMA/comments/18ire2a/best_mo... https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are_you_people_running_70b_parameter_models/ https://www.reddit.com/r/LocalLLaMA/comments/18io4lp/how_are...

deleted 3y ago

[deleted]

binkHN 3y ago

Intel Meteor Lake Analysis - Core Ultra 7 155H only convinces with GPU performance https://www.notebookcheck.net/Intel-Meteor-Lake-Analysis-Core-Ultra-7-155H-only-convinces-with-GPU-performance.783320.0.html https://www.notebookcheck.net/Intel-Meteor-Lake-Analysis-Cor...

lowbloodsugar 3y ago

It’s not the CPu it’s the memory bandwidth, isn’t it?