5 ms·

Intel is well behind AMD and Nvidia in terms of GPU processing power and it might take them a while to catch up. BUT - is there anything preventing Intel simpl

Intel is well behind AMD and Nvidia in terms of GPU processing power and it might take them a while to catch up. BUT - is there anything preventing Intel simply adding a ton of RAM to their cards? Why not make a 64GB or 128GB card - even with Intel's underpowered GPU, won't the AI folks all be loving it? Surely that's not that hard to do and yet would make Intel GPUs appealing, would it not? Please correct me as I have no technical expertise on this - maybe there is something about GPUs that means its not possible to add lots of RAM? Nvidia and AMD are notoriously stingy with GPU RAM, and even if there's lots of RAM there's a ridiculously high price. Is this an angle for Intel to get an advantage? And while they are at it, drop 8 or 16 video encoder cores on there too.

wmf 3y ago
GDDR6 doesn't support huge capacities. Maybe Intel could create an A770 32GB though.
- userbinator 3y ago
  What do you mean by "doesn't support huge capacities"? You can always add more channels in parallel.
  - wtallis 3y ago
    Yeah, if you tape out a new chip; but that's a pretty vacuous statement. The capacities of GDDR6 modules available do impose a meaningful constraint on the total RAM that any given GPU can be equipped with, and while Intel is still struggling to get any foothold in the GPU market at all, it would be a big risk for them to invest in bringing to market a chip with eg. 4x the memory controller to compute ratio typically used by GPUs just so they could claim to have the cheapest 128GB GPU.
  - davrosthedalek 3y ago
    At some point you run out of pins, and the routing gets harder and harder. GDDR6 has very hard constraints on signal path length/quality. It gets expensive very quickly.
zontorol 3y ago
You also need memory bandwidth, that's harder than just quantity.
ryukoposting 3y ago
The 3060/4060 16GB is Nvidia's answer to "I'm a hobbyist/tiny business and I need an okay AI rig at a low price." If Intel could jam 32GB onto a card with comparable performance, and still compete on price, they might have a winner. They'd have my money, that's for sure. I want to run Stable Diffusion at stupid resolutions, and I want it cheap.
- andrewstuart 3y ago
  I'm curious to know what prevents Intel putting 64GB or 128GB on a GPU - is there a technical limitation that makes it impossible to go to 128GB? There would certainly be an awkward silence in response from Nvidia. Given that Intel needs some angle to get ahead in the GPU wars I would have thought adding RAM would be an easy way to do that - especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe.
  - ryukoposting 3y ago
    As another commented, it's an easy way to enforce segmentation. So, not a good reason, but a reason. I'd also imagine that the hardware on the smaller/cheaper dies doesn't have an address bus wide enough for 128GB.
  - qball 3y ago
    >especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe. Well, it's only really nVidia. And the reason it's only nVidia doing this is because they don't want to self-cannibalize; if the 4080 had 24GB literally no prosumer or small organization wanting to play with the tech would even be considering the 4090 (and to a limited extent, the same is true of the 3080 vs. the 3090, but everyone was too busy mining Ethereum to notice that). As far as the other manufacturers are concerned, there's no objective need for anything over 16GB on midrange cards (2070/3060/4060 equivalents) because that's the kind of GPU in the consoles and nobody releases exclusively for PC any more; you'll have to upgrade your card to have parity when the next generation comes out anyway so why provide more/drive costs up? It's not like AMD or Intel GPUs are in any way able to use that RAM for anything else, and why offer it now and kneecap themselves should they somehow get into a position where they actually can offer a good AI suite and want to sell 24GB cards for 2000+ dollars too? And nVidia has taken full advantage of that fact, and that's why the 4080 only has 16GB even though the 7900XTX is performance-competitive with it and has the expected memory compliment. The 4080, including the Super, is a sub-par product because it kind of has to be.
dotnet00 3y ago
Having a ton of VRAM doesn't really have much value without the speed to use it. After a point your calculation becomes heavy enough that you can get by with smaller VRAM and just accessing system memory, paying the data transfer or streaming cost.
latchkey 3y ago
> Intel is well behind AMD and Nvidia in terms of GPU processing power and it might take them a while to catch up. Have you read this article? https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-accelerators https://www.databricks.com/blog/llm-training-and-inference-i...
- maxwell22 3y ago
  There I read that Intel Gaudi is behind all NVIDIA products in all metrics except performance per dollar. Since one can only access these in the Intel cloud, where Intel has essentially made the few of these that exist freely available, isn’t perf per dollar kind of a joke metric? I mean, if they are free to use, they have infinite perf per dollar. But if Intel were to sell them, you were to buy them, and then you were to pay for their power, then wouldn’t perf per dollar look completely different ? The article seems to try really hard to avoid saying how much perf per watt these have and how expensive they are. Reads a bit like a marketing piece as part of some Intel / data bricks collab (all companies and vendors do these).
  - Wytwwww 3y ago
    > wouldn’t perf per dollar look completely different Isn't the cost of actually running the card dwarfed by the cost of the card itself? e.g. even if we look at an old card like the A100 at 100% 24/7 that's about 3500 kWh per year which is what $300-$500 depending on the location? if we Intel could come out with a card that could do anything the A100 does at half the price but 4x power usage it could still be somewhat competitive.
    - maxwell22 3y ago
      That’d depend on how many of these cards you need to get the same perf as an A100. Most of the results being shown are batch 1 (use the cards to serve 1 user). But in practice you’d use a single A100 to serve thousands of users concurrently (you can’t do that with the Gaudi 2 though). The article measures that Gaudi 2 is competitive at “latency for batch size 1”, but that isn’t really a metric anyone cares about. So Intel would need to sell these for much less than half, and then perf per watt would need to be much better (the article measures that it is currently worse than A100). Comparing these things is hard, “if Intel were to sell these” is already speculation, since they aren’t on sale. The article is right that perf per dollar is better, but that’s only because Intel is not making money out of these. As a user that’s a red flag, because if that continues these will be discontinued, and then any investment I do right now in supporting these in my SW stack goes to waste.
hackerlight 3y ago
> BUT - is there anything preventing Intel simply adding a ton of RAM to their cards? The market isn't demanding it. For gaming customers, modern games don't need more VRAM, 2k-4k textures can fit into 12-16GB of VRAM and 8k textures mostly don't improve the look at a 4k monitor resolution, unless these textures are stretched over large objects. With upscaling technology, Nvidia customers aren't running at 4k native resolution, so 2k textures do fine for 1080p upscaled to 1440p or 1440p upscaled to 4k. Gaming customers want raster performance primarily, assuming VRAM is above 10-12GB. We will see retail demand for more VRAM when monitor resolutions exceed 4k, and this won't happen on a 1-3 year timeframe. For AI customers, there's already specialized cards like H100.
- mcny 3y ago
  > For AI customers, there's already specialized cards like H100 I don't know anything about machine learning but I think that is the question. Isn't nVidia professional tier like the H100 rare and expensive? I remember someone here on HN started a sharing club where they said they'd pool resources together to buy these things and allow people to basically timeshare or rent them? Is this like the situation where nVidia is basically writing your code for you by providing abstractions like it does with nVidia hair works and things like that on the gaming side? Would Intel have to basically develop its own answer to CUDA or could it gang up with AMD on this?
  - nvm0n2 3y ago
    Yes H100s are very expensive (think ~$40k per unit), and very hard to obtain. For this reason they're basically all sent to cloud operators who can sign large bulk deals up front and then rent them out. At that point rubber really hits the road because it's where the hardware is basically auctioned off on an hour by hour basis, so you can easily pay $40k per month for access to the cards because there's so much money flooding into AI right now. This is why OpenAI seem to have burned through 10s of billions of dollars of Azure compute credits in, like, a few years. Yes CUDA is basically a giant developer platform made by NVIDIA for their cards. It's libraries, tools, languages, tutorials, and other ecosystem stuff like the fact that lots of people use it. Intel and AMD might or might not need their own answer to it. They need at least parts of an answer but CUDA is a lot more than AI, so if that's all you care about, you can skip parts of it. Now AMD has been trying to develop a CUDA competitor for a long time but it's clear either their heart isn't in it, or they just really struggle to attract skilled dev plaform people. ROCm seems to suck so hard people would rather pay NVIDIA's monopoly pricing than deal with it, which is impressively bad. Of course the reason NVIDIA is in this position is that they supported all kinds of R&D with their CUDA effort, and it turned out to be AI that hit big. If you just implements the bits of CUDA that seem most useful you'll always be behind.
godelski 3y ago
> won't the AI folks all be loving it? No, because that's not the only part of the system that matters. This indeed would be useful and Nvidia sure is pushing for more memory (note that they are also continuing development on GPUDirect), but there are limitations. Strapping a SSD onto the side isn't really going to make things any better. Nvidia's secret sauce is CUDA. AMD has been doing great, but they still got a long way to go here. They caught up on hardware, but that's maybe half the battle? If it weren't, you'd see ML people flocking to AMD for lower prices and simply due to availability. It's not that these companies are stingy with RAM, it is that GPU RAM isn't the same RAM you throw into your motherboard. They do have differences. If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too.
- pjmlp 3y ago
  And CUDA is not only C++, rather a quite mature polyglot ecosystem, with graphical debugging and plenty of frameworks to chose from. Also to note, most folks that are driving ISO C++ heterogenous compute story are actually NVidia employees.
- danielEM 3y ago
  > If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too No one nowadays throws a price based on costs + margin for R&D and little extra stay afloat. Those times seem to end somewhere early in 2000s. I wouldn't be surprised at all if that extra 40GB of faster RAM is in 30% cost increase range.
- andrewstuart 3y ago
  Cost matters.
- qwytw 3y ago
  > look at the price difference between an A100 40GB and A100 80GB Nvidia is effectively a monopoly. They can charge whatever they want regardless of cost (which is what they are doing according to their reported margins)
nurettin 3y ago
> even with Intel's underpowered GPU, won't the AI folks all be loving it? No, they need to support at least pascal architecture.
pjmlp 3y ago
Intel has been behind discrete GPUs for decades, even when they thought Larrabee would be a better alternative. While they might score on Linux land by open sourcing their drivers, long time ago their OpenGL drivers weren't that great either.
- qwytw 3y ago
  > Intel has been behind discrete GPUs for decades They only entered the discrete GPU market a couple of years ago to be fair.
  - pjmlp 3y ago
    Fortunely I am old enough to remember what a couple of years ago means, like 1998. https://en.wikipedia.org/wiki/List_of_Intel_graphics_processing_units https://en.wikipedia.org/wiki/List_of_Intel_graphics_process...
    - qwytw 3y ago
      So the last time they released a non integrated GPU prior to 2022 was back in 1998? That seems hardly relevant..
mrtksn 3y ago
That's Apple Silicon territory. It's always amusing how all these LLM or diffusion models work on some beefy Nvidia GPU's or Macbook Air M1's with 16GB of RAM because all that RAM can be utilised by the GPU or the Neural Engine.

wmf 3y ago

GDDR6 doesn't support huge capacities. Maybe Intel could create an A770 32GB though.

userbinator 3y ago
What do you mean by "doesn't support huge capacities"? You can always add more channels in parallel.
- wtallis 3y ago
  Yeah, if you tape out a new chip; but that's a pretty vacuous statement. The capacities of GDDR6 modules available do impose a meaningful constraint on the total RAM that any given GPU can be equipped with, and while Intel is still struggling to get any foothold in the GPU market at all, it would be a big risk for them to invest in bringing to market a chip with eg. 4x the memory controller to compute ratio typically used by GPUs just so they could claim to have the cheapest 128GB GPU.
- davrosthedalek 3y ago
  At some point you run out of pins, and the routing gets harder and harder. GDDR6 has very hard constraints on signal path length/quality. It gets expensive very quickly.

zontorol 3y ago

You also need memory bandwidth, that's harder than just quantity.

ryukoposting 3y ago

The 3060/4060 16GB is Nvidia's answer to "I'm a hobbyist/tiny business and I need an okay AI rig at a low price." If Intel could jam 32GB onto a card with comparable performance, and still compete on price, they might have a winner. They'd have my money, that's for sure. I want to run Stable Diffusion at stupid resolutions, and I want it cheap.

andrewstuart 3y ago
I'm curious to know what prevents Intel putting 64GB or 128GB on a GPU - is there a technical limitation that makes it impossible to go to 128GB? There would certainly be an awkward silence in response from Nvidia. Given that Intel needs some angle to get ahead in the GPU wars I would have thought adding RAM would be an easy way to do that - especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe.
- ryukoposting 3y ago
  As another commented, it's an easy way to enforce segmentation. So, not a good reason, but a reason. I'd also imagine that the hardware on the smaller/cheaper dies doesn't have an address bus wide enough for 128GB.
- qball 3y ago
  >especially given Nvidia and AMD seem to dish out RAM like its the most precious substance in the universe. Well, it's only really nVidia. And the reason it's only nVidia doing this is because they don't want to self-cannibalize; if the 4080 had 24GB literally no prosumer or small organization wanting to play with the tech would even be considering the 4090 (and to a limited extent, the same is true of the 3080 vs. the 3090, but everyone was too busy mining Ethereum to notice that). As far as the other manufacturers are concerned, there's no objective need for anything over 16GB on midrange cards (2070/3060/4060 equivalents) because that's the kind of GPU in the consoles and nobody releases exclusively for PC any more; you'll have to upgrade your card to have parity when the next generation comes out anyway so why provide more/drive costs up? It's not like AMD or Intel GPUs are in any way able to use that RAM for anything else, and why offer it now and kneecap themselves should they somehow get into a position where they actually can offer a good AI suite and want to sell 24GB cards for 2000+ dollars too? And nVidia has taken full advantage of that fact, and that's why the 4080 only has 16GB even though the 7900XTX is performance-competitive with it and has the expected memory compliment. The 4080, including the Super, is a sub-par product because it kind of has to be.

dotnet00 3y ago

Having a ton of VRAM doesn't really have much value without the speed to use it. After a point your calculation becomes heavy enough that you can get by with smaller VRAM and just accessing system memory, paying the data transfer or streaming cost.

latchkey 3y ago

> Intel is well behind AMD and Nvidia in terms of GPU processing power and it might take them a while to catch up. Have you read this article? https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-accelerators https://www.databricks.com/blog/llm-training-and-inference-i...

maxwell22 3y ago
There I read that Intel Gaudi is behind all NVIDIA products in all metrics except performance per dollar. Since one can only access these in the Intel cloud, where Intel has essentially made the few of these that exist freely available, isn’t perf per dollar kind of a joke metric? I mean, if they are free to use, they have infinite perf per dollar. But if Intel were to sell them, you were to buy them, and then you were to pay for their power, then wouldn’t perf per dollar look completely different ? The article seems to try really hard to avoid saying how much perf per watt these have and how expensive they are. Reads a bit like a marketing piece as part of some Intel / data bricks collab (all companies and vendors do these).
- Wytwwww 3y ago
  > wouldn’t perf per dollar look completely different Isn't the cost of actually running the card dwarfed by the cost of the card itself? e.g. even if we look at an old card like the A100 at 100% 24/7 that's about 3500 kWh per year which is what $300-$500 depending on the location? if we Intel could come out with a card that could do anything the A100 does at half the price but 4x power usage it could still be somewhat competitive.
  - maxwell22 3y ago
    That’d depend on how many of these cards you need to get the same perf as an A100. Most of the results being shown are batch 1 (use the cards to serve 1 user). But in practice you’d use a single A100 to serve thousands of users concurrently (you can’t do that with the Gaudi 2 though). The article measures that Gaudi 2 is competitive at “latency for batch size 1”, but that isn’t really a metric anyone cares about. So Intel would need to sell these for much less than half, and then perf per watt would need to be much better (the article measures that it is currently worse than A100). Comparing these things is hard, “if Intel were to sell these” is already speculation, since they aren’t on sale. The article is right that perf per dollar is better, but that’s only because Intel is not making money out of these. As a user that’s a red flag, because if that continues these will be discontinued, and then any investment I do right now in supporting these in my SW stack goes to waste.

hackerlight 3y ago

> BUT - is there anything preventing Intel simply adding a ton of RAM to their cards? The market isn't demanding it. For gaming customers, modern games don't need more VRAM, 2k-4k textures can fit into 12-16GB of VRAM and 8k textures mostly don't improve the look at a 4k monitor resolution, unless these textures are stretched over large objects. With upscaling technology, Nvidia customers aren't running at 4k native resolution, so 2k textures do fine for 1080p upscaled to 1440p or 1440p upscaled to 4k. Gaming customers want raster performance primarily, assuming VRAM is above 10-12GB. We will see retail demand for more VRAM when monitor resolutions exceed 4k, and this won't happen on a 1-3 year timeframe. For AI customers, there's already specialized cards like H100.

mcny 3y ago
> For AI customers, there's already specialized cards like H100 I don't know anything about machine learning but I think that is the question. Isn't nVidia professional tier like the H100 rare and expensive? I remember someone here on HN started a sharing club where they said they'd pool resources together to buy these things and allow people to basically timeshare or rent them? Is this like the situation where nVidia is basically writing your code for you by providing abstractions like it does with nVidia hair works and things like that on the gaming side? Would Intel have to basically develop its own answer to CUDA or could it gang up with AMD on this?
- nvm0n2 3y ago
  Yes H100s are very expensive (think ~$40k per unit), and very hard to obtain. For this reason they're basically all sent to cloud operators who can sign large bulk deals up front and then rent them out. At that point rubber really hits the road because it's where the hardware is basically auctioned off on an hour by hour basis, so you can easily pay $40k per month for access to the cards because there's so much money flooding into AI right now. This is why OpenAI seem to have burned through 10s of billions of dollars of Azure compute credits in, like, a few years. Yes CUDA is basically a giant developer platform made by NVIDIA for their cards. It's libraries, tools, languages, tutorials, and other ecosystem stuff like the fact that lots of people use it. Intel and AMD might or might not need their own answer to it. They need at least parts of an answer but CUDA is a lot more than AI, so if that's all you care about, you can skip parts of it. Now AMD has been trying to develop a CUDA competitor for a long time but it's clear either their heart isn't in it, or they just really struggle to attract skilled dev plaform people. ROCm seems to suck so hard people would rather pay NVIDIA's monopoly pricing than deal with it, which is impressively bad. Of course the reason NVIDIA is in this position is that they supported all kinds of R&D with their CUDA effort, and it turned out to be AI that hit big. If you just implements the bits of CUDA that seem most useful you'll always be behind.

godelski 3y ago

> won't the AI folks all be loving it? No, because that's not the only part of the system that matters. This indeed would be useful and Nvidia sure is pushing for more memory (note that they are also continuing development on GPUDirect), but there are limitations. Strapping a SSD onto the side isn't really going to make things any better. Nvidia's secret sauce is CUDA. AMD has been doing great, but they still got a long way to go here. They caught up on hardware, but that's maybe half the battle? If it weren't, you'd see ML people flocking to AMD for lower prices and simply due to availability. It's not that these companies are stingy with RAM, it is that GPU RAM isn't the same RAM you throw into your motherboard. They do have differences. If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too.

pjmlp 3y ago
And CUDA is not only C++, rather a quite mature polyglot ecosystem, with graphical debugging and plenty of frameworks to chose from. Also to note, most folks that are driving ISO C++ heterogenous compute story are actually NVidia employees.
danielEM 3y ago
> If it were easy, trust me, Nvidia and AMD would both pile on RAM and charge out the wazoo for it. I mean... look at the price difference between an A100 40GB and A100 80GB. They're the same chipset and there's a 2x price difference. Though I should mention the 80GB version is faster memory too No one nowadays throws a price based on costs + margin for R&D and little extra stay afloat. Those times seem to end somewhere early in 2000s. I wouldn't be surprised at all if that extra 40GB of faster RAM is in 30% cost increase range.
andrewstuart 3y ago
Cost matters.
qwytw 3y ago
> look at the price difference between an A100 40GB and A100 80GB Nvidia is effectively a monopoly. They can charge whatever they want regardless of cost (which is what they are doing according to their reported margins)

nurettin 3y ago

> even with Intel's underpowered GPU, won't the AI folks all be loving it? No, they need to support at least pascal architecture.

pjmlp 3y ago

Intel has been behind discrete GPUs for decades, even when they thought Larrabee would be a better alternative. While they might score on Linux land by open sourcing their drivers, long time ago their OpenGL drivers weren't that great either.

qwytw 3y ago
> Intel has been behind discrete GPUs for decades They only entered the discrete GPU market a couple of years ago to be fair.
- pjmlp 3y ago
  Fortunely I am old enough to remember what a couple of years ago means, like 1998. https://en.wikipedia.org/wiki/List_of_Intel_graphics_processing_units https://en.wikipedia.org/wiki/List_of_Intel_graphics_process...
  - qwytw 3y ago
    So the last time they released a non integrated GPU prior to 2022 was back in 1998? That seems hardly relevant..

mrtksn 3y ago

That's Apple Silicon territory. It's always amusing how all these LLM or diffusion models work on some beefy Nvidia GPU's or Macbook Air M1's with 16GB of RAM because all that RAM can be utilised by the GPU or the Neural Engine.