6 ms·

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-1200-80.png.webp https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge problem because in theory the Arc A770 is faster! It's theoretical performance (TFLOPS) is more than twice as fast as an Nvidia 4060 (see: https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-1200-80.png https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-120... ). So why does it perform so poorly? Because everything AI-related has been developed and optimized to run on Nvidia's CUDA. Mostly, this is a mindshare issue. If Intel offered a workstation GPU (i.e. not a ridiculously expensive "enterprise" monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell! They'd sell zillions of them! In fact, I'd wager that they'd be so popular it'd be hard for consumers to even get their hands on one because it would sell out everywhere. It doesn't even need to be the fastest card. It just needs to offer more VRAM than the competition. Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU. How does Intel not see this‽ They have the power to step up and take over a huge section of the market but instead they're just copying (poorly) what everyone else is doing.

Workaccount2 3y ago
Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB. Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.
- dheera 3y ago
  Exactly, I'd love to have 1TB of RAM that can be accessed at 6000 MT/s.
  - talldayo 3y ago
    Optane is crying and punching the walls right now.
    - yjftsjthsd-h 3y ago
      Does optane have an advantage over RAM here?
      - watersb 3y ago
        Optane products were sold as DIMMS with single-DIMM capacity as high as 512 GB. With an Intel memory controller that could make it look like DRAM. 512 GB. It was slower than conventional DRAM. But for AI models, Optane may have an advantage: it's bit-addressable. I'm not aware of any memory controllers that exposed that single-bit granularity; Optane was fighting to create a niche for itself, between DRAM and NAND Flash: pretending to be both, when it was neither. Bit-level operations, computational units in the same device as massive storage, is an architecture that has yet to be developed. AI GPUs try to be such an architecture by plopping 16GB of HBM next to a sea of little dot-product engines.
        wtallis 3y ago
        > But for AI models, Optane may have an advantage: it's bit-addressable. That's an advantage over NAND but not over DRAM. Fundamentally, DRAM is also bit-addressable, but everybody uses DRAM parts with memory cells organized into a hierarchy of groupings for reasons that mostly apply to 3D XPoint memory.
glitchc 3y ago
I think the answer to that is fairly straightforward. Intel isn't in the business of producing RAM. They would have to buy and integrate a third-party product which is likely not something their business side has ever contemplated as a viable strategy.
- monocasa 3y ago
  Their GPUs as sold already include RAM.
  - glitchc 3y ago
    Yes, but they don't fab their own RAM. It's a cost center for them.
    - monocasa 3y ago
      If they can sell the board with more RAM for more than their extra RAM costs, or can sell more GPUs total but the RAM itself is priced essentially at cost, then it's not a cost center.
    - RussianCow 3y ago
      That's not what a cost center is. There is an opportunity for them to make more money by putting more RAM into their GPUs and exposing themselves to a different market. Whether they physically manufacture that RAM doesn't matter in the slightest.
      - glitchc 3y ago
        I agree with you, it makes good business sense. No doubt. I'm merely positing that Intel executives are not used to doing business as integrators. Their core business has always been that of a supplier.
- actionfromafar 3y ago
  I wonder what the market dynamics for NVidia would be if Intel bought as much VRAM as it could.
ponector 3y ago
I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.
- talldayo 3y ago
  > Who will buy it? Frustrated AMD customers willing to put their money where their mouth is?
- bt1a 3y ago
  I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inference can take advantage of splitting the layers among multiple GPUs, high memory GPUs that aren't super expensive are desirable. To share a personal perspective, I have a desktop with a 3090 and an M1 Max Studio with 64GB of memory. I use the M1 for local LLMs because I can use up to 57~GB of memory, even though the output (in terms of tok/s) is much slower than ones I can fit on a 3090.
  - Dalewyn 3y ago
    >models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.
    - hnfong 3y ago
      It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worrying rate these days, but let's keep the discussion on topic...

Workaccount2 3y ago

Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB. Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.

dheera 3y ago
Exactly, I'd love to have 1TB of RAM that can be accessed at 6000 MT/s.
- talldayo 3y ago
  Optane is crying and punching the walls right now.
  - yjftsjthsd-h 3y ago
    Does optane have an advantage over RAM here?
    - watersb 3y ago
      Optane products were sold as DIMMS with single-DIMM capacity as high as 512 GB. With an Intel memory controller that could make it look like DRAM. 512 GB. It was slower than conventional DRAM. But for AI models, Optane may have an advantage: it's bit-addressable. I'm not aware of any memory controllers that exposed that single-bit granularity; Optane was fighting to create a niche for itself, between DRAM and NAND Flash: pretending to be both, when it was neither. Bit-level operations, computational units in the same device as massive storage, is an architecture that has yet to be developed. AI GPUs try to be such an architecture by plopping 16GB of HBM next to a sea of little dot-product engines.
      - wtallis 3y ago
        > But for AI models, Optane may have an advantage: it's bit-addressable. That's an advantage over NAND but not over DRAM. Fundamentally, DRAM is also bit-addressable, but everybody uses DRAM parts with memory cells organized into a hierarchy of groupings for reasons that mostly apply to 3D XPoint memory.

glitchc 3y ago

I think the answer to that is fairly straightforward. Intel isn't in the business of producing RAM. They would have to buy and integrate a third-party product which is likely not something their business side has ever contemplated as a viable strategy.

monocasa 3y ago
Their GPUs as sold already include RAM.
- glitchc 3y ago
  Yes, but they don't fab their own RAM. It's a cost center for them.
  - monocasa 3y ago
    If they can sell the board with more RAM for more than their extra RAM costs, or can sell more GPUs total but the RAM itself is priced essentially at cost, then it's not a cost center.
  - RussianCow 3y ago
    That's not what a cost center is. There is an opportunity for them to make more money by putting more RAM into their GPUs and exposing themselves to a different market. Whether they physically manufacture that RAM doesn't matter in the slightest.
    - glitchc 3y ago
      I agree with you, it makes good business sense. No doubt. I'm merely positing that Intel executives are not used to doing business as integrators. Their core business has always been that of a supplier.
actionfromafar 3y ago
I wonder what the market dynamics for NVidia would be if Intel bought as much VRAM as it could.

ponector 3y ago

I don't agree. Who will buy it? A few enthusiasts who wants to run LLM locally but cannot afford M3 or 4090? It will be a niche product with poor sales.

talldayo 3y ago
> Who will buy it? Frustrated AMD customers willing to put their money where their mouth is?
bt1a 3y ago
I think there's more than a few enthusiasts who would be very interesting in buying 1 or more of these cards (if they had 32+ GB of memory), but I don't have any data to back that opinion up. It is not only those who can't afford a 4090 though. While the 4090 can run models that use less than 24GB of memory at blistering speeds, models are going to continue to scale up and 24GB is fairly limiting. Because LLM inference can take advantage of splitting the layers among multiple GPUs, high memory GPUs that aren't super expensive are desirable. To share a personal perspective, I have a desktop with a 3090 and an M1 Max Studio with 64GB of memory. I use the M1 for local LLMs because I can use up to 57~GB of memory, even though the output (in terms of tok/s) is much slower than ones I can fit on a 3090.
- Dalewyn 3y ago
  >models are going to continue to scale up and 24GB is fairly limiting >24GB is fairly limiting Can I take a moment to suggest that maybe we're very spoiled? 24GB of VRAM is more than most peoples' system RAM, and that is "fairly limiting"? To think Bill once said 640KB would be enough.
  - hnfong 3y ago
    It doesn't matter whether anyone is "spoiled" or not. The fact is large language models require a lot of VRAM, and the more interesting ones need more than 24GB to run. The people who are able to afford systems with more than 24GB VRAM will go buy hardware that gives them that, and when GPU vendors release products with insufficient VRAM they limit their market. I mean inequality is definitely increasing at a worrying rate these days, but let's keep the discussion on topic...