9 ms·

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD

The company that did 4-cores-forever, has the opportunity to redeem itself, in its next consumer GPU release, by disrupting the "8-16GB VRAM forever" that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point. Intel definitely seems to be doing all the right things on software support.

sitkack 3y ago
What is obvious to us, is an industry standard to Product Managers. When is the last time you have seen an industry player upset the status quo? Intel has not changed that much.
zoobab 3y ago
"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.
- stefanka 3y ago
  That would be an interesting information. Which MB works with with which APU with 32 or more GB of VRAM. Can you post your findings please?
- LoganDark 3y ago
  When has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.
  - zoobab 3y ago
    "APU ever been as fast as a GPU" Ryzen5 has both CPU+GPU on one chip, the BIOS allows you set the amount of VRAM. They share the same RAM bank, you can set 16GB of VRAM and 16GB for the OS if you use a 32GB RAM bank.
    - LoganDark 3y ago
      What I'm saying is that GPUs rely on having the memory close to the die so that it actually has enough bandwidth to saturate the cores. System memory is not very close to the CPU (compared to GPUs), so I have doubts about whether an APU would be able to reach GPU levels of performance over gigabytes of model weights.
whalesalad 3y ago
still wondering why we can't have gpu's with sodimm slots so you can crank the vram
- amir_karbasi 3y ago
  I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.
  - frognumber 3y ago
    The universe used to have hierarchies. Fast memory close, slow memory far. Registers. L1. L2. L3. RAM. Swap. The same thing would make a lot of sense here. Super-fast memory close, with overflow into classic DDR slots. As a footnote, going parallel also helps. 8 sticks of RAM at 1/8 the bandwidth each is the same as one stick of RAM at 8x the bandwidth, if you don't multiplex onto the same traces.
    - riskable 3y ago
      It's not so simple... The way GPU architecture works is that it needs as-fast-as-possible access to its VRAM. The concept of "overflow memory" for a GPU is your PC's RAM. Adding a secondary memory controller and equivalent DRAM to the card itself would only provide a trivial improvement over, "just using the PC RAM". Point of fact: GPUs don't even use all the PCI Express lanes they have available to them! Most GPUs (even top of the line ones like Nvidia's 4090) only use about 8 lanes of bandwidth. This is why some newer GPUs are being offered with M.2 slots so you can add an SSD (https://press.asus.com/news/asus-dual-geforce-rtx-4060-ti-ssd-m2-nvme-thermal-performance/ https://press.asus.com/news/asus-dual-geforce-rtx-4060-ti-ss... ).
    - wongarsu 3y ago
      GPUs have memory hierarchies too. A 4090 has about 16MB of L1 cache and 72MB of L2 cache, followed by the 24GB of GDDR6 RAM, followed by host ram that can be accessed via PCIe. The issue is that GPUs are massively parallel. A 4090 has 128 streaming multiprocessors, each executing 128 "threads" or "lanes" in parallel. If each "thread" works on a different part of memory that leaves you with 1kB of L1 cache per thread, and 4.5kB of L2 cache each. For each clock cycle you might be issuing thousands of request to your memory controller for cache misses and prefetching. That's why you want insanely fast RAM. You can write CUDA code that directly accesses your host memory as a layer beyond that, but usually you want to transfer that data in bigger chunks. You probably could make a card that adds DDR4 slots as an additional level of hierarchy. It's the kind of weird stuff Intel might do (the Phi had some interesting memory layout ideas).
chessgecko 3y ago
Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.
- frognumber 3y ago
  Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.
  - 0cf8612b2e1e 3y ago
    At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all. I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.
    - hnfong 3y ago
      A maxed out Mac Studio probably fits your requirements as stated.
      - 0cf8612b2e1e 3y ago
        If I were willing to drop $4k on that setup, I might as well get the real NVidia offering. The hobbyist market needs something priced well under $1k to make it accessible.
  - rnewme 3y ago
    How comes you don't care about full bandwidth?
riskable 3y ago
No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-1200-80.png.webp https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge problem because in theory the Arc A770 is faster! It's theoretical performance (TFLOPS) is more than twice as fast as an Nvidia 4060 (see: https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-1200-80.png https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-120... ). So why does it perform so poorly? Because everything AI-related has been developed and optimized to run on Nvidia's CUDA. Mostly, this is a mindshare issue. If Intel offered a workstation GPU (i.e. not a ridiculously expensive "enterprise" monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell! They'd sell zillions of them! In fact, I'd wager that they'd be so popular it'd be hard for consumers to even get their hands on one because it would sell out everywhere. It doesn't even need to be the fastest card. It just needs to offer more VRAM than the competition. Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU. How does Intel not see this‽ They have the power to step up and take over a huge section of the market but instead they're just copying (poorly) what everyone else is doing.
- Workaccount2 3y ago
  Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB. Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.
  - dheera 3y ago
    Exactly, I'd love to have 1TB of RAM that can be accessed at 6000 MT/s.
    - talldayo 3y ago
belter 3y ago
AMD making drivers of high quality? I would pay to see that :-)
deleted 3y ago
[deleted]
haunter 3y ago
First crypto then AI, I wish GPUs were left alone for gaming.
- azinman2 3y ago
  Didn’t nvidia try to block this in software by slowing down mining? Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.
  - wongarsu 3y ago
    But Nvidia doesn't want to make consumer compute cards because those might steal market share from the datacenter compute cards they are selling at 5x markup.
    - talldayo 3y ago
      Nvidia does sell consumer compute cards, they're just sold at datacenter compute prices: https://www.newegg.com/p/pl?d=nvidia+quadro https://www.newegg.com/p/pl?d=nvidia+quadro Nvidia's approach to software certainly deserves scrutiny, but their hardware lineup is so robust that I find it hard to complain. Jetson already exists for low-wattage solutions, and gaming cards can run Nvidia datacenter drivers on headless Linux without issue. The consumer compute cards are already here, you just aren't using them.
    - dragonwriter 3y ago
      What we need is more real competition for NVidia cards at all levels, so rather than avoiding competing with themselves, they are worried about actually competing with the competition. But capital doesn't want to invest in competition when it can instead invest in a chance at a moat that allows charging monopoly rents.
- talldayo 3y ago
  Are there actually gamers out there that are still struggling to source GPUs? Even at the height of the mining craze, it was still possible to backorder cards at MSRP if you're patient. The serious crypto and AI nuts are all using custom hardware. Crypto moved onto ASICs for anything power-efficient, and Nvidia's DGX systems aren't being cannibalized from the gaming market.
UncleOxidant 3y ago
> Intel definitely seems to be doing all the right things on software support. Can you elaborate on this? Intel's reputation for software support hasn't been stellar, what's changed?
OkayPhysicist 3y ago
The issue from the manufacturer's perspective is that they've got two different customer bases with wildly different willingness to pay, but not substantially different needs from their product. If Nvidia and AMD didn't split the two markets somehow, then there would be no cards available to the PC market, since the AI companies with much deeper pockets would buy up the lot. This is undesirable from the manufacturer's perspective for a couple reasons, but I suspect a big one is worries that the next AI winter would cause their entire business to crater out, whereas the PC market is pretty reliable for the foreseeable future. Right now, the best discriminator they have is that PC users are willing to put up with much smaller amounts of VRAM.

sitkack 3y ago

What is obvious to us, is an industry standard to Product Managers. When is the last time you have seen an industry player upset the status quo? Intel has not changed that much.

zoobab 3y ago

"It would be poetic to see 32-48GB at a non-eye-watering price point." I heard some Asrock motherboard BIOSes could set the VRAM up to 64GB on Ryzen5. Doing some investigations with different AMD hardware atm.

stefanka 3y ago
That would be an interesting information. Which MB works with with which APU with 32 or more GB of VRAM. Can you post your findings please?
LoganDark 3y ago
When has an APU ever been as fast as a GPU? How much cache does it have, a few hundred megabytes? That can't possibly be enough for matmul, no matter how much slow DDR4/5 is technically addressable.
- zoobab 3y ago
  "APU ever been as fast as a GPU" Ryzen5 has both CPU+GPU on one chip, the BIOS allows you set the amount of VRAM. They share the same RAM bank, you can set 16GB of VRAM and 16GB for the OS if you use a 32GB RAM bank.
  - LoganDark 3y ago
    What I'm saying is that GPUs rely on having the memory close to the die so that it actually has enough bandwidth to saturate the cores. System memory is not very close to the CPU (compared to GPUs), so I have doubts about whether an APU would be able to reach GPU levels of performance over gigabytes of model weights.

whalesalad 3y ago

still wondering why we can't have gpu's with sodimm slots so you can crank the vram

amir_karbasi 3y ago
I believe that the issue is that graphic cards require really fast memory. This requires close memory placement (that's why the memory is so close to the core on the board). expandable memory will not be able to provide the required bandwidth here.
- frognumber 3y ago
  The universe used to have hierarchies. Fast memory close, slow memory far. Registers. L1. L2. L3. RAM. Swap. The same thing would make a lot of sense here. Super-fast memory close, with overflow into classic DDR slots. As a footnote, going parallel also helps. 8 sticks of RAM at 1/8 the bandwidth each is the same as one stick of RAM at 8x the bandwidth, if you don't multiplex onto the same traces.
  - riskable 3y ago
    It's not so simple... The way GPU architecture works is that it needs as-fast-as-possible access to its VRAM. The concept of "overflow memory" for a GPU is your PC's RAM. Adding a secondary memory controller and equivalent DRAM to the card itself would only provide a trivial improvement over, "just using the PC RAM". Point of fact: GPUs don't even use all the PCI Express lanes they have available to them! Most GPUs (even top of the line ones like Nvidia's 4090) only use about 8 lanes of bandwidth. This is why some newer GPUs are being offered with M.2 slots so you can add an SSD (https://press.asus.com/news/asus-dual-geforce-rtx-4060-ti-ssd-m2-nvme-thermal-performance/ https://press.asus.com/news/asus-dual-geforce-rtx-4060-ti-ss... ).
  - wongarsu 3y ago
    GPUs have memory hierarchies too. A 4090 has about 16MB of L1 cache and 72MB of L2 cache, followed by the 24GB of GDDR6 RAM, followed by host ram that can be accessed via PCIe. The issue is that GPUs are massively parallel. A 4090 has 128 streaming multiprocessors, each executing 128 "threads" or "lanes" in parallel. If each "thread" works on a different part of memory that leaves you with 1kB of L1 cache per thread, and 4.5kB of L2 cache each. For each clock cycle you might be issuing thousands of request to your memory controller for cache misses and prefetching. That's why you want insanely fast RAM. You can write CUDA code that directly accesses your host memory as a layer beyond that, but usually you want to transfer that data in bigger chunks. You probably could make a card that adds DDR4 slots as an additional level of hierarchy. It's the kind of weird stuff Intel might do (the Phi had some interesting memory layout ideas).

chessgecko 3y ago

Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.

frognumber 3y ago
Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards. If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.
- 0cf8612b2e1e 3y ago
  At this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all. I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.
  - hnfong 3y ago
    A maxed out Mac Studio probably fits your requirements as stated.
    - 0cf8612b2e1e 3y ago
      If I were willing to drop $4k on that setup, I might as well get the real NVidia offering. The hobbyist market needs something priced well under $1k to make it accessible.
- rnewme 3y ago
  How comes you don't care about full bandwidth?

riskable 3y ago

No kidding... Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren't competitive. You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is twice as fast for AI workloads in reality (see: https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-1200-80.png.webp https://cdn.mos.cms.futurecdn.net/FtXkrY6AD8YypMiHrZuy4K-120... ) This is a huge problem because in theory the Arc A770 is faster! It's theoretical performance (TFLOPS) is more than twice as fast as an Nvidia 4060 (see: https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-1200-80.png https://cdn.mos.cms.futurecdn.net/Q7WgNxqfgyjCJ5kk8apUQE-120... ). So why does it perform so poorly? Because everything AI-related has been developed and optimized to run on Nvidia's CUDA. Mostly, this is a mindshare issue. If Intel offered a workstation GPU (i.e. not a ridiculously expensive "enterprise" monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell! They'd sell zillions of them! In fact, I'd wager that they'd be so popular it'd be hard for consumers to even get their hands on one because it would sell out everywhere. It doesn't even need to be the fastest card. It just needs to offer more VRAM than the competition. Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU. How does Intel not see this‽ They have the power to step up and take over a huge section of the market but instead they're just copying (poorly) what everyone else is doing.

Workaccount2 3y ago
Based on leaks, it looks like intel somehow missed an easy opportunity here. There is an insane demand for high VRAM cards now, and it seems the next intel cards will be 12GB. Intel, screw everything else, just pack as much VRAM in those as you can. Build it and they will come.
- dheera 3y ago
  Exactly, I'd love to have 1TB of RAM that can be accessed at 6000 MT/s.
  - talldayo 3y ago

belter 3y ago

AMD making drivers of high quality? I would pay to see that :-)

deleted 3y ago

[deleted]

haunter 3y ago

First crypto then AI, I wish GPUs were left alone for gaming.

azinman2 3y ago
Didn’t nvidia try to block this in software by slowing down mining? Seems like we just need consumer matrix math cards with literally no video out, and then a different set of requirements for those with a video out.
- wongarsu 3y ago
  But Nvidia doesn't want to make consumer compute cards because those might steal market share from the datacenter compute cards they are selling at 5x markup.
  - talldayo 3y ago
    Nvidia does sell consumer compute cards, they're just sold at datacenter compute prices: https://www.newegg.com/p/pl?d=nvidia+quadro https://www.newegg.com/p/pl?d=nvidia+quadro Nvidia's approach to software certainly deserves scrutiny, but their hardware lineup is so robust that I find it hard to complain. Jetson already exists for low-wattage solutions, and gaming cards can run Nvidia datacenter drivers on headless Linux without issue. The consumer compute cards are already here, you just aren't using them.
  - dragonwriter 3y ago
    What we need is more real competition for NVidia cards at all levels, so rather than avoiding competing with themselves, they are worried about actually competing with the competition. But capital doesn't want to invest in competition when it can instead invest in a chance at a moat that allows charging monopoly rents.
talldayo 3y ago
Are there actually gamers out there that are still struggling to source GPUs? Even at the height of the mining craze, it was still possible to backorder cards at MSRP if you're patient. The serious crypto and AI nuts are all using custom hardware. Crypto moved onto ASICs for anything power-efficient, and Nvidia's DGX systems aren't being cannibalized from the gaming market.

UncleOxidant 3y ago

> Intel definitely seems to be doing all the right things on software support. Can you elaborate on this? Intel's reputation for software support hasn't been stellar, what's changed?

OkayPhysicist 3y ago

The issue from the manufacturer's perspective is that they've got two different customer bases with wildly different willingness to pay, but not substantially different needs from their product. If Nvidia and AMD didn't split the two markets somehow, then there would be no cards available to the PC market, since the AI companies with much deeper pockets would buy up the lot. This is undesirable from the manufacturer's perspective for a couple reasons, but I suspect a big one is worries that the next AI winter would cause their entire business to crater out, whereas the PC market is pretty reliable for the foreseeable future. Right now, the best discriminator they have is that PC users are willing to put up with much smaller amounts of VRAM.