6 ms·

The M4 Max needs an enormous 512bit memory bus to extract enough bandwidth out of those LPDDR5x chips, while the GPUs that Intel just launched are 192/160bit an

The M4 Max needs an enormous 512bit memory bus to extract enough bandwidth out of those LPDDR5x chips, while the GPUs that Intel just launched are 192/160bit and even flagships rarely exceed 384bit. They can't just slap more memory on the board, they would need to dedicate significantly more silicon area to memory IO and drive up the cost of the part, assuming their architecture would even scale that wide without hitting weird bottlenecks.

p1esk 2y ago
Apple could do it. Why can’t Intel?
- jsheard 2y ago
  Because Apple isn't playing the same game as everyone else. They have the money and clout to buy out TSMCs bleeding-edge processes and leave everyone else with the scraps, and their silicon is only sold in machines with extremely fat margins that can easily absorb the BOM cost of making huge chips on the most expensive processes money can buy.
  - p1esk 2y ago
    Bleeding edge processes is what Intel specializes in. Unlike Apple, they don’t need TSMC. This should have been a huge advantage for Intel. Maybe that’s why Gelsinger got the boot.
    - AlotOfReading 2y ago
      Intel Arc hardware is manufactured by TSMC, specifically on N6 and N5 for this latest announcement. Intel doesn't currently have nodes competitive with TSMC or excess capacity in their better processes.
      - knowitnone 2y ago
        Serious question, why don't they have excess capacity? They aren't producing many CPUs...that people want.
        AlotOfReading 2y ago
        Because their production capacity is much smaller than TSMC, it's declined over the past few years, and their newer nodes are having yield issues.
        vel0city 2y ago
        Hard to have excess capacity when your latest fabs aren't able to produce anything reliably. They don't even have competitive capacity for all their CPU needs. They have negative spare capacity overall.
    - duskwuff 2y ago
      > Bleeding edge processes is what Intel specializes in. Unlike Apple, they don’t need TSMC. Intel literally outsourced their Arrow Lake manufacturing to TSMC because they couldn't fabricate the parts themselves - their 20A (2nm) process node never reached a production-ready state, and was eventually cancelled about a month ago.
      - ubercore 2y ago
        Wasn't it cancelled in favor of 18A?
        duskwuff 2y ago
        It was, but that only puts them further away from shipping product.
      - p1esk 2y ago
        OK, so the question becomes: TSMC could do it. Why can’t Intel?
        BonoboIO 2y ago
        They are trying … for like 10 years
        ac29 2y ago
        Intel is maybe a year or two behind TSMC right now. They might or might not catch up since it is a moving target, but I dont think there is anything TSMC is doing today that Intel wont be doing in the near future.
    - wtallis 2y ago
      These days, Intel merely specializes in bleeding processes. They spent far too many years believing the unrealistic promises from their fab division, and in the past few years they've been suffering the consequences as the problems are too big to be covered up by the cost savings of vertical integration.
    - jsheard 2y ago
      Intel's foundry side has been floundering so hard that they've resorted to using TSMC themselves in an attempt to keep up with AMD. Their recently launched CPUs are a mix of Intel-made and TSMC-made chiplets, but the latter accounts for most of the die area.
      - anon373839 2y ago
        I'm not certain this is quite as damning as it sounds. My understanding is that the foundry business was intentionally walled off from the product business, and that the latter wasn't going to be treated as a privileged customer.
        barkingcat 2y ago
        no, in fact, it sounds even more damning because client side was able to pick whatever was best on the market, and it wasn't intel. Client side could to learn and customize their designs to use another company's processes (this is an extremely hard thing to do by the way) faster than intel foundry could even get their pants up in the morning. Intel foundry screwed up so badly that Nokia's server division was almost shut down because of Intel Foundry's failure. (imagine being so bad at your job, that your clients go out of business) If Intel client side chose to use Foundry, there just wouldn't be any chips to sell.
    - barkingcat 2y ago
      if you think intel has bleeding edge processes, that hasn't been the case for over 6 years ...
  - JBiserkov 2y ago
    > and their silicon is only sold in machines with extremely fat margins Like the brand new Mini that cost 600 USD and went to 500 during Black week.
    - michaelmrose 2y ago
      The 600 mini is on the m4 with the anemic 10 core GPU. This is for grandama or your 10 year old. The good one which is still slower than m4 max is 2200. If you want the max you need at least a macbook pro starting at 3200 and if you want the better one with 128G RAM it starts at about 5k
      - barkingcat 2y ago
        No matter how few GPU cores the M4 has, it is still an extremely potent product on the whole. Better than most of the pc's out there.
        michaelmrose 2y ago
        Do you mean subjectively because you find Mac pleasant to use? The $600 mini certainly isn't more performant than 85% of desktops.
        my123 2y ago
        > The $600 mini certainly isn't more performant than 85% of desktops. The average desktop isn't exactly a gaming machine. But a corporate box or a low end home desktop with a Core i5 and using the iGPU.
  - ryao 2y ago
    Transistor IO logic scaling died a while ago, which is what prompted AMD to go with a chiplet architecture. Being on a more advanced process does not make implementing an 512-bit memory bus any easier for Apple. If anything, it makes it more expensive for Apple than it would be for Intel.
- dragontamer 2y ago
  Because LPDDR5x is soldered on RAM. Everyone else wants configurable RAM that scales both down (to 16GB) and up (to 2TB), to cover smaller laptops and bigger servers. GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. So the 8GB or 16GB offered by NVidia or AMD is just far superior at vid o game graphics (where textures are the priority)
  - jsheard 2y ago
    > GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. Apple is doing 800GB/sec on the M2 Ultra and should reach about 1TB/sec with the M4 Ultra, but that's still lagging behind GPUs. The 4090 was already at the 1TB/sec mark two years ago, the 5090 is supposedly aiming for 1.5TB/sec, and the H200 is doing 5TB/sec.
    - dragontamer 2y ago
      HBM is kind of not fair lol. But 4096-line bus is gonna have more bandwidth than any competitor. It's pretty expensive though. The 500GB/sec number is for a more ordinary GPU like the B580 Battlemage in the $250ish price range. Obviously the $2000ish 4090 will be better, but I don't expect the typical consumer to be using those.
      - kimixa 2y ago
        But an on-package memory bus has some of the advantages of HBM, just to a lesser extent, so it's arguably comparable as an "intermediate stage" between RAM chips and HBM. Distances are shorter (so voltage drop and capacitance are lower, so can be driven at lower power), routing is more complex but can be worked around by more layers, which increases cost but on a significantly smaller area than required for dimms, and the dimms connections themselves can hurt performance (reflection from poor contacts, optional termination makes things more complex, and the expectations of mix-and-match for dimm vendors and products likely reduce fine tuning possibilities). There's pretty much a direct opposite scaling between flexibility and performance - dimms > soldered ram > on-package ram > die-interconnects.
Der_Einzige 2y ago
It doesn't matter if the "cost is driven up". Nvidia has proven that we're all lil pay pigs for them. 5090 will be 3000$ for 32gb of VRAM. Screenshot this now, it will age well. We'd be happy to pay 5000 for 128gb from Intel.
- pixelpoet 2y ago
  You are absolutely correct, and even my non-prophetic ass echoed exactly the first sentence of the top comment in this HN thread ("Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch?"). Yes, yes, it's not trivial to have a GPU with 128gb of memory with cache tags and so on, but is that really in the same universe of complexity of taking on Nvidia and their CUDA / AI moat any other way? Did Intel ever give the impression they don't know how to design a cache? There really has to be a GOOD reason for this, otherwise everyone involved with this launch is just plain stupid or getting paid off to not pursue this. Saying all this with infinite love and 100% commercial support of OpenCL since version 1.0, a great enjoyer of A770 with 16GB of memory, I live to laugh in the face of people who claimed for over 10 years that OpenCL is deprecated on MacOS (which I cannot stand and will never use, yet the hardware it runs on...) and still routinely crushes powerful desktop GPUs, in reality and practice today.
  - timschmidt 2y ago
    Both Intel and AMD produce server chips with 12 channel memory these days (that's 12x64bit for 768bit) which combined with DDR5 can push effective socket bandwidth beyond 800GB/s, which is well into the area occupied by single GPUs these days. You can even find some attractive deals on motherboard/ram/cpu bundles built around grey market engineering sample CPUs on aliexpress with good reports about usability under Linux. Building a whole new system like this is not exactly as simple as just plugging a GPU into an existing system, but you also benefit from upgradeability of the memory, and not having to use anything like CUDA. llamafile, as an example, really benefits from AVX-512 available in recent CPUs. LLMs are memory bandwidth bound, so it doesn't take many CPU cores to keep the memory bus full. Another benefit is that you can get a large amount of usable high bandwidth memory with a relatively low total system power usage. Some of AMD's parts with 12 channel memory can fit in a 200W system power budget. Less than a single high end GPU.
hughesjj 2y ago
Man, I'm old enough to remember when 512 was a thing for consumer cards back when we had 4-8gb memory Sure that was only gddr5 and not gddr6 or lpddr5, but I would have bet we'd be up to 512bit again 10 years down the line.. (I mean supposedly hbm3 has done 1024-2048bit busses but that seems more research or super high end cards, not consumer)
- jsheard 2y ago
  Rumor is the 5090 will be bringing back the 512bit bus, for a whopping 1.5TB/sec bandwidth.
CoastalCoder 2y ago
> The M4 Max needs an enormous 512bit memory bus to extract enough bandwidth out of those LPDDR5x chips Does M4 Max have 64-byte cache lines? If they can fetch or flush an entire cache line in a single memory-bus transaction, I wonder if that opens up any additional hardware / performance optimizations.
- rowanG077 2y ago
  A single memory transaction is almost always a 16n burst for LPDDR5x.
- my123 2y ago
  > Does M4 Max have 64-byte cache lines? on the CPU side: 64 bytes at L1, 128 byte cachelines at L2
modeless 2y ago
The memory controller would be bigger, and the cost would be higher, but not radically higher. It would be an attractive product for local inference even at triple the current price and the development expense would be 100% justified if it helped Intel get any kind of foothold in the ML market.
Zandikar 2y ago
> They can't just slap more memory on the board, they would need to dedicate significantly more silicon area to memory IO and drive up the cost of the part, In the pedantic sense of just literally slapping more on existing boards? No, they might have one empty spot for an extra BGA VRAM chip, but not enough for the gain's we're talking about. But this is absolutely possible, trivially so for someone like Intel/AMD/NVidia, that has full control over the architectural and design process. Is it a switch they flip at the factory 3 days before shipping? No, obviously not. But if they intended this to be the case ~2 years ago when this was just a product on the drawing board? Absolutely. There is 0 technical/hardware/manufacturing reason they couldn't do this. And considering the "entry level" competitor product is the M4 Max which starts at at least $3,000 (for a 128GB equipped one), the margin on pricing more than exists to cover a few hundred extra in ram and extra overhead in higher-layer more populated PCB's. The real impediment is what you landed on at the end there combined with the greater ecosystem not having support for it. Intel could drop a card that is, by all rights, far better performing hardware than a competing Nvidia GPU, but Nvidia's dominance in API's, CUDA, Networking, Fabric-switches (NVLink, mellanox, bluefield), etc etc for that past 10+ years and all of the skilled labor that is familiar with it would largely render a 128GB Arc GPU a dud on delivery, even if it was priced as a steal. Same thing happened with the Radeon VII. Killer compute card that no one used because while the card itself was phenomenal, the rest of the ecosystem just wasn't there. Now, if intel committed to that card, and poured their considerable resources into that ecosystem, and continued to iterate on that card/family, then now we're talking, but yeah, you can't just 10X VRAM on a card that's currently a non-player in the GPGPU market and expect anyone in the industry to really give a damn. Raise an eyebrow or make a note to check back in a year? Sure. But raise the issue to get a greenlight on the corpo credit line? Fat chance.
- ryao 2y ago
  A 128GB VRAM Intel Arc card at a low price would be an OSS developer’s dream come true. It would be the WRT54G of inference.
  - Zandikar 2y ago
    Of course. A cheap card with oodles of VRAM would benefit some people, I'm not denying that. I'm tackling the question of would it benefit intel (as the original question was "why doesn't intel do this"), and the answer is: Profit/Loss. There's a huge number of people in that community that would love to have such a card. How many are actually willing and able to pony up >=$3k per unit? How many units would they buy? Given all of the other considerations that go into making such cards useful and easy to use (as described), the answer is - in intel's mind - nowhere near enough, especially when the financial side of the company's jimmies are so rustled that they sacked Pat G without a proper replacement and nominated some finance bros in as interim CEO's. Intel is ALREADY taking a big risk and financial burden trying to get into this space in the first place, and they're already struggling, so the prospect of betting the house like that just isn't going to fly for the finance bros that can't see passed the next 2 quarters. To be clear, I personally think there is huge potential value in trying to support the OSS community to, in essence, "crowd source" and speedrun some of that ecosystem by supplying (Compared to the competition) "cheap" cards that aschew the artificial segmentation everyone else is doing and investing in that community. But I'm not running Intel, so while that'd be nice, it's not really relevant.
AnthonyMouse 2y ago
> They can't just slap more memory on the board Why not? It doesn't have to be balanced. RAM is cheap. You would get an affordable card that can hold a large model and still do inference e.g. 4x faster than a CPU. The 128GB card doesn't have to do inference on a 128GB model as fast as a 16GB card does on a 16GB model, it can be slower than that and still faster than any cost-competitive alternative at that size. The extra RAM also lets you do things like load a sparse mixture of experts model entirely into the GPU, which will perform well even on lower end GPUs with less bandwidth because you don't have to stream the whole model for each token, but you do need enough RAM for the whole model because you don't know ahead of time which parts you'll need.
- elabajaba 2y ago
  To get 128GB of RAM on a GPU you'd need at least a 1024 bit bus. GDDR6x is 16Gbit 32 pins, so you'd need 64 GDDR6x chips, which good luck even trying to fit that around the GPU die since traces need to be the same length, and you want to keep them as short as possible. There's also a good chance you can't run a clamshell setup so you'd have to double the bus width to 2048 because 32 GDDR6x chips would kick off way too much heat to be cooled on the back of a GPU. Such a ridiculous setup would obviously be extremely expensive and would use way too much power. A more sensible alternative would be going with HBM, except good luck getting any capacity for that since it's all being used for the extremely high margin data center GPUs. HBM is also extremely expensive both in terms of the cost of buying the chips and due to it's advanced packaging requirements.
  - ryao 2y ago
    You do not need a 1024-bit bus to put 128GB of some DDR variant on a GPU. You could do a 512-bit bus with dual rank memory. The 3090 had a 384-bit bus with dual rank memory and going to 512-bit from that is not much of a leap. This assumes you use 32Gbit chips, which will likely be available in the near future. Interestingly, the GDDR7 specification allows for 64Gbit chips: > the GDDR7 standard officially adds support for 64Gbit DRAM devices, twice the 32Gbit max capacity of GDDR6/GDDR6X https://www.anandtech.com/show/21287/jedec-publishes-gddr7-specifications-pam3-ecc-higher-density https://www.anandtech.com/show/21287/jedec-publishes-gddr7-s...
ryao 2y ago
If their memory IO supports multiple ranks like the RTX 3090 (it used dual rank) did, they could do a new PCB layout and then add more memory chips to it. No additional silicon area would be necessary.

p1esk 2y ago

Apple could do it. Why can’t Intel?

jsheard 2y ago
Because Apple isn't playing the same game as everyone else. They have the money and clout to buy out TSMCs bleeding-edge processes and leave everyone else with the scraps, and their silicon is only sold in machines with extremely fat margins that can easily absorb the BOM cost of making huge chips on the most expensive processes money can buy.
- p1esk 2y ago
  Bleeding edge processes is what Intel specializes in. Unlike Apple, they don’t need TSMC. This should have been a huge advantage for Intel. Maybe that’s why Gelsinger got the boot.
  - AlotOfReading 2y ago
    Intel Arc hardware is manufactured by TSMC, specifically on N6 and N5 for this latest announcement. Intel doesn't currently have nodes competitive with TSMC or excess capacity in their better processes.
    - knowitnone 2y ago
      Serious question, why don't they have excess capacity? They aren't producing many CPUs...that people want.
      - AlotOfReading 2y ago
        Because their production capacity is much smaller than TSMC, it's declined over the past few years, and their newer nodes are having yield issues.
      - vel0city 2y ago
        Hard to have excess capacity when your latest fabs aren't able to produce anything reliably. They don't even have competitive capacity for all their CPU needs. They have negative spare capacity overall.
  - duskwuff 2y ago
    > Bleeding edge processes is what Intel specializes in. Unlike Apple, they don’t need TSMC. Intel literally outsourced their Arrow Lake manufacturing to TSMC because they couldn't fabricate the parts themselves - their 20A (2nm) process node never reached a production-ready state, and was eventually cancelled about a month ago.
    - ubercore 2y ago
      Wasn't it cancelled in favor of 18A?
      - duskwuff 2y ago
        It was, but that only puts them further away from shipping product.
    - p1esk 2y ago
      OK, so the question becomes: TSMC could do it. Why can’t Intel?
      - BonoboIO 2y ago
        They are trying … for like 10 years
      - ac29 2y ago
        Intel is maybe a year or two behind TSMC right now. They might or might not catch up since it is a moving target, but I dont think there is anything TSMC is doing today that Intel wont be doing in the near future.
  - wtallis 2y ago
    These days, Intel merely specializes in bleeding processes. They spent far too many years believing the unrealistic promises from their fab division, and in the past few years they've been suffering the consequences as the problems are too big to be covered up by the cost savings of vertical integration.
  - jsheard 2y ago
    Intel's foundry side has been floundering so hard that they've resorted to using TSMC themselves in an attempt to keep up with AMD. Their recently launched CPUs are a mix of Intel-made and TSMC-made chiplets, but the latter accounts for most of the die area.
    - anon373839 2y ago
      I'm not certain this is quite as damning as it sounds. My understanding is that the foundry business was intentionally walled off from the product business, and that the latter wasn't going to be treated as a privileged customer.
      - barkingcat 2y ago
        no, in fact, it sounds even more damning because client side was able to pick whatever was best on the market, and it wasn't intel. Client side could to learn and customize their designs to use another company's processes (this is an extremely hard thing to do by the way) faster than intel foundry could even get their pants up in the morning. Intel foundry screwed up so badly that Nokia's server division was almost shut down because of Intel Foundry's failure. (imagine being so bad at your job, that your clients go out of business) If Intel client side chose to use Foundry, there just wouldn't be any chips to sell.
  - barkingcat 2y ago
    if you think intel has bleeding edge processes, that hasn't been the case for over 6 years ...
- JBiserkov 2y ago
  > and their silicon is only sold in machines with extremely fat margins Like the brand new Mini that cost 600 USD and went to 500 during Black week.
  - michaelmrose 2y ago
    The 600 mini is on the m4 with the anemic 10 core GPU. This is for grandama or your 10 year old. The good one which is still slower than m4 max is 2200. If you want the max you need at least a macbook pro starting at 3200 and if you want the better one with 128G RAM it starts at about 5k
    - barkingcat 2y ago
      No matter how few GPU cores the M4 has, it is still an extremely potent product on the whole. Better than most of the pc's out there.
      - michaelmrose 2y ago
        Do you mean subjectively because you find Mac pleasant to use? The $600 mini certainly isn't more performant than 85% of desktops.
        my123 2y ago
        > The $600 mini certainly isn't more performant than 85% of desktops. The average desktop isn't exactly a gaming machine. But a corporate box or a low end home desktop with a Core i5 and using the iGPU.
- ryao 2y ago
  Transistor IO logic scaling died a while ago, which is what prompted AMD to go with a chiplet architecture. Being on a more advanced process does not make implementing an 512-bit memory bus any easier for Apple. If anything, it makes it more expensive for Apple than it would be for Intel.
dragontamer 2y ago
Because LPDDR5x is soldered on RAM. Everyone else wants configurable RAM that scales both down (to 16GB) and up (to 2TB), to cover smaller laptops and bigger servers. GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. So the 8GB or 16GB offered by NVidia or AMD is just far superior at vid o game graphics (where textures are the priority)
- jsheard 2y ago
  > GPUs with soldered on RAM has 500GB/sec bandwidths, far in excess of Apples chips. Apple is doing 800GB/sec on the M2 Ultra and should reach about 1TB/sec with the M4 Ultra, but that's still lagging behind GPUs. The 4090 was already at the 1TB/sec mark two years ago, the 5090 is supposedly aiming for 1.5TB/sec, and the H200 is doing 5TB/sec.
  - dragontamer 2y ago
    HBM is kind of not fair lol. But 4096-line bus is gonna have more bandwidth than any competitor. It's pretty expensive though. The 500GB/sec number is for a more ordinary GPU like the B580 Battlemage in the $250ish price range. Obviously the $2000ish 4090 will be better, but I don't expect the typical consumer to be using those.
    - kimixa 2y ago
      But an on-package memory bus has some of the advantages of HBM, just to a lesser extent, so it's arguably comparable as an "intermediate stage" between RAM chips and HBM. Distances are shorter (so voltage drop and capacitance are lower, so can be driven at lower power), routing is more complex but can be worked around by more layers, which increases cost but on a significantly smaller area than required for dimms, and the dimms connections themselves can hurt performance (reflection from poor contacts, optional termination makes things more complex, and the expectations of mix-and-match for dimm vendors and products likely reduce fine tuning possibilities). There's pretty much a direct opposite scaling between flexibility and performance - dimms > soldered ram > on-package ram > die-interconnects.

Der_Einzige 2y ago

It doesn't matter if the "cost is driven up". Nvidia has proven that we're all lil pay pigs for them. 5090 will be 3000$ for 32gb of VRAM. Screenshot this now, it will age well. We'd be happy to pay 5000 for 128gb from Intel.

pixelpoet 2y ago
You are absolutely correct, and even my non-prophetic ass echoed exactly the first sentence of the top comment in this HN thread ("Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch?"). Yes, yes, it's not trivial to have a GPU with 128gb of memory with cache tags and so on, but is that really in the same universe of complexity of taking on Nvidia and their CUDA / AI moat any other way? Did Intel ever give the impression they don't know how to design a cache? There really has to be a GOOD reason for this, otherwise everyone involved with this launch is just plain stupid or getting paid off to not pursue this. Saying all this with infinite love and 100% commercial support of OpenCL since version 1.0, a great enjoyer of A770 with 16GB of memory, I live to laugh in the face of people who claimed for over 10 years that OpenCL is deprecated on MacOS (which I cannot stand and will never use, yet the hardware it runs on...) and still routinely crushes powerful desktop GPUs, in reality and practice today.
- timschmidt 2y ago
  Both Intel and AMD produce server chips with 12 channel memory these days (that's 12x64bit for 768bit) which combined with DDR5 can push effective socket bandwidth beyond 800GB/s, which is well into the area occupied by single GPUs these days. You can even find some attractive deals on motherboard/ram/cpu bundles built around grey market engineering sample CPUs on aliexpress with good reports about usability under Linux. Building a whole new system like this is not exactly as simple as just plugging a GPU into an existing system, but you also benefit from upgradeability of the memory, and not having to use anything like CUDA. llamafile, as an example, really benefits from AVX-512 available in recent CPUs. LLMs are memory bandwidth bound, so it doesn't take many CPU cores to keep the memory bus full. Another benefit is that you can get a large amount of usable high bandwidth memory with a relatively low total system power usage. Some of AMD's parts with 12 channel memory can fit in a 200W system power budget. Less than a single high end GPU.

hughesjj 2y ago

Man, I'm old enough to remember when 512 was a thing for consumer cards back when we had 4-8gb memory Sure that was only gddr5 and not gddr6 or lpddr5, but I would have bet we'd be up to 512bit again 10 years down the line.. (I mean supposedly hbm3 has done 1024-2048bit busses but that seems more research or super high end cards, not consumer)

jsheard 2y ago
Rumor is the 5090 will be bringing back the 512bit bus, for a whopping 1.5TB/sec bandwidth.

CoastalCoder 2y ago

> The M4 Max needs an enormous 512bit memory bus to extract enough bandwidth out of those LPDDR5x chips Does M4 Max have 64-byte cache lines? If they can fetch or flush an entire cache line in a single memory-bus transaction, I wonder if that opens up any additional hardware / performance optimizations.

rowanG077 2y ago
A single memory transaction is almost always a 16n burst for LPDDR5x.
my123 2y ago
> Does M4 Max have 64-byte cache lines? on the CPU side: 64 bytes at L1, 128 byte cachelines at L2

modeless 2y ago

The memory controller would be bigger, and the cost would be higher, but not radically higher. It would be an attractive product for local inference even at triple the current price and the development expense would be 100% justified if it helped Intel get any kind of foothold in the ML market.

Zandikar 2y ago

> They can't just slap more memory on the board, they would need to dedicate significantly more silicon area to memory IO and drive up the cost of the part, In the pedantic sense of just literally slapping more on existing boards? No, they might have one empty spot for an extra BGA VRAM chip, but not enough for the gain's we're talking about. But this is absolutely possible, trivially so for someone like Intel/AMD/NVidia, that has full control over the architectural and design process. Is it a switch they flip at the factory 3 days before shipping? No, obviously not. But if they intended this to be the case ~2 years ago when this was just a product on the drawing board? Absolutely. There is 0 technical/hardware/manufacturing reason they couldn't do this. And considering the "entry level" competitor product is the M4 Max which starts at at least $3,000 (for a 128GB equipped one), the margin on pricing more than exists to cover a few hundred extra in ram and extra overhead in higher-layer more populated PCB's. The real impediment is what you landed on at the end there combined with the greater ecosystem not having support for it. Intel could drop a card that is, by all rights, far better performing hardware than a competing Nvidia GPU, but Nvidia's dominance in API's, CUDA, Networking, Fabric-switches (NVLink, mellanox, bluefield), etc etc for that past 10+ years and all of the skilled labor that is familiar with it would largely render a 128GB Arc GPU a dud on delivery, even if it was priced as a steal. Same thing happened with the Radeon VII. Killer compute card that no one used because while the card itself was phenomenal, the rest of the ecosystem just wasn't there. Now, if intel committed to that card, and poured their considerable resources into that ecosystem, and continued to iterate on that card/family, then now we're talking, but yeah, you can't just 10X VRAM on a card that's currently a non-player in the GPGPU market and expect anyone in the industry to really give a damn. Raise an eyebrow or make a note to check back in a year? Sure. But raise the issue to get a greenlight on the corpo credit line? Fat chance.

ryao 2y ago
A 128GB VRAM Intel Arc card at a low price would be an OSS developer’s dream come true. It would be the WRT54G of inference.
- Zandikar 2y ago
  Of course. A cheap card with oodles of VRAM would benefit some people, I'm not denying that. I'm tackling the question of would it benefit intel (as the original question was "why doesn't intel do this"), and the answer is: Profit/Loss. There's a huge number of people in that community that would love to have such a card. How many are actually willing and able to pony up >=$3k per unit? How many units would they buy? Given all of the other considerations that go into making such cards useful and easy to use (as described), the answer is - in intel's mind - nowhere near enough, especially when the financial side of the company's jimmies are so rustled that they sacked Pat G without a proper replacement and nominated some finance bros in as interim CEO's. Intel is ALREADY taking a big risk and financial burden trying to get into this space in the first place, and they're already struggling, so the prospect of betting the house like that just isn't going to fly for the finance bros that can't see passed the next 2 quarters. To be clear, I personally think there is huge potential value in trying to support the OSS community to, in essence, "crowd source" and speedrun some of that ecosystem by supplying (Compared to the competition) "cheap" cards that aschew the artificial segmentation everyone else is doing and investing in that community. But I'm not running Intel, so while that'd be nice, it's not really relevant.

AnthonyMouse 2y ago

> They can't just slap more memory on the board Why not? It doesn't have to be balanced. RAM is cheap. You would get an affordable card that can hold a large model and still do inference e.g. 4x faster than a CPU. The 128GB card doesn't have to do inference on a 128GB model as fast as a 16GB card does on a 16GB model, it can be slower than that and still faster than any cost-competitive alternative at that size. The extra RAM also lets you do things like load a sparse mixture of experts model entirely into the GPU, which will perform well even on lower end GPUs with less bandwidth because you don't have to stream the whole model for each token, but you do need enough RAM for the whole model because you don't know ahead of time which parts you'll need.

elabajaba 2y ago
To get 128GB of RAM on a GPU you'd need at least a 1024 bit bus. GDDR6x is 16Gbit 32 pins, so you'd need 64 GDDR6x chips, which good luck even trying to fit that around the GPU die since traces need to be the same length, and you want to keep them as short as possible. There's also a good chance you can't run a clamshell setup so you'd have to double the bus width to 2048 because 32 GDDR6x chips would kick off way too much heat to be cooled on the back of a GPU. Such a ridiculous setup would obviously be extremely expensive and would use way too much power. A more sensible alternative would be going with HBM, except good luck getting any capacity for that since it's all being used for the extremely high margin data center GPUs. HBM is also extremely expensive both in terms of the cost of buying the chips and due to it's advanced packaging requirements.
- ryao 2y ago
  You do not need a 1024-bit bus to put 128GB of some DDR variant on a GPU. You could do a 512-bit bus with dual rank memory. The 3090 had a 384-bit bus with dual rank memory and going to 512-bit from that is not much of a leap. This assumes you use 32Gbit chips, which will likely be available in the near future. Interestingly, the GDDR7 specification allows for 64Gbit chips: > the GDDR7 standard officially adds support for 64Gbit DRAM devices, twice the 32Gbit max capacity of GDDR6/GDDR6X https://www.anandtech.com/show/21287/jedec-publishes-gddr7-specifications-pam3-ecc-higher-density https://www.anandtech.com/show/21287/jedec-publishes-gddr7-s...

ryao 2y ago

If their memory IO supports multiple ranks like the RTX 3090 (it used dual rank) did, they could do a new PCB layout and then add more memory chips to it. No additional silicon area would be necessary.