7 ms·
Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide me
by wtallis 2y ago
Just how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre card. "Just add more RAM" doesn't work the way you wish it could.
- treprinum 2y agoM3/M4 Max MacBooks with 128GB RAM are already way better than an A6000 for very large local LLMs. So even if the GPU is as slow as the one in M3/M4 Max (<3070), and using some basic RAM like LPDDR5x it would still be way faster than anything from NVidia.
- deleted 2y ago[deleted]
- kevingadd 2y agoAre you suggesting that Intel 'just' release a GPU at the same price point as an M4 Max SOC? And that there would be a large market for it if they did so? Seems like an extremely niche product that would be demanding to manufacture. The M4 Max makes sense because it's a complete system they can sell to Apple's price-insensitive audience, Intel doesn't have a captive market like that to sell bespoke LLM accelerator cards to yet. If this hypothetical 128GB LLM accelerator was also a capable GPU that would be more interesting but Intel hasn't proven an ability to execute on that level yet.
- m00x 2y agoIt's also impossible and it would need to be a CPU. CPUs and GPUs access memory very differently.
- treprinum 2y agoNothing in my comment says about pricing it at the M4 Max level. Apple charges as much because they can (typing this on an $8000 M3 Max). 128GB LPDDR5 is dirt cheap these days just Apple adds its premium because they like to. Nothing prevents Intel from releasing a basic GPU with that much RAM for under $1k.
- wtallis 2y agoYou're asking for a GPU die at least as large as NVIDIA's TU102 that was $1k in 2018 when paired with only 11GB of RAM (because $1k couldn't get you a fully-enabled die to use 12GB of RAM). I think you're off by at least a factor of two in your cost estimates.
- treprinum 2y agoIntel has Xeon Phi which was a spin-off of their first attempt at GPU so they have a lot of tech in place they can reuse already. They don't need to go with GDDRx/HBMx designs that require large dies.
- ksec 2y agoI don't want to further this discussions but may be you dont realise some of the people who replied to you either design hardware for a living or has been in the hardware industry for longer than 20 years.
- anon373839 2y ago> release a GPU at the same price point as an M4 Max SOC Why would it need to be introduced at Apple's high-margin pricing?
- jsheard 2y agoThe M4 Max needs an enormous 512bit memory bus to extract enough bandwidth out of those LPDDR5x chips, while the GPUs that Intel just launched are 192/160bit and even flagships rarely exceed 384bit. They can't just slap more memory on the board, they would need to dedicate significantly more silicon area to memory IO and drive up the cost of the part, assuming their architecture would even scale that wide without hitting weird bottlenecks.
- p1esk 2y agoApple could do it. Why can’t Intel?
- jsheard 2y agoBecause Apple isn't playing the same game as everyone else. They have the money and clout to buy out TSMCs bleeding-edge processes and leave everyone else with the scraps, and their silicon is only sold in machines with extremely fat margins that can easily absorb the BOM cost of making huge chips on the most expensive processes money can buy.
- p1esk 2y agoBleeding edge processes is what Intel specializes in. Unlike Apple, they don’t need TSMC. This should have been a huge advantage for Intel. Maybe that’s why Gelsinger got the boot.
- AlotOfReading 2y agoIntel Arc hardware is manufactured by TSMC, specifically on N6 and N5 for this latest announcement. Intel doesn't currently have nodes competitive with TSMC or excess capacity in their better processes.
- knowitnone 2y agoSerious question, why don't they have excess capacity? They aren't producing many CPUs...that people want.
- wtallis 2y agoThat would basically mean Intel doubling the size of their current GPU die, with a different memory PHY. They're clearly not ready to make that an affordable card. Maybe when they get around to making a chiplet-based GPU.
- amelius 2y agoWhat if they put 8 identical GPUs in the package, each with 1/8 the memory? Would that be a useful configuration for a modern LLM?
- keyboard_slap 2y agoIt could work, but would it be cost-competitive?
- rini17 2y agoAlso, cooling.
- ben_w 2y agoLast I've heard, the architecture makes that difficult. But my information may be outdated, and even if it isn't, I'm not a hardware designer and may have just misunderstood the limits I hear others discuss.
- numeri 2y agoGPU inference is always a balancing act, trying to avoid bottlenecks on memory bandwidth (loading data from the GPU's global memory/VRAM to the much smaller internal shared memory, where it can be used for calculations) and compute (once the values are loaded). Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs. No matter how well you connect them, it'll be slower than transfer within a single GPU. Still, the fact that you can fit an 8× larger GPU might be worth it to you. It's a trade-off that's almost universally made while training LLMs (sometimes even with the model split down both its width and length), but is much less attractive for inference.
- amelius 2y ago> Splitting the model up between several GPUs would add a third much worse bottleneck – memory bandwidth between the GPUs. What if you allowed the system to only have a shared memory between every neighboring pair of GPUs? Would that make sense for an LLM?
- ksec 2y agoThank You Wtallis. Somewhere along the line, this basic "knowledge" of hardware is completely lost. I dont expect this to be explained in any comment section on old Anandtech. It seems hardware enthusiast has mostly disappeared, I guess that is also why Anandtech closed. We now live in a world where most site are just BS rumours.
- ethbr1 2y agoThat's because Anand Lal Shimpi is a CompE by training. Not too many hardware enthusiast site editors have that academic background. And while fervor can sometimes substitute for education... probably not in microprocessor / system design.
- scrlk 2y agoThe Real World Technologies forum is still an absolute gold mine for hardware discussion. As for articles, IMO, Chips and Cheese is the closest thing we have to Anandtech or RWT in their peak.
- ryao 2y agoQualcomm made an AI inferencing card with 128GB RAM without using HBM: https://www.qualcomm.com/news/onq/2023/11/introducing-qualcomm-cloud-ai-100-ultra https://www.qualcomm.com/news/onq/2023/11/introducing-qualco... Would someone with “basic ‘knowledge’ of hardware” explain why a GPU cannot be made with the same memory configuration?
- westurner 2y ago> "Just add more RAM" doesn't work the way you wish it could. Re: Tomasulo's algorithm the other day: https://news.ycombinator.com/item?id=42231284 https://news.ycombinator.com/item?id=42231284 Cerebras WSE-3 has 44 GB of on-chip SRAM per chip and it's faster than HBM. https://news.ycombinator.com/item?id=41702789#41706409 https://news.ycombinator.com/item?id=41702789#41706409 Intel has HBM2e off-chip RAM in Xeon CPU Max series and GPU Max; What is the difference between DDR, HBM, and Cerebras' 44GB of on-chip SRAM?
- westurner 2y agoHow do architectural bottlenecks due to modified Von Neumann architectures' debuggable instruction pipelines limit computational performance when scaling to larger amounts of off-chip RAM? Tomasulo's algorithm also centralizes on a common data bus (the CPU-RAM data bus) which is a bottleneck that must scale with the amount of RAM. Can in-RAM computation solve for error correction without redundant computation and consensus algorithms? Can on-chip SRAM be built at lower cost? Von Neumann architecture: https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_Neumann_bottleneck https://en.wikipedia.org/wiki/Von_Neumann_architecture#Von_N... : > The term "von Neumann architecture" has evolved to refer to any stored-program computer in which an instruction fetch and a data operation cannot occur at the same time (since they share a common bus). This is referred to as the von Neumann bottleneck, which often limits the performance of the corresponding system. [4] > The von Neumann architecture is simpler than the Harvard architecture (which has one dedicated set of address and data buses for reading and writing to memory and another set of address and data buses to fetch instructions). Modified Harvard architecture > Comparisons: https://en.wikipedia.org/wiki/Modified_Harvard_architecture https://en.wikipedia.org/wiki/Modified_Harvard_architecture C-RAM: Computational RAM > DRAM-based PIM Taxonomy, See also: https://en.wikipedia.org/wiki/Computational_RAM https://en.wikipedia.org/wiki/Computational_RAM SRAM: Static random-access memory https://en.wikipedia.org/wiki/Static_random-access_memory https://en.wikipedia.org/wiki/Static_random-access_memory : > Typically, SRAM is used for the cache and internal registers of a CPU while DRAM is used for a computer's main memory.
- imtringued 2y ago
- ryao 2y agoIt is possible to have multiple memory ranks to reduce the bus width requirements for a given amount of memory. Nvidia has demonstrated that this is doable with GDDR6X on the RTX 3090. The RTX 3090 has a 384-bit bus with 24 memory ICs, despite only needing 12 to reach 384-bit. That means it has every two chips sharing one 32-bit interface, which is a dual rank configuration. If you look at the history of computer memory, you can find many examples of multi-rank configurations. I also recall LR-DIMMs as being another way of achieving this. Achieving 128GB VRAM with a 256-bit bus (which seems like a reasonable bus width) would mean some multiple of 8 chips. If Micron, Samsung or SK Hynix made 128Gb GDDR7 chips, then 8 would suffice. The best right now seems 24Gb, although 32Gb seems likely to follow (and it would likely come sooner if a large customer such as Intel asked for it), so they would just need to have 32 chips in a quad rank configuration to achieve 128GB. This assumes that there is no limit in the GDDR7 specification that prevents quad rank configurations. If there is and it still supports dual rank like GDDR6X did, then a 512-bit bus could be done. It would likely be extremely pricy and require a new chip tape out that has much more IO logic transistors to handle the additional bus width (and IO logic transistor scaling is dead, so the die area would be huge), but it is hypothetically possible. Given how much people are willing to pay for more VRAM, it could make business sense to do. Even if there is no limit in the GDDR7 specification that prevents quadrank, their memory IO logic would need to support it and if it does not, they would need to redesign that and do a new chip tape out in addition to a new board design. This would also be very expensive, although not as expensive as going to a 512-bit memory interface. In summary, adding more memory would cost more to do and it would not improve competitiveness in the target market for these cards, which I imagine is the main reason that they do not do it. By the way, the reason that Nvidia implemented support for 2 chips per channel is because they wanted to be able to reach 48GB VRAM on the workstation variant of the 3090 that is known as the RTX A6000 (non-Ada). I do not know why they used 24x 8Gb chips rather than 12x 16Gb on the 3090, although if I had to guess, it had something to do with rank interleaving.
- wtallis 2y agoHaving four chips per channel is exactly why this is implausible. DDR5 can barely operate with four ranks per channel, at severely reduced speeds. Pulling that off with GDDR6 or GDDR7 is not something we can presume to be possible without specific evidence. The highest-density configurations possible for LPDDR5x are dual-rank and byte mode (one chip per 8 bits of the memory bus, so two chips ganged together to populate a 16-bit channel) — and that still operates at less than half the speed of GDDR6. I've not seen any proposals for buffering LPDDR or GDDR, so an analog to LRDIMMs is not a readily-available technology. GDDR is the memory technology that operates at the edge of what's possible for per-pin bandwidth. Loading that memory bus down with many ranks is not something we can expect to be achievable by just putting down more pads on the PCB.