29 ms·
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to
by treprinum 2y ago
Why don't they just release a basic GPU with 128GB RAM and eat NVidia's local generative AI lunch? The networking effect of all devs porting their LLMs etc. to that card would instantly put them as a major CUDA threat. But beancounters running the company would never get such an idea...
- gs17 2y agoEven 24 or 32 GB for an accessible price would sell out fast. NVIDIA wants $2000 for the 5090 to get 32.
- 01HNNWZ0MV43FF 2y agoJudging by the number of 16 GB laptops I see around, 128 GB of RAM would probably cost a bajillion dollars
- gs17 2y agoOne of the great things about having a desktop is being able to get that much for under $300 instead of the price of a second laptop.
- Cumpiler69 2y agoDesktop PCs RAM don't have the bandwidth and latency required for running AI.
- heraldgeezer 2y ago[flagged]
- pjmlp 2y agoThey all look like amusement parks nowadays.
- heraldgeezer 2y agoYou can get a clean Fractal Design case with 0 lights or RGB. You are showing your ignorance. :) Also, RGB means RGB... just set the color to white for a more clean look? But reading the manual is not for you.
- pjmlp 2y agoNope, because you can't get those at the mall downtown, rather have to custom build, ordering from online shops.
- talldayo 2y agoConsidering how HN professedly struggles to use non-reversible cables, it wouldn't surprise me if someone started to complain about getting skillchecked by a Molex or SATA. And that CPU, with the little gold tab you gotta line up? That's a whopping three ways you could go wrong! Forget about it, might as well just phone Dell or Apple and pay the idiot tax.
- pjmlp 2y agoGiven my experience with PC parts in the past, most folks would be skillchecked when getting any card for their slots, that don't exactly match the motherboard firmware, bus protocol version or speed, even a x.1 variation makes all the difference. Also better check if the UEFI firmware does understand the nice NVMe SSD brand, version that was bought. Be sure to not fill all the slots either, or ensure the electromagnetic radiation from each board doesn't cause issues with their neighbour, they might be a possible reason for random freezes. Custom PCs aren't IKEA, and my time building them is long gone, I have better things to do.
- talldayo 2y ago> Also better check if the UEFI firmware does understand the nice NVMe SSD brand, version that was bought. That's funny! I haven't heard of this being an issue since 2009. And back then, it was an issue with a select few manufacturers that took it upon themselves to modify the stock ACPI tables to behave differently from how the OEM intended it to behave. Not naming any names, but it really was a pain in the ass for the people that bought commercial dual-boot software and couldn't run ostensibly recent OSes. Something you didn't encounter if you braved the impossible gauntlet of building a PC from scratch, as there was no "we know better than you" interruption from companies with a sizable investment in watching UEFI burn. Of course, you still have to put up with other fierce trials such as "which way does the DIMM line up" and "how do screws work" which will triumph over so-called hackers that thought they were smart enough to build a PC. Just a thought. Custom PCs will never consume the world, but we are going to point and laugh if you buy a preconfigured rackmount or tower at the sticker price. That's like watching a car enthusiast proudly exit the used Honda dealership after paying three times the Blue Book price for their ride. Puts you up there with Elon and Zuckerberg in the pantheon of people that let FOMO drive irrational hardware purchases. > Custom PCs aren't IKEA, Correct, IKEA furniture actually demands literacy from the person installing it. > I have better things to do. You're a Hacker News user with 14 years of seniority and enough karma to make the Buddha blush. We both know that if you had anything better to do, you wouldn't be writing paragraph-length replies to arguments you supposedly don't take seriously in the first place. Whatever "better things" exist on your agenda, they must have quite the lengthy refractory period for you to find the time in your horribly busy life for this and the other 63,548 comments you've submit to this site. If you can't prove that PC building is inconvenient, you can't prove that your time is worth something and you can't prove that your alternative is better value, what are you trying to tell me? That you're lazier than a middle schooler and too dumb to shop at Radioshack? Touché and well met, your rhetoric moves me.
- qwytw 2y agoNot the laptop RAM. It costs pennies, Apple's is just charging $200 for 12GB because they can. It's way too slow though.. And Nvidia doesn't want to cannibalize its high end chips but putting more memory into consumer ones.
- Numerlor 2y ago48 GB is at the tail end of what's reasonale for normal GPUs. The IO requires a lot of die space. And intel's architecture is not very space efficient right now compared to nvidia's
- jsheard 2y ago> The IO requires a lot of die space. And even if you spend a lot of die space on memory controllers, you can only fit so many GDDR chips around the GPU core while maintaining signal integrity. HBM sidesteps that issue but it's still too expensive for anything but the highest end accelerators, and the ordinary LPDDR that Apple uses is lacking in bandwidth compared to GDDR, so they have to compensate with ginormous amounts of IO silicon. The M4 Ultra is expected to have similar bandwidth to a 4090 but the former will need a 1024bit bus to get there while the latter is only 384bit.
- Numerlor 2y agoGoing off of how the 4090 and 7900 xtx is arranged I think you could maybe fit on or two chips more around the die over their 12, but that's still a far cry from 128. That would probably just need a shared bus like normal DDR as you're not fitting that much with 16 gbit density
- SmellTheGlove 2y agoWhat if we did what others suggested was the practical limit - 48GB. Then just put 2-3 cards in the system and maybe had a little bridge over a separate bus for them to communicate?
- deleted 2y ago[deleted]
- Numerlor 2y agoI believe that would need some software work from Intel where they're lacking a bit now with their delayed start. Not sure how the frameworks themselves split up the inference work to avoid crossing GPUs as the bandwidth is horrible there. If we're being reasonable and say that you're not using a modern HEDT CPU that costs a couple thousand, the best a consumer botherboard can get right now would be 2x 8x PCIe gen 5 at 32GB/s and one chipset x8 PCIe gen 4 at 16GB/s. I'm not sure if a motherboard like that actually exists but Intel's chipset should allow it; AMD only does x4 to chipset so the third slot is limited by that
- rapsey 2y agoWho manufactures the type of RAM and can they buy enough capacity? I know nVidia bought up the high bandwidth memory supply for years to come.
- blagie 2y agoHonestly, I don't think we would need "this type of RAM." The confused part of this discussion is the belief that we need obscene bandwidth. If I need 300GB/s memory bandwidth for my workload, that can be accomplished with: * One RAM chip with 300GB/s * Two RAM chips with 150GB/s each * Four RAM chips with 75GB/s each Etc. Stepping up from 16GB to 196GB, the bandwidth requirements for each chip go down 10-fold, and you can use much cheaper RAM as a result. And all the signalling requirements relax too. Much of this discussion presumes a 200GB card would individually need the same capacity to each RAM chip as a 12GB card. This is just false. An A770 or 4060-grade card couldn't keep up with that much data. And if I'm using a small model, I can get the same bandwidth by properly distributing it among RAM chips (which most hardware does automatically). An A770 or 4060-grade card, with the same total memory capacity as we have today, but 200GB RAM, would allow us to run high-quality LLMs locally or do high-resolution renders. That wouldn't have the same performance as a $200k card, obviously, but for many inferences uses, that's just not very important. If I were buying for my own uses, I'd want 12x 32GB PC3200 DIMMs for a total of 384GB RAM at $600 for the RAM (say $2k total), with an individual throughput of 25GB/sec and a total throughput of 300GB/sec. I'd be okay with 4060-grade performance. My own uses are a bit niche, and I think for most other people's uses, something with a little more throughput and a little less capacity (48-196GB) might make more sense. But you definitely don't need the same throughput as existing GPU RAM.
- wtallis 2y agoJust how "basic" do you think a GPU can be while having the capability to interface with that much DRAM? Getting there with GDDR6 would require a really wide memory bus even if you could get it to operate with multiple ranks. Getting to 128GB with LPDDR5x would be possible with the 256-bit bus width they used on the top parts of the last generation, but would result in having half the bandwidth of an already mediocre card. "Just add more RAM" doesn't work the way you wish it could.
- treprinum 2y agoM3/M4 Max MacBooks with 128GB RAM are already way better than an A6000 for very large local LLMs. So even if the GPU is as slow as the one in M3/M4 Max (<3070), and using some basic RAM like LPDDR5x it would still be way faster than anything from NVidia.
- deleted 2y ago[deleted]
- kevingadd 2y agoAre you suggesting that Intel 'just' release a GPU at the same price point as an M4 Max SOC? And that there would be a large market for it if they did so? Seems like an extremely niche product that would be demanding to manufacture. The M4 Max makes sense because it's a complete system they can sell to Apple's price-insensitive audience, Intel doesn't have a captive market like that to sell bespoke LLM accelerator cards to yet. If this hypothetical 128GB LLM accelerator was also a capable GPU that would be more interesting but Intel hasn't proven an ability to execute on that level yet.
- m00x 2y agoIt's also impossible and it would need to be a CPU. CPUs and GPUs access memory very differently.
- treprinum 2y agoNothing in my comment says about pricing it at the M4 Max level. Apple charges as much because they can (typing this on an $8000 M3 Max). 128GB LPDDR5 is dirt cheap these days just Apple adds its premium because they like to. Nothing prevents Intel from releasing a basic GPU with that much RAM for under $1k.
- chessgecko 2y agoGDDR isnt like the ram that connects to cpu, it's much more difficult and expensive to add more. You can get up to 48GB with some expensive stacked gddr, but if you wanted to add more stacks you'd need to solve some serious signal timing related headaches that most users wouldn't benefit from. I think the high memory local inference stuff is going to come from "AI enabled" cpus that share the memory in your computer. Apple is doing this now, but cheaper options are on the way. As a shape its just suboptimal for graphics, so it doesn't make sense for any of the gpu vendors to do it.
- treprinum 2y agoThey can use LPDDR5x, it would still massively accelerate inference of large local LLMs that need more than 48GB RAM. Any tensor swapping between CPU RAM and GPU RAM kills the performance.
- chessgecko 2y agoI think we don't really disagree, I just think that this shape isn't really a gpu its just a cpu because it isn't very good for graphics at that point.
- treprinum 2y agoThat's why I said "basic GPU". It doesn't have to be too fast but it should still be way faster than a regular CPU. Intel already has Xeon Phi so a lot of things were developed already (like memory controller, heavy parallel dies etc.)
- chessgecko 2y agoI guess it's hard to know how well this would compete with integrated gpus, especially at a reasonable pricepoint. If you wanted to spend $4000+ on it, it could be very competitive and might look something like nvidias grace-hopper superchip, but if you want the product to be under $1k I think it might be better just to buy separate cards for your graphics and ai stuff.
- beAbU 2y agoThey are probably held back by same reason thats preventing AMD and nVidia from doing it either.
- treprinum 2y agoNVidia and AMD make $$$ on datacenter GPUs so it makes sense they don't want to discount their own high-end. Intel has nothing there so they can happily go for commodization of AI hardware like what Meta did when releasing LLaMA to the wild.
- beAbU 2y agoIs nVidia or AmD offering 128gb cards in any configuration?
- latchkey 2y agoThey aren't "cards" but MI300x has 192GB and MI325x has 256GB.
- phkahler 2y agoYou can run an AMD APU with 128GB of shared RAM.
- treprinum 2y agoIt's too slow and not very compatible. Most BIOSes also don't allow sharing that much memory with GPU (max like 16GB).
- ryao 2y agoIsn’t that setting just a historical thing annd ann integrated GPU is able to access any system memory that is mapped by the IOMMU? I assume this is how it works for people using the NVIDIA Jetson AGX Orin 64GB Developer Kit to do inference. I do not know why it would be different for AMD APUs.
- Muskyinhere 2y agoBecause if they could just do that and it would rival what NVidia has, they would just do it. But obvoiusly they don't. And for reasons: NVidia has worked on CUDA for ages, do you believe they just replace this whole thing in no time?
- treprinum 2y agollama.cpp and its derivatives say yes.
- pjmlp 2y agoA fraction of CUDA capabilities.
- treprinum 2y agoSufficient for LLMs and image/video gen.
- m00x 2y agoFLUX.1 D generation is about a minute at 20 steps on a 4080, but takes 35 minutes on the CPU.
- treprinum 2y ago4080 won't do video due to low RAM. The GPU doesn't have to be as fast there, it can be 5x slower which is still way faster than a CPU. And Intel can iterate from there.
- m00x 2y agoIt won't be 5x slower, it would be 20-50x slower if you would implement it as you said. You can't just "add more ram" to GPUs and have them work the same way. Memory access is completely different than on CPUs.
- 2y ago
- heraldgeezer 2y agoThis is a gaming card. Look at benchmarks.
- whatudb 2y agoMeta comment: "why don't they just" phrase usually indicates significant ignorance about a subject, it's better to learn a little bit before dispensing criticism about beancounters or whatnot. In this case, the die I/O limits precludes more than a reasonable number of DDR channels.
- FuriouslyAdrift 2y agoHBM3E memory is at least 3x the price of DDR5 (it requires 3x the wafer as DDR5) and capacity is sold out for all of 2025 already... that's the price and production bottleneck. High speed, low latency server grade DDR5 is around $800-$1600 for 128GB. Triple that for $2400 - $4800 just for the memory. Still need the GPUs/APUs, card, VRMs, etc. Even the nVidia H100 with "only" 94GB starts at $30k...
- adventured 2y agoNvidia's $30,000 is a 90% margin product at scale. They could charge 1/3 that and still be very profitable. There has rarely been such a profitable large corporation in terms of the combo of profit & margin. Their last quarter was $35b in sales and $26b in gross profit ($21.8b op income; 62% op income margin vs sales). Visa is notorious for their extreme margin (66% op income margin vs sales) due to being basically a brand + transaction network. So the fact that a hardware manufacturer is hitting those levels is truly remarkable. It's very clear that either AMD or Intel could accept far lower margins to go after them. And indeed that's exactly what will be required for any serious attempt to cut into their monopoly position.
- talldayo 2y ago> And indeed that's exactly what will be required for any serious attempt to cut into their monopoly position. You misunderstand why and how Nvidia is a monopoly. Many companies make GPUs, and all those GPUs can be used for computation if you develop compute shaders for them. This part is not the problem, you can already go buy cheaper hardware that outperforms Nvidia if price is your only concern. Software is the issue. That's it - it's CUDA and nothing else. You cannot assail Nvidia's position, and moreover their hardware's value, without a really solid reason for datacenters to own them. Datacenters do not want to own GPUs because once the AI bubble pops they'll be bagholders for Intel and AMD's depreciated software. Nvidia hardware can at least crypto mine, or be leased out to industrial customers that have their own remote CUDA applications. The demand for generic GPU compute is basically nonexistent, the reason this market exists at all is because CUDA exists, and you cannot turn over Nvidia's foothold without accepting that fact. The only way the entire industry can fuck over Nvidia is if they choose to invest in a complete CUDA replacement like OpenCL. That is the only way that Nvidia's value can be actually deposed without any path of recourse for their business, and it will never happen because every single one of Nvidia's competitors hate each other's guts and would rather watch each other die in gladiatorial combat than help each other fight the monster. And Jensen Huang probably revels in it, CUDA is a hedged bet against the industry ever working together for common good.
- elorant 2y agoAMD has a 192GB GPU. I don’t see them eating NVidia’s lunch with it.
- treprinum 2y agoThey are charging as much as Nvidia for it. Now imagine they offered such a card for $2k. Would that allow them to eat Nvidia's lunch?
- p1esk 2y agoWe would also need to imagine AMD fixing their software.
- treprinum 2y agoI think plenty of enthusiastic open source devs would jump at it and fix their software if the software was reasonably open. The same effect as what happened when Meta released LLaMA.
- jjmarr 2y agoIt is open and they regularly merge PRs. https://github.com/ROCm/ROCm/pulls?q=is%3Apr+is%3Aclosed https://github.com/ROCm/ROCm/pulls?q=is%3Apr+is%3Aclosed
- treprinum 2y agoAMD GPUs aren't very attractive to ML folks because they don't outshine Nvidia in any single aspect. Blasting lots of RAM onto a GPU would make it attractive immediately with lots of attention from devs occupied with more interesting things.
- genewitch 2y agodoes the 7900xt outperform the 3090ti? if so, there's already a market because those are the same price. I don't mean in theory are there any workloads that the 7900xt can do better? Even if they're practically equal performance you get a warranty and support with your new 7900xt. also i didn't know there was a 192GB amd GPU.
- daft_pink 2y agoTotally agree. Someone needs to exploit the lack of available gpu memory in graphics cards for model runners. Even training tensors tends to run against memory issues with the current cards.
- zamalek 2y agoI think a better idea would be an NPU with slower memory, or tie it to the system DDR. I don't think consumer inference (possibly even training) applications would need the memory bandwidth offered by GDDR/HBM. Inference on my 7950x is already stupid fast (all things considered). The deeper problem is that the market for this is probably incredibly niche.
- m3kw9 2y agoBecause they can’t
- Sparkyte 2y agoBecause you can't stack that much ram on a GPU without sufficient channels to do so. You could probably do 64GB on GDDR6 but you can't do 128GB on GDDR6 without more memory channels. 2GB per chip per channel is the current limit for GDDR6 this is why HBM was invented. It is why you can only see GPUs with 24GB of memory at the moment. HBM2 can handle 64GB ( 4 x 8GB Stack ) ( Total capacity 128GB ) HBM3 can handle 192GB ( 4 x 24GB Stack ) ( Total capacity 384GB ) You can not do this with GDDR6.
- ryao 2y agoLook at the RTX 3090 and RTX A6000 (the non-Ada one). They both have 24 memory chips with a 384-bit memory bus, but one has 24GB of VRAM and the other has 48GB of VRAM. They both have two chips per channel. This breaks the 24GB VRAM limit that you claim to exist.
- Sparkyte 2y agoNope it doesn't that is 12 channels 2 modules per channel each module at 2GB is the largest the chip gets and the cost of those drastically increase the cost. The problem you experience with GDDR6 and channels is width. You get higher in size but you'll never be able to fill the memory fast enough to be cost effective. Also why HBM exists. Lets say the top is 960GB/s for an A6000 for GPU memory. HBM3 is 3.35 TB/s. If Intel wanted to make better GPUs it needs to switch to HBM.
- ryao 2y ago> Nope it doesn't that is 12 channels 2 modules per channel each module at 2GB You say nope and then proceed to say exactly what I said. When if the ICs are limited to 16Gbit density for now, the 24GB limit you mentioned was wrong, since you assumed single rank was the highest possible. As for HBM, that is earmarked for their gaudi 3 line. There is no chance of it being put into a consumer product, as it would be like selling gold at pyrite pricing.
- bayindirh 2y agoDisclosure: HPC admin who works with NIVIDA cards here. Because, no. It's not as simple as that. NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency. For inter-server communication they have libraries which coordinate cards, workloads and computations. They bought Mellanox, but that can be used by anyone, so there's no lock-in for now. As a tangent, NVIDIA has a whole set of standards for pumping tremendous amount of data in and out of these mesh of cards. Let it be GPU-Direct storage or specialized daemons which handle data transfers on and off cards. If you think that you can connect n cards on PCIe bus and just send workloads to them and solve problems magically, you'll hurt yourself a lot, both performance and psychology wise. You have to build a stack which can perform these things with maximum possible performance to be able to compute with NVIDIA. It's not just emulating CUDA, now. Esp., on the high end of the AI spectrum (GenAI, MultiCard, MultiSystem, etc.). For other lower end, multi-tenant scenarios, they have card virtualization, MIG, etc. for card sharing. You have to complete on that, too, for cloud and smaller applications.
- dgfitz 2y agoHow does any of this make money?
- arcticbull 2y agoHaving the complete ecosystem affords them significant margins.
- dgfitz 2y agoAgainst what?
- arcticbull 2y agoAs of today they have SaaS company margins as a hardware company which is practically unheard of.
- segmondy 2y agoThey don't need to do 128gb, 48gb+ would eat their lunch, Intel and AMD are sleeping.
- kristianp 2y agoThe size of the local inference market is too small. Maybe a couple of thousand LLM enthusiasts? It's not enough to make a profit or even breakeven on the development costs for the hardware.
- dogcomplex 2y agoFor now. This might very well change once the general public realizes they can be movie directors (or generative world gamers) just by downloading some model and plugging in an eGPU. The potential inference market is huge
- bastloing 2y agoI would think someday we can use AI to port the software. Maybe even use AI to design the cards.
- p1necone 2y agoIsn't nobody actually making anything close to a profit in the AI space? When your entire business model is propped up by VC money making your bill of materials that you use to make negative profit cheaper is probably not very high on your list of priorities.
- to11mtm 2y agoI've said this for a while... I do think one challenge is, AFAIK with most GDDR5/6 there's a density issue that requires either larger memory bus paths or other additional complexity to support large sizes. That said, the lack of even a 16GB variant is sus. I'll take some copium in that maybe they're trying to solve the 'size' issue somehow and are just making sure whatever system they use isn't gonna be an i820 MTH debacle before they pull the trigger on announcing it.
- varelse 2y ago[dead]
- ryao 2y agoQualcomm built a card designed to do inferencing with 128GB of RAM: https://www.qualcomm.com/news/onq/2023/11/introducing-qualcomm-cloud-ai-100-ultra https://www.qualcomm.com/news/onq/2023/11/introducing-qualco... I have no idea how much it costs. They do not sell it via PC parts channels.
- ein0p 2y agoPretty obvious stuff, right? I mean, you don't even need HBM for that, you just need a TON of memory channels. Sure, that kind of setup would only be efficient for highly coalesced reads/writes, but that's what you need these days for inference - highly coalesced reads and writes. You could even get by with 64GB of DDR5. DDR5-4800 (rather modest) is 38.4GB/s per channel. To get 1TB/s you'd only need 26 channels. With the more expensive DDR5-6400 you'd only need 20. That doesn't at all sound insurmountable for a company of Intel's caliber. Heck, break up the dies (and the channels) across several chiplets even, if the interconnect is decent it'll still run really well.
- IceHegel 2y agoAfter carefully reviewing all of the other comments explaining the many technical and organizational reasons why they should not “just do that,” I have come to the conclusion that it was a big missed opportunity by intel.
- PeterStuer 2y agoThe question is is there a real market for this? I do think it could bootstrap from local inference enthousiasts, but it is not clearcut. Rather than go all in with 128GB, they could test the waters easily with a cheap 32GB offering and take ot from there.
- lhl 2y agoThis GPU has a 192-bit memory bus. At 32-bit GDDR bus width (well w/ 6, 2x16b data channels per chip), that means you have 6 channels. With regular GDDR6, the largest produced size is 16Gb (2GB), so 12GB is what you get. You could double that up w/ a beefed up PCB if the memory controller supports it to get up to 24GB (in the way workstation cards like W7900 and A6000). Beyond that, you'd have to move to GDDR7 (which has 24Gb/3GB chips incoming) or to HBM stacks, but at that point you're well beyond a "basic GPU". I think the only way you could get to 128GB would be either using regular (LP)DDR or HBM. Note, Apple M chips have weak GPUs with decent MBW and large memory capacities (up to 192GB @ 800GB/s for an M2 Ultra, launched mid 2023) and have not been a major CUDA threat so I don't think your hypothesis actually stands up.
- rldjbpin 2y agoarmchair hardware enthusiast opinion: because silicon in the high-end is expensive, and not just a matter of slapping things together. besides the clear limitation of the memory technology they are using compared to the nvidia's enterprise solution, for such large GPU chips that could really make use of such memory, they need to make binning possible by selling cut-down versions of them as well. nvidia can pull this off because they can sell lower-end chips at the same time. intel is barely making a dent in sales, and making a high-end chip will only be very risky, at the cost of potentially benefiting a niche crowd. > put them as a major CUDA threat. that is a software/ecosystem problem, which hardware alone cannot solve. for all the devs that use Macs, even in AI it is only about inference at the moment. nobody is coming at CUDA for training for the near future. amd tried and failed plenty already.
- deleted 2y ago[deleted]