5 ms·
Why does AMD or Intel not release a medium performant GPU with minimum 128gb of memory for a good consumer price. These models require lots of memory to 'single
by holoduke 3y ago
Why does AMD or Intel not release a medium performant GPU with minimum 128gb of memory for a good consumer price. These models require lots of memory to 'single' pass an operation. Throughput could be bit slower.
A 1080 Nvidia with 256gb of memory would run all these models fast right?
Or am I forgetting something here.
- hackernudes 3y agoI don't think there was a market for it before LLMs. Still might not be (especially if they don't want to cannibalize data center products). Also, they might have hardware constraints. I wouldn't be that surprised if we see some high ram consumer GPUs in the future, though. It won't work out unless it becomes common to run LLMs locally. Kind of a chicken-and-egg problem so I hope they try it!
- the8472 3y ago> I don't think there was a market for it before LLMs. At $work CGI assets sometimes grow pretty big and throwing more VRAM at the problem would be easier than optimizing the scenes in the middle of the workflow. They can be optimized, but that often makes it less ergonomic to work with them. Perhaps asset-streaming (nanite&co) will make this less of an issue, but that's also fairly new. Do LLM implementations already stream the weights layer by layer or in whichever order they're doing the evaluation or is PCIe bandwidth too limited for that?
- tpetry 3y agoBut you are not the home user target audience. They want to sell you the more expensive workstation or server models.
- the8472 3y agoEven an A6000 tops out at 48GB while you can attach terabytes of RAM to server-class CPUs.
- elabajaba 3y agoAMD had the Radeon pro SSG that let you attach 1TB of pcie3 nvme SSDs directly to the GPU, but no one bought them and afaik they were basically unobtainable on the consumer market. Also asset streaming has been a thing for like 20 years now in gaming, it's not really a new thing. Nanite's big thing is that it gets you perfect LODs without having to pre-create them and manually tweak them (eg. how far away does the LOD transition happen, what's the lowest LOD before it disappears, etc)
- the8472 3y agoLoading assets JIT for the next frame from NVMe hasn't been a thing for 20 years though. Different kinds of latency floors. What I was asking is whether LLM inference can be structured in such a way that only a fraction of the weight is needed at a time and then the next ones can be loaded JIT as the processing pipeline advances.
- deleted 3y ago[deleted]
- duxup 3y ago>for a good consumer price Was there a consumer market for them until recently?
- 0xcde4c3db 3y agoProbably because if they take that exact same GPU+VRAM configuration and slap it on a rackmount-optimized board, they can charge AI companies 5-10x the price for it.
- vegabook 3y agothis is where the new third player, Intel, can (if it can tear itself away from identical behaviour in the consumer/server CPU market) hopefully break the duopoly. Love to see a 32 or 64GB card from Intel. Their software stack on Linux is competent enough (unlike the dumpster fire that is AMD's ROCm).
- jsheard 3y agoThey don't even offer that much VRAM on cards aimed at those price-insensitive customers, Nvidias current lineup maxes out at 48GB for GDDR-based models or 80GB for HBM-based models. Even if money is no object there's still practical engineering limits on how much memory they can put on a card without sacrificing bandwidth.
- kristianp 3y agoSo when is Micron due to introduce higher density dram chips? Nvidia is limited to what's available from RAM manufacturers there.
- elabajaba 3y agoSamsung has announced GDDR6W which doubles the density, but also doubles the pin count (so same amount of VRAM in total, but takes up half the space on the board). I'm assuming that companies will save any more density improvements for GDDR7 which is rumored to be used in next-gen GPUs (probably 2nd half 2024).
- fomine3 3y agoThis must be the reason. AI homelab market is too small while enterprise market will become extremely big.
- layer8 3y agoReleasing a new model takes time, and it’s unclear how large the consumer market would actually be. Maybe they’re working on it right now.
- Kye 3y agoGDDR probably hasn't seen the same cost reduction benefits of volume DDR has.
- TaylorAlexander 3y agoOne question I have is: can they use cheaper kinds of RAM and still be perfectly usable for large ML models? They could put 4GB of GDDR and 128GB of cheap RAM maybe? I do realize as others are saying, this would be a new kind of card so they will need time to develop it. But would this work?
- andromeduck 3y agoNot without a redesigned memory controller or one off chip. You'd probably just want the host's memory to be directly accessible over PCIE or something faster like NVLINK. Such solutions already exist just not in the consumer space.
- andromeduck 3y agoBecause then memory would be 90% of the BOM.
- dragonwriter 3y ago> Why does AMD or Intel not release a medium performant GPU with minimum 128gb of memory for a good consumer price. They do. Well, not “medium performant”, but for VRAM-bound tasks they’d still be an improvement over CPUs if you could use them — iGPUs use main memory. What they don’t have is support for them for popular GPGPU frameworks (though there was a third party CUDA-for-Intel-iGPUs a while ago.)
- elabajaba 3y agoBecause they can't do that for a "good consumer price". If you want more than ~48GB, you're looking at HBM which is extremely expensive (HBM chips are very expensive, packaging+interposer is extremely expensive, designing and producing a new GPU is expensive). Normal GPUs are limited by both their bus width (wider bus = more pins = harder to design, more expensive to produce, and increases power consumption), and GDDR6(x) (which maxes out at 2GB/chip currently), so on a 384bit bus (4090/7900xtx, don't expect anyone to make a 512bit busses anymore) you need 12x2GB (GDDR6 uses 32 pins per package) which gives you 24GB. You can double the memory capacity to 48GB, but that requires putting the chips on the back of the GPU which leads to a bunch of cooling issues (and GDDR6 is expensive). Of course, even if they did all that they're selling expensive GPUs to a small niche market and cannibalizing sales of their own high end products (and even if AMD somehow managed to magic up a 128GB gpu for $700 people still wouldn't buy it because so much of the ML software is CUDA only).
- eurekin 3y ago3090 has a lot of vram chips on the back though
- elabajaba 3y agoAnd because of it there were issues with the vram overheating in memory intensive workloads, and on some GPUs the vram even separated off the board. https://www.igorslab.de/en/looming-pads-and-too-hot-gddrx6-memory-siliconitis-on-a-geforce-rtx-3080/ https://www.igorslab.de/en/looming-pads-and-too-hot-gddrx6-m...
- eurekin 3y agoYes I have one that has one fried chip. Once I gave another a Bykski water block with backplate it's been no issue for a year
- fennecfoxy 3y agoAh, filler problems. Microsoft called, they want to tell you about a billion dollar mistake.
- pbhjpbhj 3y agoThere's a type of DMA for GPUs to access NVMe on the motherboard, IIRC. Perhaps that is a better solution here? https://developer.nvidia.com/blog/gpudirect-storage/ https://developer.nvidia.com/blog/gpudirect-storage/
- boppo1 3y agoIsn't pci-e latency dramatically higher than onboard vram?
- Tuna-Fish 3y agoIt would not be trivial to do. GDDR achieves higher speeds than normal DDR mainly by specifying much tighter tolerances on the electrical interface, and using wider interface to the memory chips. This means that using commodity GDDR (which is the only fast DRAM that will be reasonably cheap), you have fairly strict limitations on the maximum amount of RAM your can use with the same GPUs that are manufactured for consumer use. (Typically, at most 4x difference between the lowest-end reasonable configuration and the highest-end one, 2x from higher density modules and 2x from using clamshell memory configuration, although often you only have one type of module for a new memory interface generation.) If the product requires either a new memory or GPU die configuration, it's cost will be very high. The only type of memory that can support very different VRAM sizes for an efficiently utilized bus of the same size is HBM, and so far that is limited to the very high end.
- q7xvh97o2pDhNrh 3y agoDo you happen to know where Apple's integrated approach falls on this spectrum? I was actually wondering about this the other day. A fully maxed out Mac Studio is about $6K, and it comes with a "64-core GPU" and "128GB integrated memory" (whatever any of that means). Would that be enough to run a decent Llama?
- cudder 3y agoThe Mac's "integrated memory" means it's shared between the CPU and GPU. So the GPU can address all of that and you can load giant (by current consumer GPU standards) models. I have no idea how it actually performs though.
- smoldesu 3y agoIt's certainly enough to run a decent Llama, but hardly the most cost-effective. Apple's approach falls between the low-bandwith Intel/AMD laptops and the high-bandwith PCIe HPC components. In a way it's trapped between two markets - ultra-cheap Android/Windows hardware with 4-8gb of RAM that can still do AI inferencing, and ultra-expensive GPGPU setups that are designed to melt these workloads. The genial thing to say is that it performs very favorable against other consumer inferencing hardware. The numbers get ugly fast once you start throwing money at the problem, though.
- fooker 3y agoThat's exactly what the next generation of 'accelerators' will be like. Whether it will be co-located with a GPU for consumer hardware remains to be seen. The thing to determine is how essential running LLMs locally is for consumers. BigTech is pushing hard to make their clouds the only place to run LLMs unfortunately, so unless there is a killer app that is just better locally (like games were for GPUs), this might not change.
- boppo1 3y ago> unless there is a killer app that is just better locally Therapy & relationship bots, like the movie 'Her'. It's ugly, but it's coming.
- fooker 3y agoThere's no technical reason it has to be run locally. Massive privacy implications for sure, but people do consume all sorts of adult material online. Games though, no one has been able to make it work as well as local so far.
- kevingadd 3y agoThe margins on VRAM are pretty bad for them since they don't manufacture it themselves. And every memory module they add needs additional power delivery and memory controller muscle to drive, so adding that memory is going to raise the cost of the card significantly. Most games and consumer workloads won't use all that extra memory. Keep in mind video cards don't use the same kind of RAM as consumer CPUs do, they typically use GDDR or HBM.