6 ms·

I would love to see a consumer graphics card with 128GB VRAM Would be nice to be able to work with some of the larger open source LLM models.

• datameta 3y ago
  I think that goal is fine and good, but I would rather see huge investments toward in-memory compute like ReRAM and such. If we bridge the efficiency advancements of TinyML with the leap of LLM abilities, perhaps we can start on the road of not being limited by the impact of training on climate.
  • asicsarecool 3y ago
    Why the fuck was this downvoted. Very occasionally I get the feeling HN is entering the /. phase
    • wmf 3y ago
      I didn't downvote it but in-memory compute is crackpot and alternative memory tech is really crackpot. It's not going to happen and it's ridiculous to propose it on the same level as GPUs with more RAM.
      • zozbot234 3y ago
        GPU's are already closer to "in-memory compute" compared to CPU's. It's just taking the existing pattern of NUMA (non-uniform memory access) to a greater extent, as a principled approach to the so-called 'Von Neumann bottleneck'.
      • imtringued 3y ago
        In-memory compute is very easy, you just don't have to fall for the pipe dream of using the same process for both the memory and the compute. All you have to do is follow a package on package strategy like we already do with smartphones. A Raspberry PI 5 gets 25GB/s memory bandwidth and it only has a single DRAM chip if I recall correctly. So if you had a DIMM with 16 of these chips, you would already be on the same bandwidth as HBM. 96 DIMMs and you get 40TB/s memory bandwidth.
        • deleted 3y ago
          [deleted]
      • datameta 3y ago
        Can you explain further why you believe it isn't worth exploring? Is it that you can't imagine how it would scale to the level of today's high performance compute hardware? Just ten years back, squeezing ML models onto microcontrollers sounded completely insane, given their tight memory and power constraints. We've seen NN compilers developed, game-changing techniques like quantization, pruning, and graph-level optimizations pruning. This allowed deployment of ML models in microcontrollers with a newly developed framework like TFLite Micro.
        • wmf 3y ago
          These technologies have already been explored and they failed every time. It's throwing good money after bad. Also, speculative basic research isn't comparable to adding more RAM to a graphics card. You can't substitute one for the other.
  • epistasis 3y ago
    Agree with the comment, but riffing on the last few words: Climate is pretty much my #1 concern about the world, but LLM use of energy is really really far down on the list of important actions for climate. First and foremost are removing roadblocks for deploying existing technologies for clean energy, and speeding up the necessary supporting infrastructure such as transmission and market policies for choosing cheapest possible solutions (over the objections of dinosaur execs that choose last century's solutions). Then the big hard to decarbonize parts of industry like cement and steel, as well as deploying electrolyzers to get ammonia fertilizer production switched over to carbon neutral production rather than from fossil-generated hydrogen. Reducing energy consumption is important for advancing AI in general, but ultimately all its energy consumption will be from clean energy sources anyway, and the switch that needs to happen is that switch in energy sources. Reducing energy use by 2x or 10x is not good enough, we must change the sources fundamentally.
• amelius 3y ago
  That's overkill for any type of graphics application though. What you want is a DL or parallel compute card, not a graphics card. They are far more expensive though because compute doesn't sell to the average consumer like graphics does.
  • sigmoid10 3y ago
    Meh, that's like saying 640kb of RAM is more than anyone would ever need. Demand follows hardware development, which in turn accelerates demand. I'm sure game developers would easily find a way to use 128GB of VRAM if it was commonly available in their target market.
    • late2part 3y ago
      To be fair, 640kb is more than anyone really needs. It's just far, far less than we want.
    • Takennickname 3y ago
      Can you imagine the download sizes?
    • amelius 3y ago
      Turns out that demand for graphics memory got stuck at a point where compute is still hungry for more. That may certainly change but it doesn't help compute much, today.
      • LoganDark 3y ago
        It's supply that's down - Nvidia is strictly enforcing their artificial segmentation where the jump from 24GB to 40GB of VRAM multiplies the price by around 5 (!). They know that if you need a 40GB card, you probably can't use a 24GB card at all because it'd run out of VRAM. They charge such insane prices for those higher capacity cards, because they know that they can make you pay thousands upon thousands for every tiny scrap of extra VRAM, and you will have literally no choice but to pay that price because you need the VRAM. Demand isn't low, demand is actually so high that they don't care if lowly consumers want VRAM, they already have enterprise customers that need it so badly that they'll pay prices higher than most consumers would ever imagine. And that's why they're so stingy with VRAM on the consumer cards. They need to be careful not to accidentally make them useful in the datacenter, so they can ensure that their enterprise customers continue to be forced to pay those extortionary prices. You see, it's all about the money, and it always will be. Capitalism, baby!
• Aurornis 3y ago
  Unfortunately, as soon as you make a card with specifications that make it great at enterprise-grade tasks, it will be bought in mass quantities by people building out data centers. This pushes the price up, as we’ve already seen. So labeling it “consumer” doesn’t really mean much. They’ve tried to enforce the distinction with EULAs before, but that doesn’t work well.
  • sydd 3y ago
    > it will be bought in mass quantities by people building out data centers. Isn't this good? The production at scale effects will kick in, lowering the price and supply will meet demand after some hiccups.
    • Aurornis 3y ago
      If demand drove prices down like that then we’d already have cheap cards available. Demand puts upward pressure on prices. Supply is already maxed out and growing as fast as possible.
      • piva00 3y ago
        > Isn't this good? The production at scale effects will kick in, lowering the price and supply will meet demand after some hiccups. You didn't account for the "hiccups", which can vary from 5-20 years until competition catches up, longer than the life of many companies. In spherical cows worlds of economics that would be just a hiccup.
      • LoganDark 3y ago
        > Supply is already maxed out and growing as fast as possible. No, it's not anywhere near maxed out anymore. The chip shortage has been over for a while; now there is too much supply. But that's not the issue. The issue is that Nvidia really wants to force these enterprise customers to pay extremely high prices. This is why they are so afraid to give any more VRAM to the consumer chips, because if they were at all suitable for VRAM-heavy workloads, every HPC company would buy a couple $2,000 consumer cards in place of each $15,000 datacenter card. That'd lose Nvidia something like 70% which they would find entirely unacceptable.
• karolist 3y ago
  Not exactly what you've asked but Mac Studio exists, with 192GB at that.
  • rrrix1 3y ago
    Calling a Mac Studio equipped with 192GB of RAM "consumer grade" is a big stretch. "Consumer ${thing}" to me is somewhere =< $3000.
    • karolist 3y ago
      True, but you get a full general compute machine for under $6k, after a few years you can sell it for 3-4k and upgrade to the latest one. Personally I think it's the best thing to happen to local LLMs, by accident. I have an M1 Max 64GB and I'm blown away every day by these local models. Apple didn't plan for it, but it happened by accident due to unified memory with godzillion throughput having integrated silicon.
• brucethemoose2 3y ago
  We are getting that (in early 2025?) With AMD Strix Halo. 40 CUs, 256 bit LPDDR5X, 16 CPU cores. Or so the rumors say.
• blackoil 3y ago
  As a side effect we are seeing lot of investment/innovation in 2-13B models. Considering price of 4090, 128GB GPU will be >6000 which practically no one can afford.
• elabajaba 3y ago
  This isn't possible currently unless you use HBM, which is significantly more expensive both to actually buy the HBM, and to package it (and all the capacity for packaging HBM interposers is already used for data center hardware). The most VRAM you could have on a consumer GPU today is 48GB, and that'd be on a 4090 or 7900xtx with clamshell VRAM (which increases cost and makes cooling significantly harder due to putting GDDR6 chips on the back of the GPU where there aren't any fans). To calculate how much VRAM is possible, you just need to divide the GPU's bus width by 32 (or 64 for Samsung's new weird double capacity but double bit width) and multiply that by the largest GDDR capacity currently available (16Gbit). As for why GPUs don't increase their bus width, there have been GPUs with 512bit busses in the past, but it makes it quite a bit more expensive (more vram chips, more traces to run, might require more/heavier PCB layers) and increases power draw.