6 ms·
We need a lot more memory for desktop GPUs with the need to run AI locally for productivity and gaming. 48gb+ would be the minimum ideal for running 70b+ models
by jadbox 3y ago
We need a lot more memory for desktop GPUs with the need to run AI locally for productivity and gaming. 48gb+ would be the minimum ideal for running 70b+ models. If Nvidia, AMD, and Intel fail to deliver more to desktop users, I can see a lot more people switching to Apple's M3 or later chips with 80-128gb shared memory.
Similar to rewind.ai, I want a 100% offline AI with Vision to run on every file and image I touch. Every file I use can be categorized and available to prompt against.
- brookst 3y agoI really hope that one day I will run into someone involved in Apple Silicon design who will tell me whether UMA was chosen with post-graphics GPU needs like AI in mind.
- brucethemoose2 3y agoThe UMA itself actually isn't being used in AI, just the big memory pool for a GPU. Intel canceled, and AMD sidelined, big APUs because they found the market didn't want them.
- brookst 3y agoA 128GB MacBook can devote almost all of that memory to AI.
- zamadatix 3y agoI don't think you'll find anything in the desktop segment cracking beyond 48 GB in the next 5 years. That said, the M3s with that much shared memory are really in the workstation segment, and priced like it. Macs are actually a great value for this use case all the way up through maxing them out. The inference performance itself is a bit lacking though, but at least you can do it.
- brucethemoose2 3y agoThere's also the huge caveat of CUDA project compatibility. AMD/Intel seem to be a more promising in that regard, and thats not saying much. Not that it isn't appealing... If the price is right. Also, both AMD and Intel are both reportedly coming out with M Pro-like SoCs. AMD's "Strix Halo" is rumored for 2025. Less is known about Intel's Arrow Lake SKU at the moment.
- sangnoir 3y agoHow many people are writing their own CUDA kernels? I think Pytorch compatibility is a bigger win for AMD amd Intel, when solving for the vast majority of AI practitioners who live in Jupyter Notebooks
- brucethemoose2 3y agoI don't see the distinction. Tons of popular PyTorch projects run hand made CUDA kernels, whether through a library or one written themselves. In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch, but the performance hit is often severe, and not many inference projects are using torch.compile or triton kernels to compensate.
- sangnoir 3y ago> I don't see the distinction Performant Pytorch compatibility can be achieved without CUDA compatibility. > In some cases one can replace rocm/openvino incompatible libraries with pure PyTorch This is the crux of my point: AMD ought to focus on fixing these Pytorch incompatibilities instead of splitting focus with chasing CUDA translations.
- Baldbvrhunter 3y agoGreater than zero. Julia has a CUDA compiling and I've hand crafted CUDA kernels myself. https://cuda.juliagpu.org/stable/ https://cuda.juliagpu.org/stable/
- 3y ago
- ttt3ts 3y ago5 years?! You think the enterprise hardware hype will last that long? When it slows companies will look at other less valuable markets. Unless something fundamentally changes with the capabilities of produced tech I think demand will be saturated before 5 years.
- zamadatix 3y agoI think, in general, hardware markets are permanently splitting and the high end side will not stop scaling for at least a decade. From storage to network to compute the needs of the large scale customers grows ever farther from the needs of the consumer. The hyperscaler class are rushing in 400G NICs with hundreds of CPU cores, TBs of RAM, and hundreds of TBs of high speed flash per rack unit asking when they'll be able to make that 800G or 1.6 T and so on. Meanwhile consumers are just warming up to the idea wired and wireless connections can be slightly faster than a gig and their wild gaming desktop can have 16 cores with 64 GB of RAM and 4 TB of flash storage. GPUs aren't really any different - AMD has even had a completely separate computer architecture for years and Nvidia has entire custom heterogenous scale out designs. If you had asked me "do you think it's possible x86 will scale to over 500 cores per rack unit by 2023" in 2013 I would have said "doubt it, Xeon Phi doesn't even look to be on track for people to want to half ass getting to that number". If you were to ask me now how much we'll have scaled in 10 years this time I'd rather now say "not sure, but we probably aren't going to have stopped cold at the 5 year mark because the hype for more scaling died out". As a result of this split I think consumer hardware will still continue to grow slower. Not because of lack of capability but lack of market interest. People don't have a use case for hundreds of cores or the like, they'd rather smaller, quieter, and more efficient at ever decreasing price points. I'm much more worried investment in consumer GPUs and CPUs for traditional desktops will actually die out and start to stagnate just because the market is getting squeezed ever smaller from both the low end and high end sides.
- ttt3ts 3y agoInteresting. I guess I agree on the idea just not the specific number of 48GB. Amd & Nvidia both offer prosumer GPUs with 48GB, the W7900 and A6000 respectively. I don't see how they would not increase that in 5 years?
- jmnicolas 3y agoI don't think it represents a lot of people, Apple high end offering is absolutely unaffordable for most of us. If you can afford a max specced M3 you can also afford 2 RTX 4090 which should theoretically be faster.
- joshstrange 3y agoA max-specced M3 would give you 128GB of ram vs 48 total from the 2 RTX cards and you'll still need a computer (CPU/MB/RAM/case/fans/etc). They might even out in price but you have more ram with the M3 and a way lower power usage if you care about (I know some people don't), not to mention it's portable. If all you want to do is AI/ML then yes, a M3 is not your best option but if you want an extremely powerful laptop that can run AI models locally fairly easily (and it's well supported) then the M3 is a great choice. I love being able to download and test out AI models on my M3.
- Der_Einzige 3y agoApple is well supported in that it's "better than everyone except Nvidia", and that's really only because a bunch of well known boutique developers seem to like the high quality of apple laptops and refuse to budge about mainly developing on them. Still, trying to do significantly fun or interesting things in any major AI ecosystem, such as the Stable Diffusion one through Automatic1111 or with the LLMs in Oobabooga (which supports nearly all LLM backends i.e. llama.cpp), will be mostly crippled and stuff will break in ways that simply don't happen to Nvidia hardware. I'll take the 2 4090s all day, because I know that quantization techniques which shouldn't even be possible (who knows maybethe fabled 1 bit quantization seems inevitable at this point) will make it possible for me to stuff even the most bloated LLM into my measly 48 GBs will be available on Nvidia first and Apple (maybe) second.
- J_Shelby_J 3y agoI bring this up all the time and get downvoted by apple defenders (usually I’m one). Apples policy of locking out 3rd party gpus might be the cause of a serious decline in Mac OS over the next decade. That or devs will start building Linux hosts for their gpus at home.
- brucethemoose2 3y agoThat limitation is totally artificial. There is nothing stopping AMD from allowing the sale of an affordable 48GB 7900 (or 32GB 7800) right now... In fact they already do in the form of the W7900, for $4500. They just don't sell those at competitive consumer prices, as they are playing the anticompetitive artificial segmentation game right alongside Nvidia, because they somehow think their tiny workstation market is more valuable than seeding AI development on their cards. Technically speaking, AMD could sell "cheap" 96GB+ GPUs by simply swapping the 7000 series memory controller die. This would take some time and money, but not an eternity + fortune like Nvidia would need to tape out a whole new GPU die. Intel has no workstation market to lose. I suspect the Arc A770 was kinda late and skimpy for a 32GB "AI" SKU, but the next generation could be very interesting.
- gruez 3y ago>as they are playing the anticompetitive artificial segmentation game right alongside Nvidia Product segmentation might be anti-consumer (in the sense that they can charge higher prices overall), but it's hardly "anticompetitive". >than seeding AI development on their cards *ceding
- badsectoracula 3y agoI understood "seeding AI development on their cards" as in "making sure AI development in the future will happen on their cards". "ceding" sounds like the opposite of that (i.e. giving up).
- brucethemoose2 3y agoThe anti competitiveness is from the pricing in the face of competition. W7900 margins are enormous. In a competitive market, AMD would massively undercut the 48GB RTX A6000 and the 24GB 4090, which Nvidia is price gouging even more. They absolutely can... But they don't. Separately, its also (IMO) a bone headed business decision.
- Der_Einzige 3y agoYou have no idea how deep the Nvidia lock-in goes. AMD only supports small fractions of the deep custom eco-systems built around models such as Stable Diffusion. Good luck getting half of the shit built for ComfyUI or Automatic1111 to run at all, and frequently those features are the killer ones which motivate folks to want to do AI in the first place. There's a reason why Nvidia is a first class citizen among all of the AI clouds. AMD is even behind Intel in this regard, because at least Intel had an existing robust ecosystem of accelerators (i.e. intel optimized sklearn/pytorch, OpenVino, etc) AMD can be 4x better in every way compared to Nvidia hardware wise but if they can't get developer communities to build around their equivalent to CUDA and its ecosystem than it's not looking rosy for AMDs future.
- dleeftink 3y agoWouldn't we rather trend model memory requirements down? This way models/devices meet in the middle.
- brucethemoose2 3y agoThe thing about ML is that memory helps everything. Take almost any given model that fits in X memory pool for some task. Double that, and everything gets better. Training quality goes up, inference quality potentially goes up, you gain the capacity for more batching which dramatically increases performance. Maybe more caching, depending on the app. You can add more models to the pipeline without much fuss.
- eachro 3y agowhy is this? I would have thought that more memory would help up to some point (wherever the entire model can fit in RAM) and then no gains after that. I'd also have thought that things like accuracy wouldnt be affected.
- brucethemoose2 3y agoEvery model and ecosystem has different compromises, and obviously small models hit diminishing returns quick. But the big models that are in fashion now... GPUs are basically always memory capacity limited to some extent.
- daemonologist 3y agoAgreed, and VRAM (GDDR6 at least) is _not_ that expensive. The going rate for an 8 Gbit chip is around $3 [0][1], so around $150 in raw chips for 48 GB. Unfortunately I suspect VRAM capacities in consumer cards will continue to be limited in order to differentiate enterprise products. [0] https://www.dramexchange.com/ https://www.dramexchange.com/ [1] https://www.tomshardware.com/news/gddr6-vram-prices-plummet https://www.tomshardware.com/news/gddr6-vram-prices-plummet
- deleted 3y ago[deleted]
- gruez 3y ago> The going rate for an 8 Gbit chip is around $3 [0][1], so around $150 in raw chips for 48 GB. I doubt 48GB cards are using 8 Gbit chips. How are you going to physically route 48 chips on a PCB?
- schmorptron 3y agoWouldn't you also need other higher-end components to interface with and support higher bandwidth to effectively make use of all of that VRAM?
- to11mtm 3y agoYes. Realistically you'd just go with higher density RAM chips, which I'm guessing are gonna be more costly per GB as well... But the upshot for a manufacturer for higher density memory, if the 'failure rate' of fresh parts is the same, well it's better to have less parts of a fixed failure rate ofc.
- ksec 3y agoIt wasn't that long ago the price for GDDR6 was up to $16, your 48GB would have cost $800. Or roughly $2000 more in retail price. What a lot of people dont understand is that RSP dont change with commodities ups and downs. If you price your product $1500 cheaper now due to DRAM cost reduciton, you will have a hard time getting it back up $1500 if things ever resume to normal. So generally speaking RSP comes down very slowly.
- staticman2 3y agoThe Apple M3 chips are expensive, not so great at gaming, and have significantly slower memory bandwidth than Nvidia chips. (A 3080 has 912 GB/s vs M3's Max being 400 GB/s with most M3 models being much slower). There are significant compromises in Apple's offerings.
- jadbox 3y agoThanks, I should have mentioned this. Lots of tradeoffs here and Apple is far from ideal. I really want a desktop equivalent of an A100 with 80gb, although $17k is just insane*. https://www.cdw.com/product/pny-nvidia-a100-80gb-pcie-gen-4-graphic-card/7065275 https://www.cdw.com/product/pny-nvidia-a100-80gb-pcie-gen-4-...
- iknowstuff 3y agoI guess you could get an M2 Ultra with 800GB/s if you care about this metric specifically?
- timschmidt 3y agoEven so, bandwidth to the CPUs, GPUs, and NPUs are segmented such that none have access to the full 800GB/s. The CPU gets more like 400GB/s in that SKU.
- iknowstuff 3y agoCorrect me if I'm wrong but 400GB/s is ludicrous for CPU memory isn't it? > DDR5 octuples the maximum DIMM capacity from 64 GB to 512 GB.[8][3] DDR5 also has higher frequencies than DDR4, up to 8000 MT/s which translates into 64 GB/s (8000 MT/s * 64-bit width / 8 bits/byte = 64 GB/s) of bandwidth per DIMM.
- timschmidt 3y agoDepends. DDR5 provides less bandwidth than Apple's on-package DRAM precisely because it allows larger sizes and user upgradeability. Intel has produced similarly packaged CPUs since https://en.wikipedia.org/wiki/Xeon_Phi https://en.wikipedia.org/wiki/Xeon_Phi in 2010, at least. A small amount of very fast memory is great so long as your workload fits within it. The moment it doesn't, everything goes awry. The best memory architecture is always the one which works for your workload. AMD produced a desktop part - the 7800X3D - which provides 2.5TB/s of bandwidth to 96Mb of L3. And an Epyc part with 768mb of L3 in a similar configuration across 8 chiplets. Then there are issues of prefetching algorithms, cache eviction and concurrency strategies as well as speed and associativity, TLB size, branch prediction algorithm especially as it relates to pipeline length and flushes, decoder width and complexity and execution units available for dispatch. There's just a lot more that goes into fast than memory bandwidth go brrrrr. Also x86 (and increasingly ARM and Risc-V) have many SKUs above those Apple offers in sleek packaging. Particularly silicon fabbed for high end desktop workstation or datacenter duty is a cut above what's available via other channels. For the cost of a Mac Pro you can easily afford an HP Z series or a whitebox from SuperMicro with dual sockets, 16 channels of DRAM, and enough PCIe lanes to make starving children cry.
- deleted 3y ago[deleted]
- ore0s 3y agoJust curious, why is rewind.ai falling short of this use case? I thought they looked at every file/photo
- pants2 3y agoApple's memory configurations for the M3 are confusing and expensive. Charging $400 for an additional 18GB of RAM is ridiculous. If they halved their memory upgrade prices maybe I would consider it, but as-is it's very hard to justify getting memory upgrades when you could just run LLMs on a dedicated Linux box.