5 ms·
For people wanting to do LLM work at home, a fully loaded MacBook has more memory available to it than anything else a consumer can buy. All in a laptop form fa
by com2kid 3y ago
For people wanting to do LLM work at home, a fully loaded MacBook has more memory available to it than anything else a consumer can buy. All in a laptop form factor (versus Nvidia GPUs).
A small market, sure, but one they have on lock down.
- LoganDark 3y agoNot to mention that this memory is also VRAM. You can have over 100GB of it, in an age where just 80GB of VRAM in a single card costs over $13,000.
- ric2b 3y agoYou can do the same with any integrated GPU on an Intel or AMD processor. Actually you can go well beyond that amount of memory.
- wtallis 3y agoThe integrated GPUs Intel and AMD are shipping don't really have enough performance to be worth the trouble, and the configs that support 100+ GB of RAM and have integrated GPUs don't have much memory bandwidth (running at speeds like 6400MT/s is a lot harder with multiple ranks of memory).
- remexre 3y agoGoogling suggests Intel's A770M has 16.79 TFLOPS, M3 Max's GPU has 16.4 TFLOPS?
- wtallis 3y agoIntel's Arc A770M is a discrete GPU with only 16GB of DRAM. It is not to be confused with the "UHD Graphics 770" found in their top desktop processors. The discrete graphics is on the order of 16x larger and faster than the desktop integrated graphics, and the latter is the only one that can plausibly be said to have access to 100+ GB of RAM.
- deleted 3y ago[deleted]
- LoganDark 3y agoA770M can come with only 12GB, too. And it's GDDR6, not "just" DRAM (although it technically still counts).
- aurareturn 3y agoTflops isn't a reliable way to measure GPU performance anymore. For example. Nvidia Tflop numbers for their new GPUs are 4-5x the performance of 2080Ti but in actual performance, it's like 2x only.
- LoganDark 3y agoNormal DRAM (i.e. not GDDR) is far too slow to facilitate this, you're basically limited by the CPU's cache if you want to reach a level of performance even remotely comparable to a GPU that has GDDR or HBM.
- wtallis 3y agoGDDR isn't as special as you think, and it's special in exactly the opposite way that HBM is special. HBM achieves high bandwidth by using extremely large bus widths running at fairly slow speeds per pin. GDDR achieves high bandwidth by running at very high speeds per pin and using medium to large bus widths. Desktop CPUs using DDR5 have low memory bandwidth because they have narrow bus widths (128bit total) at low speeds. Server CPUs use the same memory at the same per-pin speeds but with much larger bus widths to achieve total bandwidth comparable to a mid-range GPU but supporting orders of magnitude more memory capacity. Apple's high-end chips likewise uses significantly larger bus widths than desktop CPUs, combined with slightly per-pin speeds (LPDDR rather than DDR) to also reach GPU-like total bandwidth. I don't think it's accurate to single out any one of the above technologies as being the "normal" DRAM. GDDR, LPDDR, and DDR are all fairly different from each other and all quite mainstream.
- LoganDark 3y agoGDDR is "special" because it's far closer to the GPU die. System memory isn't that close to the CPU, so any integrated GPU won't be able to make as good use of it as a dedicated GPU with co-located VRAM.
- wtallis 3y ago> GDDR is "special" because it's far closer to the GPU die. LPDDR is always at least as close to the processor, and is often in-package so significantly closer. HBM is always in-package. But all of that is a red herring, because the distance is only loosely correlated with performance and is a completely irrelevant metric if you've already accounted for bus width and clock rate.
- flir 3y agoYou can buy a lot of cloud compute instead of that extra memory though. (I haven't done the math myself, but I'd be curious if anyone has).
- chii 3y agoeven a 5k spend on cloud compute is not going out outlast the expected 3-5 yrs of use that a mac M3 would have.
- Our_Benefactors 3y agoWhat? This is a bizarre apples to oranges comparison. That 5k of compute can do things that aren’t possible on a Mac, like training a model.
- uf00lme 3y agoNot many people use compute 24/7, so if your use cases can be ephemeral than spot/savings plans can be a good saving and using the cloud allows for opex rather than capex controls and taxes
- pulse7 3y agoWait for just 2 days, 17 hours and 18 minutes - and NVIDIA will give you (rumored) 96GB of RAM inside GeForce RTX 4090 TI: https://www.nvidia.com/en-eu/geforce/special-event/ https://www.nvidia.com/en-eu/geforce/special-event/
- doctorpangloss 3y agoThere is no chance the 4090 TI is going to ship with 96GB of VRAM. And besides, it has no NVLink, so it's a stranded asset. It would appear the poster misread reports about the L2 cache (96MB) for the VRAM (24GB).
- cjbprime 3y agoThe lack of nvlink doesn't matter much for LLMs, which can work on distinct layers of the model on completely independent GPUs.
- com2kid 3y agoDoesn't fit in a backpack. Can't take it to a coffee shop. Heck it involves building an entire custom PC around it. Sure the m3 with 128GB of RAM costs $7k, which is an absurd number, but that will run high quality LLMs that can have sci-fi level intelligent conversations. Not a bad price for a laptop capable of doing things that was literally "haha that'll never happen" levels of unimaginable 5 years ago.
- TillE 3y agoI can't find even the sketchiest rumor claiming 96GB of VRAM on a 4090 Ti. That would be more than an H100, which costs around $30k.
- nomel 3y agoAgreed. Estimates put the H100 BOM at around $3k. If they do offer that much memory, they're going to implement some way to make it unattractive, to hold onto that sweet 800% margin [1]. 1. https://www.tomshardware.com/news/nvidia-makes-1000-profit-on-h100-gpus-report https://www.tomshardware.com/news/nvidia-makes-1000-profit-o...
- Affric 3y agoIt will be interesting to see how much further it will be possible for Apple to push their platform. I think the Jobs/Ive legacy that has pushed the thermals ahead of raw compute had given them a massive headstart in the LLM age for both notebooks (obviously) but also desktop (less obviously). With the size of a normal 4090 heat sink the ATX platform is showing its age. The biggest element of any system is a series of huge heat sinks. The graphics card ends up being massive. You have support struts coming out. Giant power supplies. Bespoke connectors. Paying for RAM twice. Apple have a form factor people are comfortable with and haven’t set the bar too high with raw performance. They aren’t hindered by existing designs. The thermals are good. And you pay for it all in one.