6 ms·
What Apple has is theoretically great on paper, but it fails to live up to expectations. Whats the point of having the RAM for running an LLM locally when the p
by chaostheory 3y ago
What Apple has is theoretically great on paper, but it fails to live up to expectations. Whats the point of having the RAM for running an LLM locally when the performance is abysmal compared to running it on even a consumer Nvidia GPU. It’s a missed opportunity that I hope either the M4 or M5 addresses
- bearjaws 3y agoIt's a 25w processor. How will it ever live up to a 400w GPU? Also you can't even run large models on a single 4090, but you can on M series laptops with enough RAM. The fact a laptop can run 70B+ parameter models is a miracle, it's not what the chip was built to do at all.
- chaostheory 3y agoThe problem is that it extends to both Mac Studio and Mac Pro.
- wongarsu 3y agoIt's a valid comparison in the very limited sense of "I have $2000 to spend on a way to run LLMs, should I get an RTX4090 for the computer I have or should I get a 24GB MacBook", or "I have $5000, should I get an RTX A6000 48GB or a 96GB MacBook". Those comparisons are unreasonable in a sense, but they are implied by statements like GPs "Hence a lot of people in the local LLMA community are really going after high-memory Macs".
- fckgw 3y agoNo, it is not a valid comparison to make between an entire laptop and a single PC part. "The computer I have" is doing a ton of heavy lifting here.
- wongarsu 3y ago"Should I upgrade what I have or buy something new" is a completely normal everyday decision. Of course it doesn't apply to everyone since it presumes you have something compatible to upgrade, but it is a real decision lots of people are making
- fckgw 3y agoBut it's also assuming everyone has a desktop PC capable of this stuff that can be upgraded.
- michaelt 3y agoSorta yes, sorta no. You're certainly right that with a macbook you get a whole computer, so you're getting more for your money. And it's a luxury high-end computer too! But personally, I've never seen anyone step directly from not-even-having-a-PC to buying a 4090 for $1800. Folks that aren't technically inclined by and large stick with hosted models like ChatGPT. More common in my experience is for technical folks with, say, an 8GB GPU to experiment with local ML, decide they're interested in it, then step up to a 4090 or something.
- oceanplexian 3y agoThe answer depends on what you plan to do with it. Do you need to do fine tuning on a smaller model and need the highest inference performance with smaller models? Are you planning to use it as a lab to learn how to work with tools that are used in Big Tech (i.e. CUDA)? Or do you just want to do slow inference on super huge models (e.g Grok)? Personally, I chose the Nvidia route because as a backend engineer, Macs aren’t seriously used in datacenters. The same frameworks I use to develop on a 3090 are transferable to massive, infiniband-connected clusters with TB of VRAM.
- 0x457 3y agoI think there are two communities: - the "hobbyists" with $5k GPUs - People that work in the industry that never used "not mac" or even if they did - explaining to IT that you need a PC with RTX A6000 48GB instead of a mac like literally everyone else in the company is a loosing battle.
- wongarsu 3y agoThere is also an important third group: - people that work outside Silicon Valley, where the entire company uses Windows centrally managed through Active Directory, and explaining IT that you need an Mac is an uphill battle. So you just submit your request for an RTX A6000 48GB to be added to your existing workstation Those people are the intended target customer of the A6000, and there are a lot of them.
- 0x457 3y agoWhile there are many people "that work outside Silicon Valley, where the entire company uses Windows centrally managed through Active Directory, and explaining IT that you need a Mac is an uphill battle" I think these companies either don't care about AI or get into it by acquiring a startup (that runs on macs). Companies that run windows and AD are too busy to make sure you move your mouse every 5 minutes while you're on the clock more. At least that is my experience.
- MichaelZuo 3y agoThere are some genuine, non-Dilbert-esque uses for AD. Where there really isn't a viable, similarly performant, alternative.
- dragonwriter 3y ago> The fact a laptop can run 70B+ parameter models is a miracle Most laptops with 64+GB of RAM can run a 70B model at 4-bit quantization. It’s not a miracle, it’s just math. M2 can do it faster than systems with slower memory bandwidth.
- deleted 3y ago[deleted]
- InTheArena 3y agoThe performance of oolama on my M1 MAX is pretty solid - and does things that my 2070 GPU can't do because of memory.
- dangus 3y agoNot that I don’t believe you but the 2070 is two generations and 5 years old. Maybe a comparison to a 4000 series would be more appropriate?
- Kirby64 3y agoThe M1 Max is also 2 generations old, and ~3 years old at this point. Seems like a fair comparison to me.
- dangus 3y agoThe 4000 series still has a bigger gap in how much of a generational leap that product was. The M3 Max has something like 33% faster overall graphics performance than the M1 Max (average benchmark) while the 4090 is something like 138% faster than the 2080Ti. Depending on which 2070 and 4070 models you compare the difference is similar, close to or exceeding 100% uplift.
- whizzter 3y agoGoogling power draw the 4090 goes up to 450w whilst the 2080ti was at 250w, adjusting for power consumption the increase is somewhere around 32%. Some architectural gains and probably optimizations in chipset workings but we're not seeing as many amazing generational leaps anymore regardless of manufacturer/designer.
- dangus 3y agoI’m still seeing over a 100% uplift comparing mobile to mobile on Nvidia products: https://gpu.userbenchmark.com/Compare/Nvidia-RTX-4090-Laptop-vs-Nvidia-RTX-2080-Mobile/m2036852vsm700275 https://gpu.userbenchmark.com/Compare/Nvidia-RTX-4090-Laptop... As far as desktop products, power consumption is irrelevant.
- zitterbewegung 3y agoBuying a m3 max with 128gb of RAM while will underperform any consumer NVIDIA GPU it will be able to load larger models in practice but slowly. I think a way for the m series chips to aggressively target GPU inference or training would need a strategy that increases the speed of the RAM to start to match GDDR6 or HBM3 or use it directly.
- deleted 3y ago[deleted]
- chaostheory 3y agoYou summed up my point better than I did
- evilduck 3y agoThat completely depends on your expectations and uses. I have a gaming rig with a 4080 with 16GB of RAM and it can't even run Mixtral (kind of the minimum bar of a useful generic LLM in my opinion) without being heavily quantized. Yeah it's fast when something fits on it, but I don't see much point in very fast generation of bad output. A refurbished M1 Max with 32GB of RAM will enable you to generate better quality LLM output than even a 4090 with 24GB of VRAM and for ~$300 less, and it's a whole computer instead of a single part that still needs a computer around it. Compared to my 4080, that GPU and the surrounding computer get you half the VRAM capacity for greater cost than the Mac. If you're building a rig with multiple GPUs to support many users or for internal private services and are willing to drop more than $3k then I think the equation swings back in favor of Nvidia, but not until then.
- soupbowl 3y agoJust buy another 16gb of ram for 80$....
- elzbardico 3y agoGPU ram?
- evilduck 3y agoRunning your larger-than-your-GPU-VRAM LLM model on regular DDR ram will completely slaughter your token/s speed to the point that the Mac comes out ahead again.
- programd 3y agoDepends on what you're doing. Just chatting with the AI? I'm getting about 7 tokens per sec for Mistral with the Q6_K on a bog standard Intel i5-11400 desktop with 32G of memory and no discrete GPU (the CPU has Intel UHD Graphics 730 built in). 2 year old low end CPU that goes for, what $150? these days. As far as I'm concerned that's conversational speed. Pop in some 8 core modern CPU and I'm betting you can double that, without even involving any GPU. People way overestimate what they need in order to play around with models these days. Use llama.cpp and buy that extra $80 worth of RAM and pay about half the price of a comparable Mac all in. Bigger models? Buy more RAM, which is very cheap these days. There's a $487 special on Newegg today with an i7-12700KF, motherboard and 32G of ram. Add another $300 worth of case, power supply, SSD and more RAM and you're under the price of a Macbook Air. There's your LLM inference machine (not for training obviously) which can run even the 70B models at home at acceptable conversational speed.
- deleted 3y ago[deleted]
- instagib 3y agoOne thing I would consider is usage throttling on a MacBook Pro. Would repeated LLM usage run into throttling? No idea what specifically everyone is pulling their performance data from or what task(s). Here is a video to help visualize the differences with a maxed out m3 max vs 16gbm1 pro vs 4090 on llm 7B/13b/70b llama 2. https://youtu.be/jaM02mb6JFM https://youtu.be/jaM02mb6JFM Here’s a Reddit comparison of 4090 vs M2 Ultra 96gb with tokens/s https://old.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_40906000_vs_m2_max_with_96gb_unified_memory/ https://old.reddit.com/r/LocalLLaMA/comments/14319ra/rtx_409... M3 pro memory BW 150 gb/s M3 max 10/30 300 gb/s M3 max 12/40 400 gb/s “Llama models are mostly limited by memory bandwidth. rtx 3090 has 935.8 gb/s rtx 4090 has 1008 gb/s m2 ultra has 800 gb/s m2 max has 400 gb/s so 4090 is 10% faster for llama inference than 3090 and more than 2x faster than apple m2 max https://github.com/turboderp/exllama https://github.com/turboderp/exllama using exllama you can get 160 tokens/s in 7b model and 97 tokens/s in 13b model while m2 max has only 40 tokens/s in 7b model and 24 tokens/s in 13b apple 40/s Memory bandwidth cap is also the reason why llamas work so well on cpu (…) buying second gpu will increase memory capacity to 48gb but has no effect on bandwidth so 2x 4090 will have 48gb vram and 1008 gb/s bandwidth and 50% utilization”
- john_alan 3y agowhat are you talking about, it's literally the fastest single core retail CPU globally, and multicore is close too - https://browser.geekbench.com/processor-benchmarks https://browser.geekbench.com/processor-benchmarks
- edward28 3y agoI would take those benchmarks with a grain of salt, given that it shows a 96 core epyc loosing in multi thread to a 64 core epyc and a 32 core xeon.
- jwr 3y agoHmm. I'm running decent LLMs locally (deepseek-coder:33b-instruct-q8_0, mistral:7b-instruct-v0.2-q8_0, mixtral:8x7b-instruct-v0.1-q4_0) on my MacBook Pro and they respond pretty quickly. At least for interactive use they are fine and comparable to Anthropic Opus in speed. That MacBook has an M3 Max and 64GB RAM. I'd say it does live up to my expectations, perhaps even slightly exceeds them.