6 ms·
Llama.cpp AI Performance with the GeForce RTX 5090 Review
- wewewedxfgdf 2y agoIt would be very interesting to see these alongside benchmarks for Apple M4, AMD Halo Strix and other AMD cards.
- teaearlgraycold 2y agoAnd workstation and server-class cards.
- daghamm 2y agoComparison for previous generations https://www.hardware-corner.net/guides/gpu-benchmark-large-language-models/ https://www.hardware-corner.net/guides/gpu-benchmark-large-l...
- mirekrusin 2y agoWhen running on apple silicon you want to use mlx, not llama.cpp as this benchmark does. Performance is much better than what's plotted there and seems to be getting better, right? Power consumption is almost 10x smaller for apple. Vram is more than 10x larger. Price wise for running same size models apple is cheaper. Upper limit (larger models, longer context) is far larger for apple (for nvidia you can easily put 2x cards, more than that it becomes whole complex setup no ordinary person can do). Am I missing something or apple is simply currently better for local llms?
- nicman23 2y agothere is a plateau where you simply need more compute and the m4 cores are not enough, so even if they have enough ram for the model the token/s is not useful
- mirekrusin 2y agoFor all models fitting 2x 5090 (2x 32GB) that's not a problem, so you can say if you have this problem then RTX is also not an option. On apple silicons you can always use MoE models, which work beautifully. On RTX it's kind of waste to be honest to run MoE, you'd be better off running single, whole active model that fills available memory (with enough space for the context).
- sgt 2y agoI'm trying to find out about that as well as I'm considering a local LLM for some heavy prototyping. I don't mind which HW I buy, but it's on a relative budget and energy efficiency is also not a bad thing. Seems the Ultra can do 40 tokens/sec on DeepSeek and nothing even comes close at that price point.
- hnfong 2y agoThe DeepSeek R1 distilled onto Llama and Qwen base models are also unfortunately called “DeepSeek” by some. Are you sure you’re looking at the right thing? The OG DeepSeek models are hundreds of GB quantized, nobody is using RTX GPUs to run them anyway…
- hnfong 2y agoUnless something changed recently, I’m not aware of big perf differences between MLX and llama.cpp on Apple hardware.
- loufe 2y agoI'm under the same impression. Llama.cpp's readme used to start with "Apple Silicon as a First Class Citizen", and IIRC Georgi works on Mac himself.
- Tostino 2y agoYou are missing something. This is a single stream of inference. You can load up the Nvidia card with at least 16 inference streams and get at much higher throughout tokens/sec. This just is just a single user chat experience benchmark.
- mhast 2y agoI have a M4 Max 128 GB MacBook Pro. It's about the same speed as my 3090 when running in LM Studio.
- Tepix 2y agoIt used to be that for locally running GenAI, VRAM per dollar was king, so used NVidia RTX 3090 cards were the undisputed darlings of DYI LLM with 24GB for 600€-800€ or so. Sticking two of these in one PC isn't too difficult despite them using 350W each. Then Apple introduced Macs with 128 GB and more unified memory at 800GB/s and the ability to load models as large as 70GB (70b FP8) or even larger ones. The M1 Ultra was unable to take full advantage of the excellent RAM speed, but with the M2 and the M3, performance is improving. Just be prepared to spend 5000€ or more for a M3 Ultra. Another alternative would be a EPYC 9005 system with 12x DDR5-6000 RAM for 576GB/s of memory bandwidth with the LLM (preferably MoE) running on the CPU instead of a GPU. However today, with the latest, surprisingly good reasoning models like QwQ-32B using up thousands or tens of thousands of tokens in their replies, performance is getting more important than previously and these systems (Macs and even RTX 3090s) might fall out of favor, because waiting for a finished reply will take several minutes or even tens of minutes. Nvidia Ampere and Apple silicon (AFAIK) are also missing FP4 support in hardware, which doesn't help. For the same reason AMD Halo Strix with a mere 273GB/s of RAM bandwidth and perhaps also NVidia Project Digits (also speculated to offer similar RAM bandwidth) might just be too slow for reasoning models with more than 50GB or so of active parameters. On the other hand, if the prices for the RTX 5090 remain at 3500€, they will likely remain insignificant for the DIY crowd for that reason alone. Perhaps AMD will take the crown with a variant of their RDNA4 RX 9070 card with 32GB of VRAM priced at around 1000€? Probably wishful thinking…
- BoredPositron 2y agoThe 3090 was always a fallacy without native fp8.
- Tepix 2y agoPerformance is good enough for non-reasoning models even if they're FP8 or FP4. Check the phoronix article, the difference between the 3090 and 4090 is rather small. There's weight-only FP8 in vLLM on NVidia Ampere: https://docs.vllm.ai/en/latest/features/quantization/fp8.html https://docs.vllm.ai/en/latest/features/quantization/fp8.htm...
- 2y ago
- benob 2y agoThe benchmark only touches 8B-class models at 8-bit quantification. Would be interesting to see how it fares with models that use more of the card ram, and under varying quantization and context lengths.
- Maxious 2y agoThere's been some INT4/NVFP4 gains too https://hanlab.mit.edu/blog/svdquant-nvfp4 https://hanlab.mit.edu/blog/svdquant-nvfp4 https://blackforestlabs.ai/flux-nvidia-blackwell/ https://blackforestlabs.ai/flux-nvidia-blackwell/
- threeducks 2y agoI agree. This benchmark should have compared the largest ~4 bit quantized model that fits into VRAM, which would be somewhere around 32B for RTX 3090/4090/5090. For text generation, which is the most important metric, the tokens per second will scale almost linearly with memory bandwidth (936 GB/s, 1008 GB/s and 1792 GB/s respectively), but we might see more interesting results when comparing prompt processing, speculative decoding with various models, vLLM vs llama.cpp vs TGI, prompt length, context length, text type/programming language (actually makes a difference with speculative decoding), cache quantization and sampling methods. Results should also be checked for correctness (perplexity or some benchmark like HumanEval etc.) to make sure that results are not garbage. If anyone from Phoronix is reading this, this post might be a good point to get you started: https://old.reddit.com/r/LocalLLaMA/comments/1h5uq43/llamacpp_bug_fixed_speculative_decoding_is_30/m08poqr/ https://old.reddit.com/r/LocalLLaMA/comments/1h5uq43/llamacp... At time of writing, Qwen2.5-Coder-32B-Instruct-GGUF with one of the smaller variants for speculative decoding is probably the best local model for most programming tasks, but keep an eye out for any new models. They will probably show up in Bartowksi's "Recommended large models" list, which is also a good place to download quantized models: https://huggingface.co/bartowski https://huggingface.co/bartowski
- regularfry 2y agoUsing aider with local models is a very interesting stress case to add on top of this. Because the support for reasoning models is a bit rough, and they aren't always great at sticking to the edit format, what you end up doing is configuring different models for different tasks (what aider calls "architect mode"). I use ollama for this, and I'm getting useful stuff out of qwq:32b as the architect, qwen2.5-coder:32b as the edit model, and dolphin3:8b as the weak model (which gets used for things like commit messages). Now what that means is that performance swapping these models in and out of the card starts to matter, because they don't all go into VRAM at once; but also using a reasoning model means that you need straight-line tokens per second as well, plus well-tuned context length so as not to starve the architect. I haven't investigated whether a speculative decoding setup would actually help here, I've not come across anyone doing that with a reasoner before now but presumably it would work. It would be good to see a benchmark based on practical aider workflows. I'm not aware of one but it should be a good all-round stress test of a lot of different performance boundaries.
- 3np 2y agoGreat to see Mr Larabel@Phoronix both maintaining consistent legit reporting and still have time for one-offs like this in these times of AI slop and other OG writers either quitting or succumbing to the vortex. Hats off!
- 3np 2y agoCorrect me if I'm wrong, but I have the impression that we'd usually expect to see bigger efficiency gains while these are marginal? If so that would confirm the notion that they've hit a ceiling and pushing against physical limitations.
- magicalhippo 2y ago> I have the impression that we'd usually expect to see bigger efficiency gains while these are marginal? The 50-series is made using the same manufacturing process ("node") as the 40-series, and there is not a major difference in design. So the 50-series is more like tweaking an engine that previously topped out at 5000 RPM so it's now topping out at 6000 RPM, without changing anything fundamental. Yes it's making more horsepower but it's using more fuel to do so.
- 3np 2y agoMakes sense, thanks!
- littlestymaar 2y agoTL;DR; performance isn't bad, but perf per Watt isn't better than 4080 or 4090 and can even be significantly lower than 4090 in certain contexts.
- regularfry 2y agoI think that's underselling it. Performance is good, up by a significant margin, and the VRAM boost is well worth it. There's just no efficiency gain to go along with it. It looks to me like you could think about it as a performance/VRAM/convenience stepping-stone between having one 4090 and having a pair. Paired 5090s, if such a thing is possible, sounds like a very good way to spend a lot of money very quickly while possibly setting things on fire, and you'd have to have a good reason for that.
- lostmsu 2y agoThey didn't try power limiting the cards.
- braycarla 2y ago[dead]
- chakintosh 2y agoCurious if the author checked whether his card doesn't have any missing ROPs.
- Guthur 2y agoWhy would that be relevant? this isn't rasterization.
- kimixa 2y agoThey might be, depending on architecture - the ROPs are responsible for taking the output from the shader core and writing it back to memory, so can be used in compute shaders even if all the fancier "Raster Operation" modes aren't really used there. No point having a second write pipeline to memory when there's already one there. But if your usecase doesn't really pressure that side of things then even if some are "missing" it might make zero difference, and my understanding of most ML models is they're heavily read bandwidth biased.
- alecco 2y agoIs llama.cpp's CUDA implementation decent? (e.g. does it use CUTLASS properly or something more low level)
- regularfry 2y agoThe implementation's here: https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/ggml-cuda https://github.com/ggml-org/llama.cpp/tree/master/ggml/src/g...
- lr3783285 2y ago[dead]
- cjtrowbridge 2y agoP40s are so slept on. 24gb vram for $150.
- baq 2y agowhen you see a deal which is too good to be true, it's probably too good to be true!
- haswell 2y agoWhat are the factors that make the P40 too good to be true?
- washadjeffmad 2y agoConvenience, support, and performance. Ampere is the floor, in practice, which is what effectively makes the "buy a Mac Studio" crowd P40 people with 10x the budget.
- datadrivenangel 2y agoIt's not too good to be true, it just takes a lot of power to run old GPUs to get slow token speeds!
- auspiv 2y agoP40s are no longer $150. On eBay, they're at least $250 from China, or $400 for US sellers
- whywhywhywhy 2y agoI think if they had more than 24gb there would be more interest but the hacks and noise with trying to make server hardware work within a desktop case and the FP16 performance is extremely poor compared to a 3090 (183.7 GFLOPS vs 35.58 TFLOPS IIRC).
- KronisLV 2y agoSometimes I daydream about a world where GPUs just have the equivalent of LPCAMM and you could put in as much RAM as you can afford and as much as the hardware supports, much like is the case with motherboards, even if something along the way would bottleneck somewhat. It'd really extend the life of some hardware, yet companies don't want that. That said, it's cool that you can even get an L4 with 24 GB of VRAM that actually performs okay, yet is passively cooled and consumes like 70W, at that point you can throw a bunch of them into a chassis and if you haven't bankrupted yourself by then, they're pretty good. I did try them out on Scaleway, the pricing isn't even that exorbitant, using consumer GPUs for LLM use cases doesn't quite hit the same since.
- perching_aix 2y agoSaw a video from Bolt Graphics, a startup trying to do that among other things. Supposedly they'll have demos at several conventions later this year, like at Hot Chips. I think it's an incredibly tall order to get into the GPU game for a startup, but it should be good entertainment if nothing else.
- someothherguyy 2y agohttps://www.youtube.com/watch?v=8m-gSSIheno https://www.youtube.com/watch?v=8m-gSSIheno Sounds like hype
- someothherguyy 2y ago> yet companies don't want that I think there are quite a few constraints you are glossing over here.
- KronisLV 2y agoOh, I definitely am! It's always cool when someone with domain specific knowledge drops by and proceeds to shatter that dream with technical reasons that are cool nonetheless, the same way how LPCAMM2 doesn't work with every platform either and how VRAM has pretty stringent requirements. That said, it's understandable that companies are sticking with whatever works for now, but occasionally you get an immensely cool project that attempts to do something differently, like Intel's Larrabee did, for example.
- jacekm 2y agoIf my budget is only ~$1000 should I buy a used 3090 or a new 5080? (for AI, I don't care about gaming)
- washadjeffmad 2y ago3090. Models like flux or wanvideo with fp16 text encoders will push your VRAM usage over 20GB. LLMs with functional context (or workflows with multi-model, not modal, tie in) will all break 16GB. There's no point to performance without context. At its pricepoint, the 5080 is purely a gaming card.
- passwordqwe 2y agoDo you think you can get a 5080 for a 1000? In the Netherland they're going for 1600 euro's minimum, more like 1800. So I would totally get a new 5080 for a 1000 dollars.
- jacekm 2y agoSigh, I knew they were more than $1000 but did not anticipate that much. You are correct, the price seems to go up every time I check.
- zamadatix 2y ago3090, the 24 GB of VRAM will be more relevant than running smaller models even faster on lower capacity card (small models already run fast by nature of being less to copy from memory and compute on).
- DrNosferatu 2y agoAnd the latest AMD cards for reference? Also, some DeepSeek models would be cool.
- magicalhippo 2y agoComing I hope, though I wouldn't have huge expectations. He already did the general compute benchmark of the two 9070 cards here[1], and between poorly optimized drivers and GDDR6's lower memory bandwidth, I wouldn't expect any great scores. In terms of memory bandwidth they're between a 4070 and a 4070 Ti SUPER, and given that LLMs are very memory-bandwidth constrained as I mentioned in another comment, at best you'd expect the LLM score to end up between the 4070 and the 4070 Ti SUPER. [1]: https://www.phoronix.com/review/amd-radeon-rx9070-linux-compute https://www.phoronix.com/review/amd-radeon-rx9070-linux-comp...
- Joel_Mckay 2y agoThe 9070 cards are hard to find, and inflated 30% above MSRP right now. Apparently the supply of fuse-crippled 9070 cards was just used to up-sell 9070 XT (easier to find.) Have a look at the GamersNexus YT rant about it... They make a fair argument in that price range an older model used RTX nvidia card may be a better value. Depends on your use-case, as rtx 5090 nvidia AI frame interpolation is dog crap hype for CGI or CUDA accelerated ML libraries. Personally, I would go with the rtx 4090 or even an rtx 3090 with 24G vram for ML and CGI workstation, as CUDA+Optix has better software support. For just gaming, the 9070 XT is a better deal when the MSRP is within range. Depends how willing you are to get ripped off by scalper prices right now. lol =3
- deleted 2y ago[deleted]
- benw214 2y ago96gb beast coming from nvidia. https://videocardz.com/newz/nvidia-rtx-pro-6000-blackwell-leaked-24064-cores-96gb-g7-memory-and-600w-double-flow-through-cooler https://videocardz.com/newz/nvidia-rtx-pro-6000-blackwell-le...
- braycarla 2y ago[dead]