8 ms·
Looks like it punches way above its weight(s). How far are we from running a GPT-3/GPT-4 level LLM on regular consumer hardware, like a MacBook Pro?
by xeckr 2y ago
Looks like it punches way above its weight(s).
How far are we from running a GPT-3/GPT-4 level LLM on regular consumer hardware, like a MacBook Pro?
- lappa 2y agoIt's easy to argue that Llama-3.3 8B performs better than GPT-3.5. Compare their benchmarks, and try the two side-by-side. Phi-4 is yet another step towards a small, open, GPT-4 level model. I think we're getting quite close. Check the benchmarks comparing to GPT-4o on the first page of their technical report if you haven't already https://arxiv.org/pdf/2412.08905 https://arxiv.org/pdf/2412.08905
- vulcanash999 2y agoDid you mean Llama-3.1 8B? Llama 3.3 currently only has a 70B model as far as I’m aware.
- anon373839 2y agoWe’re already past that point! MacBooks can easily run models exceeding GPT-3.5, such as Llama 3.1 8B, Qwen 2.5 8B, or Gemma 2 9B. These models run at very comfortable speeds on Apple Silicon. And they are distinctly more capable and less prone to hallucination than GPT-3.5 was. Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4, and they will run on MacBook Pros with at least 64GB of RAM. However, I have an M3 Max and I can’t say that models of this size run at comfortable speeds. They’re a bit sluggish.
- noman-land 2y agoThe coolness of local LLMs is THE only reason I am sadly eyeing upgrading from M1 64GB to M4/5 128+GB.
- Terretta 2y agoCompare performance on various Macs here as it gets updated: https://github.com/ggerganov/llama.cpp/discussions/4167 https://github.com/ggerganov/llama.cpp/discussions/4167 OMM, Llama 3.3 70B runs at ~7 text generation tokens per second on Macbook Pro Max 128GB, while generating GPT-4 feeling text with more in depth responses and fewer bullets. Llama 3.3 70B also doesn't fight the system prompt, it leans in. Consider e.g. LM Studio (0.3.5 or newer) for a Metal (MLX) centered UI, include MLX in your search term when downloading models. Also, do not scrimp on the storage. At 60GB - 100GB per model, it takes a day of experimentation to use 2.5TB of storage in your model cache. And remember to exclude that path from your TimeMachine backups.
- noman-land 2y agoThank you for all the tips! I'd probably go 128GB 8TB because of masochism. Curious, what makes so many of the M4s in the red currently.
- vessenes 2y agoIt's all memory bandwidth related -- what's slow is loading these models into memory, basically. The last die from Apple with all the channels was the M2 Ultra, and I bet that's what tops those leader boards. M4 has not had a Max or an Ultra release yet; when it does (and it seems likely it will), those will be the ones to get.
- ant6n 2y agoWhat if you have a Macbook Air with 16GB (the bechmarks dont seem to show memory).
- simonw 2y agoYou could definitely run an 8B model on that, and some of those are getting very capable now. The problem is that often you can't run anything else. I've had trouble running larger models in 64GB when I've had a bunch of Firefox and VS Code tabs open at the same time.
- stkdump 2y agoI bought an old used desktop computer, a used 3090, and upgraded the power supply, all for around 900€. Didn't assemble it all yet. But it will be able to comfortably run 30B parameter models with 30-40 T/s. The M4 Max can do ~10 T/s, which is not great once you really want to rely on it for your productivity. Yes, it is not "local" as I will have to use the internet when not at home. But it will also not drain the battery very quickly when using it, which I suspect would happen to a Macbook Pro running such models. Also 70B models are out of reach of my setup, but I think they are painfully slow on Mac hardware.
- jazzyjackson 2y agoI'm returning my 96GB m2 max. It can run unquantized llama 3.3 70B but tokens per second is slow as molasses and still I couldn't find any use for it, just kept going back to perplexity when I actually needed to find an answer to something.
- Tepix 2y agoInteresting. You're using the FP8 version i'm guessing? How many tokens/s are you using and which software? MLX?
- alecco 2y agoI'm waiting for next gen hardware. All the companies are aiming for AI acceleration.
- kleiba 2y agoSorry, I'm not up to date, but can you run GPTs locally or only vanilla LLMs?
- noodletheworld 2y ago[dead]
- runako 2y ago> Is a $8000 MBP regular consumer hardware? May want to double-check your specs. 16" w/128GB & 2TB is $5,400.
- anon373839 2y ago> gpt-3.5-turbo is generally considered to be about 20B params. An 8B model does not exceed it. The industry has moved on from the old Chinchilla scaling regime, and with it the conviction that LLM capability is mainly dictated by parameter count. OpenAI didn't disclose how much pretraining they did for 3.5-Turbo, but GPT 3 was trained on 300 billion tokens of text data. In contrast, Llama 3.1 was trained on 15 trillion tokens of data. Objectively, Llama 3.1 8B and other small models have exceeded GPT-3.5-Turbo in benchmarks and human preference scores. > Is a $8000 MBP regular consumer hardware? As user `bloomingkales` notes down below, a $499 Mac Mini can run 8B parameter models. An $8,000 expenditure is not required.
- tosh 2y agoA Mac with 16GB RAM can run qwen 7b, gemma 9b and similar models that are somewhere between GPT3.5 and GPT4. Quite impressive.
- jazzyjackson 2y agoon what metric? Why would OpenAI bother serving GPT4 if customers would be just as happy with a tiny 9B model?
- tosh 2y agohttps://lmarena.ai/ https://lmarena.ai/ Check out the lmsys leaderboard. It has an overall ranking as well as ranking for specific categories. OpenAI are also serving gpt4o mini. That said afaiu it’s not known how large/small mini is. Being more useful than GPT3.5 is not a high bar anymore.
- deleted 2y ago[deleted]
- kgeist 2y ago>MacBooks can easily run models exceeding GPT-3.5, such as Llama 3.1 8B, Qwen 2.5 8B, or Gemma 2 9B. If only those models supported anything other than English
- simonw 2y agoLlama 3.1 8B advertises itself as multilingual. All of the Qwen models are basically fluent in both English and Chinese.
- kgeist 2y agoLlama 8B is multilingual on paper, but the quality is very bad compared to English. It generally understands grammar, and you can understand what it's trying to say, but the choice of words is very off most of the time, often complete gibberish. If you can imagine the output of an undertrained model, this is it. Meanwhile GPT3.5 had far better output that you could use in production.
- barrell 2y agoCohere just announced Command R7B. I haven’t tried it yet but their larger models are the best multilingual models I’ve used
- numpad0 2y agoIs subtext to this uncensored Chinese support?
- simonw 2y agoWe're there. Llama 3.3 70B is GPT-4 level and runs on my 64GB MacBook Pro: https://simonwillison.net/2024/Dec/9/llama-33-70b/ https://simonwillison.net/2024/Dec/9/llama-33-70b/ The Qwen2 models that run on my MacBook Pro are GPT-4 level too.
- BoorishBears 2y agoSaying these models are at GPT-4 level is setting anyone who doesn't place special value on the local aspect up for disappointment. Some people do place value on running locally, and I'm not against then for it, but realistically no 70B class model has the amount of general knowledge or understanding of nuance as any recent GPT-4 checkpoint. That being said these models are still very strong compared to what we had a year ago and capable of useful work
- simonw 2y agoI said GPT-4, not GPT-4o. I'm talking about a model that feels equivalent to the GPT-4 we were using in March of 2023.
- int_19h 2y agoI remember using GPT-4 when it first dropped to get a feeling of its capabilities, and no, I wouldn't say that llama-3.3-70b is comparable. At the end of the day, there's only so much you can cram into any given number of parameters, regardless of what any artificial benchmark says.
- simonw 2y agoI envy your memory.
- BoorishBears 2y agoYou're free to intentionally miss their point, does them no good.
- n144q 2y ago
- refulgentis 2y agoWe're there, Llama 3.1 8B beats Gemini Advanced for $20/month. Telosnex with llama 3.1 8b GGUF from bartowski. https://telosnex.com/compare/ https://telosnex.com/compare/ (How!? tl;dr: I assume Google is sandbagging and hasn't updated the underlying Gemini)
- deleted 2y ago[deleted]
- bloomingkales 2y agoM4 Mac mini 16gb for $500. It's literally an inferencing block (small too, fits in my palm). I feel like the whole world needs one.
- alganet 2y ago> inferencing block Did you mean _external gpu_? Choose any 12GB or more video card with GDDR6 or superior and you'll have at least double the performance of a base m4 mini. The base model is almost an older generation. Thunderbolt 4 instead of 5, slower bandwidths, slower SSDs.
- kgwgk 2y ago> you'll have at least double the performance of a base m4 mini For $500 all included?
- alganet 2y agoThe base mini is 599. Here's a config for around the same price. All brand new parts for 573. You can spend the difference improving any part you wish, or maybe get an used 3060 and go AM5 instead (Ryzen 8400F). Both paths are upgradeable. https://pcpartpicker.com/list/ftK8rM https://pcpartpicker.com/list/ftK8rM Double the LLM performance. Half the desktop performance. But you can use both at the same time. Your computer will not slow down when running inference.
- bloomingkales 2y agoThat’s a really nice build.
- alganet 2y agoAnother possible build is to use a mini-pc and M.2 connections You'll need a mini-pc with two M.2 slots, like this: https://www.amazon.com/Beelink-SER7-7840HS-Computer-Display/dp/B0CQT9N951 https://www.amazon.com/Beelink-SER7-7840HS-Computer-Display/... And a riser like this: https://www.amazon.com/CERRXIAN-Graphics-Left-PCI-Express-Extension/dp/B0D9651WSN https://www.amazon.com/CERRXIAN-Graphics-Left-PCI-Express-Ex... And some courage to open it and rig the stuff in. Then you can plug a GPU on it. It should have decent load times. Better than an eGPU, worse than the AM4 desktop build, fast enough to beat the M4 (once the data is in the GPU, it doesn't matter). It makes for a very portable setup. I haven't built it, but I think it's a reasonable LLM choice comparable to the M4 in speed and portability while still being upgradable. Edit: and you'll need an external power supply of at least 400W:)
- ActorNightly 2y agoWhy would you want to though? You already can get free access to large LLMs and nobody is doing anything groundbreaking with them.
- jckahn 2y agoI only use local, open source LLMs because I don’t trust cloud-based LLM hosts with my data. I also don’t want to build a dependence on proprietary technology.