3 ms·
We're there. Llama 3.3 70B is GPT-4 level and runs on my 64GB MacBook Pro: https://simonwillison.net/2024/Dec/9/llama-33-70b/ https://simonwillison.net/2024/Dec
by simonw 2y ago
We're there. Llama 3.3 70B is GPT-4 level and runs on my 64GB MacBook Pro: https://simonwillison.net/2024/Dec/9/llama-33-70b/ https://simonwillison.net/2024/Dec/9/llama-33-70b/
The Qwen2 models that run on my MacBook Pro are GPT-4 level too.
- BoorishBears 2y agoSaying these models are at GPT-4 level is setting anyone who doesn't place special value on the local aspect up for disappointment. Some people do place value on running locally, and I'm not against then for it, but realistically no 70B class model has the amount of general knowledge or understanding of nuance as any recent GPT-4 checkpoint. That being said these models are still very strong compared to what we had a year ago and capable of useful work
- simonw 2y agoI said GPT-4, not GPT-4o. I'm talking about a model that feels equivalent to the GPT-4 we were using in March of 2023.
- int_19h 2y agoI remember using GPT-4 when it first dropped to get a feeling of its capabilities, and no, I wouldn't say that llama-3.3-70b is comparable. At the end of the day, there's only so much you can cram into any given number of parameters, regardless of what any artificial benchmark says.
- simonw 2y agoI envy your memory.
- BoorishBears 2y agoYou're free to intentionally miss their point, does them no good.
- n144q 2y agoI wouldn't call 64GB MacBook Pro "regular consumer hardware".
- jsheard 2y agoYeah, a computer which starts at $3900 is really stretching that classification. Plus if you're that serious about local LLMs then you'd probably want the even bigger RAM option, which adds another $800...
- evilduck 2y agoAn optioned up minivan is also expensive but doesn’t cost as much as a firetruck. It’s expensive but still very much consumer hardware. A 3x4090 rig is more expensive and still consumer hardware. An H100 is not, you can buy like 7 of these optioned up MBP for a single H100.
- michaelt 2y agoIn my experience, people use the term in two separate ways. If I'm running a software business selling software that runs on 'consumer hardware' the more people can run my software, the more people can pay me. For me, the term means the hardware used by a typical-ish consumer. I'll check the Steam hardware survey, find the 75th-percentile gamer has 8 cores, 32GB RAM, 12GB VRAM - and I'd better make sure my software works on a machine like that. On the other hand, 'consumer hardware' could also be used to simply mean hardware available off-the-shelf from retailers who sell to consumers. By this definition, 128GB of RAM is 'consumer hardware' even if it only counts as 0.5% in Steam's hardware survey.
- evilduck 2y agoOn the Steam Hardware Survey the average gamer uses a computer with a 1080p display too. That doesn't somehow make any gaming laptop with a 2k screen sold in the last half decade a non-consumer product. For that matter the average gaming PC on Steam is even above average relative to the average computer. The typical office computer or school Chromebook is likely several generations older doesn't have an NPU or discrete GPU at all. For AI and LLMs, I'm not aware of any company even selling the models assets directly to consumers, they're either completely unavailable (OpenAI) or freely licensed so the companies training them aren't really dependent what the average person has for commercial success.