4 ms·
Today appears to be the day you can run an LLM that is competitive with GPT-4o at home with the right hardware. Incredible for progress and advancement of the t
by netsec_burn 2y ago
Today appears to be the day you can run an LLM that is competitive with GPT-4o at home with the right hardware. Incredible for progress and advancement of the technology.
Statement from Mark: https://about.fb.com/news/2024/07/open-source-ai-is-the-path-forward/ https://about.fb.com/news/2024/07/open-source-ai-is-the-path...
- lolinder 2y ago> at home with the right hardware Where the right hardware is 10x4090s even at 4 bits quantization. I'm hoping we'll see these models get smaller, but the GPT-4-competitive one isn't really accessible for home use yet. Still amazing that it's available at all, of course!
- petercooper 2y agoIt's hardly cheap starting at about $10k of hardware, but another potential option appears to be using Exo to spread the model across a few MBPs or Mac Studios: https://x.com/exolabs_/status/1814913116704288870 https://x.com/exolabs_/status/1814913116704288870
- niutech 2y agoOr maybe using Distributed Llama? https://github.com/b4rtaz/distributed-llama https://github.com/b4rtaz/distributed-llama
- dunefox 2y agoIt's not really competitive though, is it? I tested it and 4o is just better.
- dunefox 2y agoDisclaimer: I tested llama3-8B, 3.1 might even as a small model be better, but I so far I have not seen a single small model approach 4o, ime.
- deleted 2y ago[deleted]