4 ms·
The `big' AI models are trillion parameter models. The medium sized models like GPT3 and Grok are 185b and 314b respectively. There is no way for _anyone_ to
by llm_trw 3y ago
The `big' AI models are trillion parameter models.
The medium sized models like GPT3 and Grok are 185b and 314b respectively.
There is no way for _anyone_ to run these on a sub $50k machine in 2024, and even if you can the token generation speed on CPU is under 0.1 tokens per second.
- GaggiX 3y agoChatGPT is 20B according to Microsoft researchers, also the fact that big AI models are trillion parameter models is mostly speculation, about GPT-4 it was spread by geohot.
- speedgoose 3y agoTo be precise, ChatGPT 3.5 turbo being 20B is officially a mistake from a Microsoft Researcher, quoting a wrong source published before the release of chatgpt3.5 turbo. Up to you to believe it or not. But I wouldn’t claim it’s a 20B according to Microsoft Researchers. The withdrawn paper: https://arxiv.org/abs/2310.17680 https://arxiv.org/abs/2310.17680 The wrong source: https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat/ https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is... The discussion: https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_microsoft_codediffusion_paper_suggests_gpt35/ https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_mic...
- GaggiX 3y agoIt's interesting how the paper was completely retracted instead of just being corrected.
- razodactyl 3y agoYep. It feels like a 20B parameter model.
- HarHarVeryFunny 3y agoGPT-3 was 175B, so it'd be a bit odd if GPT-4 wasn't at least 5x larger (1T), especially since it's apparently a mixture of experts.
- razodactyl 3y agoI think it became apparent when mixtral came out. I've noticed too during training that my model overwrites useful information so it makes sense for these types of models to have emerged.
- AnthonyMouse 3y agoYou can get registered DDR4 for ~$1/GB. A trillion parameter model in FP16 would need ~2TB. Servers that support that much are actually cheap (~$200), the main cost would be the ~$2000 in memory itself. That is going to be dog slow but you can certainly do it if you want to and it doesn't cost $50,000.
- fauigerzigerk 3y agoHow slow? Depending on the task I fear it could be too slow to be useful. I believe there is some research on how to distribute large models across multiple GPUs, which could make the cost less lumpy.
- AnthonyMouse 3y agoYou can get a decent approximation for LLM performance in tokens/second by dividing the model size in GB by the system's memory bandwidth. That's assuming it's well-optimized and memory rather than compute bound, but those are often both true or pretty close. And "depending on the task" is the point. There are systems that would be uselessly slow for real-time interaction but if your concern is to have it process confidential data you don't want to upload to a third party you can just let it run and come back whenever it finishes. And releasing the model allows people to do the latter even if machines necessary to do the former are still prohibitively expensive. Also, hardware gets cheaper over time and it's useful to have the model out there so it's well-optimized and stable by the time fast hardware becomes affordable instead of waiting for the hardware and only then getting to work on the code.
- bawana 3y agoWhy would increasing memory bandwidth reduce performance? You said "You can get a decent approximation for LLM performance in tokens/second by dividing the model size in GB by the system's memory bandwidth"
- deleted 3y ago[deleted]
- raincole 3y agoIt's just semantic gymnastics. I'm sure most people will consider LLaMa 70B a big model. Of course if you define big = trillion then sure big = trillion[1]. [1]: https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
- llm_trw 3y agoYes, you are engaging in the no true scotsman fallacy, please stop.
- renewiltord 3y agoMixtral 8x7b is better than both of those and runs on a top spec M3 Max wonderfully.
- woadwarrior01 3y agoIndeed! Also, Mixtral 8x7b runs just as well on older M1 Max and M2 Max Macs, since LLM inference is memory bandwidth bound and memory bandwidth hasn't significantly changed between M1 and M3.
- karolist 3y agoIt didn't change at all, rather was reduced in certain configurations.
- Takennickname 3y agoI will make a 2 trillion parameter model just so your comment becomes outdated and wrong.
- razodactyl 3y agoI approve this comment.