7 ms·
>The vast majority of users couldn't be bothered to take the steps necessary to get it running locally so I don't even think the open sourcing philosophy would
by llm_trw 3y ago
>The vast majority of users couldn't be bothered to take the steps necessary to get it running locally so I don't even think the open sourcing philosophy would have been a serious hurdle to wider commercial adoption.
The more I think about the AI space the more I realize that open sourcing large models is pointless now.
Until you can reasonably buy a rig to run the model there is simply no point in doing this. It's no like you will be edified by setting the weights either.
I think an ethical business model for these business is to release whatever model can fit into a $10,000 machine and keeping the rest closed source until above machine is able to run them.
- AnthonyMouse 3y agoThe released image generation models run on consumer GPUs. Even the big LLMs will run on a $3500 Mac with reasonable performance, and the CPU of a dirt cheap machine if you don't care about it being slow, which is sometimes important and sometimes isn't. Also, things like this are in the works: https://news.ycombinator.com/item?id=39794864 https://news.ycombinator.com/item?id=39794864 Which will put the system RAM of the new 24-channel PC servers in range of the Nvidia H100 on memory bandwidth, while using commodity DDR5.
- llm_trw 3y agoThe `big' AI models are trillion parameter models. The medium sized models like GPT3 and Grok are 185b and 314b respectively. There is no way for _anyone_ to run these on a sub $50k machine in 2024, and even if you can the token generation speed on CPU is under 0.1 tokens per second.
- GaggiX 3y agoChatGPT is 20B according to Microsoft researchers, also the fact that big AI models are trillion parameter models is mostly speculation, about GPT-4 it was spread by geohot.
- speedgoose 3y agoTo be precise, ChatGPT 3.5 turbo being 20B is officially a mistake from a Microsoft Researcher, quoting a wrong source published before the release of chatgpt3.5 turbo. Up to you to believe it or not. But I wouldn’t claim it’s a 20B according to Microsoft Researchers. The withdrawn paper: https://arxiv.org/abs/2310.17680 https://arxiv.org/abs/2310.17680 The wrong source: https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat/ https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is... The discussion: https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_microsoft_codediffusion_paper_suggests_gpt35/ https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_mic...
- GaggiX 3y agoIt's interesting how the paper was completely retracted instead of just being corrected.
- razodactyl 3y agoYep. It feels like a 20B parameter model.
- HarHarVeryFunny 3y agoGPT-3 was 175B, so it'd be a bit odd if GPT-4 wasn't at least 5x larger (1T), especially since it's apparently a mixture of experts.
- razodactyl 3y agoI think it became apparent when mixtral came out. I've noticed too during training that my model overwrites useful information so it makes sense for these types of models to have emerged.
- AnthonyMouse 3y agoYou can get registered DDR4 for ~$1/GB. A trillion parameter model in FP16 would need ~2TB. Servers that support that much are actually cheap (~$200), the main cost would be the ~$2000 in memory itself. That is going to be dog slow but you can certainly do it if you want to and it doesn't cost $50,000.
- fauigerzigerk 3y agoHow slow? Depending on the task I fear it could be too slow to be useful. I believe there is some research on how to distribute large models across multiple GPUs, which could make the cost less lumpy.
- AnthonyMouse 3y agoYou can get a decent approximation for LLM performance in tokens/second by dividing the model size in GB by the system's memory bandwidth. That's assuming it's well-optimized and memory rather than compute bound, but those are often both true or pretty close. And "depending on the task" is the point. There are systems that would be uselessly slow for real-time interaction but if your concern is to have it process confidential data you don't want to upload to a third party you can just let it run and come back whenever it finishes. And releasing the model allows people to do the latter even if machines necessary to do the former are still prohibitively expensive. Also, hardware gets cheaper over time and it's useful to have the model out there so it's well-optimized and stable by the time fast hardware becomes affordable instead of waiting for the hardware and only then getting to work on the code.
- bawana 3y agoWhy would increasing memory bandwidth reduce performance? You said "You can get a decent approximation for LLM performance in tokens/second by dividing the model size in GB by the system's memory bandwidth"
- deleted 3y ago[deleted]
- raincole 3y agoIt's just semantic gymnastics. I'm sure most people will consider LLaMa 70B a big model. Of course if you define big = trillion then sure big = trillion[1]. [1]: https://en.wikipedia.org/wiki/No_true_Scotsman https://en.wikipedia.org/wiki/No_true_Scotsman
- llm_trw 3y agoYes, you are engaging in the no true scotsman fallacy, please stop.
- renewiltord 3y agoMixtral 8x7b is better than both of those and runs on a top spec M3 Max wonderfully.
- woadwarrior01 3y agoIndeed! Also, Mixtral 8x7b runs just as well on older M1 Max and M2 Max Macs, since LLM inference is memory bandwidth bound and memory bandwidth hasn't significantly changed between M1 and M3.
- karolist 3y agoIt didn't change at all, rather was reduced in certain configurations.
- Takennickname 3y agoI will make a 2 trillion parameter model just so your comment becomes outdated and wrong.
- razodactyl 3y agoI approve this comment.
- bradley13 3y agoDisagree. A few weeks ago, I followed a step-by-step tutorial to diwnlad ollama, which in turn can download various models. On my not-soecisl laptop with a so-so graphics card, Mixtral runs just fine. As models advance, they will become - not just larger - but also more efficient. Hardware advances. Large models will run just fine on affordable hardware in just a few years.
- Closi 3y agoI’ve come to the opposite conclusion personally - AI model inference requires burst compute, which particularly suits cloud deployment (for these sort of applications). And while AIs may become more compute-efficient in some respects, the tasks we ask AIs to do will grow larger and more complex. Sure you might get a good image locally but what about when the market moves to video? Sure chat GPT might give good responses locally, but how long will it take when you want it to refactor an entire codebase? Not saying that local compute won’t have its use-cases though… and this is just a prediction that may turn out to be spectacularly wrong!
- aerhardt 3y agoOk but yesterday I was on a plane coding and I wouldn’t have minded having GPT4 as it is today available to me.
- FeepingCreature 3y agoThanks to Starlink, planes should have good internet soon.
- michaelt 3y agoHuge models are the best type to open source. You get all the benefits of academics and open source folks pushing your model forward, and a vastly improved hiring pool. But it doesn't stop you launching a commercial offering, because 99.99% of the world's population doesn't have 48GB+ of VRAM.