10 ms·
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3
by gorbypark 4y ago
If this works well, it will be a game changer. Requiring a fleet of $10k+ GPUs will kill any hope of wide spread adoption of open source "competitors" to GPT-3. Stable Diffusion is so popular because it can run on hardware mere mortals can own.
- narrator 4y agoNo doubt the corporate large language models will use it to make language models that are 10x bigger. However, at least the public will have access to 175B parameter language models which are much more sophisticated than the 6B or so parameter models consumer video cards can currently run.
- humanistbot 4y agoThis will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
- Dylan16807 4y agoOn the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.
- celdon25 4y agoI would hope publicity isn’t the motivation for doing it though.
- idiotsecant 4y agoWhat motivation would be sufficiently noble?
- celdon25 4y agoProbably one where there isn't an intrinsic conflict of interest with AI risk. Or from a more traditional angle, one where the author's vanity isn't required to be appeased in order for users/customers to be happy. I'm of the opinion that you should do something with game-changing technology because the world needs it, not because you need an ego boost. All technology brings side effects, and there is no greater example of that than "democratized" AI...
- idiotsecant 4y agoPeople often (usually) do objectively useful things because it's in their selfish interests to do so, ego or otherwise. The surest road to failure is expecting people to act virtuously. Generally systems that assume virtue fail, and systems that assume selfish action and steer that selfish action towards the greater good succeed. In other words, I don't care why people do things, only that they do.
- celdon25 4y agoThat’s fine, as long as publicity isn’t the motivation. It’s safe to assume that isn’t optimal for a projects success (Satoshi understood this). Not sure where you got the idea that the inverse of that was beneficial to such a project. I’ve seen first hand where it becomes a problem. I’m not aware of many examples of starry-eyed divas achieving great results. Usually you hear about them but only because they are exceptional cases, not the norm. It’s a matter of practicality and not virtue (to say otherwise is purely a straw man argument).
- JoshCole 4y ago> to say otherwise is purely a straw man argument This is really overconfident. That publicity isn't causally connected to success is belied by the existence of the advertising industry. While generally refuting across industries, it is worth noting that the most dominant AI company - Google - happens to be in this industry. They are explicitly known - having publicity for - their generous compensation packages. This is because of a causal model of talent attraction. Success is obviously causally connected to publicity and the idea that it isn't isn't well supported by the evidence. Contrary to your assertion, it was not a safe assumption. Your appeal to Satoshi is an appeal to authority, not a causal model of its shielding off from project impacts.
- paxys 4y agoYou can do it once, but probably not every day.
- Ajedi32 4y agoWhy would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
- paxys 4y agoBecause things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.
- Ajedi32 4y agoYou wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.
- paxys 4y agoOkay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.
- coldtea 4y ago>The code has to be updated I'm pretty sure this is not how an LLM works. >It isn't a "do it once and forget about it" task that it is being touted as in this thread. That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.
- idiotsecant 4y agoIs there information out there about how much it cost (in time or human-hours) to do the additional training necessary to make chatGPT? I am genuinely curious what the scale of the effort was.
- mensetmanusman 4y agoVCs are funding $100,000,000 AI compute efforts now, so it might be something like that.
- bitL 4y agoAny larger crypto enthusiast with a bunch of 3090s and a solar farm can do it for nearly free (assuming fixed expenses paid by Eth already...)
- JoshCole 4y agoNo, it isn't astronomical. It is smaller than that. Still large, but not astronomical.
- ipsum2 4y agoHave you tried training a large model before? If not, you're probably discounting how difficult and expensive it is.
- coldtea 4y agoWell, for those that trained the largest one atm, it cost them in the order of 10 million dollars (actually less). That's how much some tech companies pay for catering. Hell, that's in the order of a single socialite's wedding costs.
- deleted 4y ago[deleted]
- JoshCole 4y agoAt Voyager's speed it would take approximately 749,000,000 years to reach Canis Major Dwarf. OpenAI was founded in 2015. So it has been eight years. 749,000,000 - 8 = 748,999,992. 1.06809079e-8% of the time of a random astronomical time; rounded, that is about, uh, 0.00%ish. I mean, don't get me wrong. It is a very expensive project. It just isn't astronomical. Anyone reading this and thinking - oh I could never do that even in hundreds of millions of years - that would be wrong. If you won the lottery or just made good financial decisions you could do a project comparable to this instead of getting a very nice house in the Bay Area.
- naillo 4y agoThere are some open source LLM models already such as the one this repo is running and mentioning like OPT-175B
- tarr11 4y agoWonder if someone would be willing to start an open source project where we could crowdsource donations for training, and people could possibly donate their GPU usage for it.
- polishdude20 4y agoThere's gotta be something like this already. Like a SETI @ Home type of thing .
- mryab 4y agoThere is! See https://petals.ml/ https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)
- exebook 4y agoAmazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/ https://aqualxx.github.io/stable-ui/
- efreak 4y agohttps://github.com/aqualxx/stable-horde-notebook https://github.com/aqualxx/stable-horde-notebook My only problem with stable horde is that their anti-cp measure involves checking the prompt for words like small, meaning I can't use a nsfw-capable model with certain prompts (holding a very small bag, etc). That, and seeing great things in the image rating and being unable to reproduce because it doesn't provide the prompt.
- scottmf 4y agoThat’s what Stability AI has been doing… There are already open source LLMs the size of GPT-3 such as OPT and Bloom
- gaogao 4y agoMeta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
- stavros 4y agoAre there any that perform anywhere close to GPT-3?
- jejeyyy77 4y agoStable Diffusion was behind DALLE, but has since surpassed it. Don't hear anyone talking about DALLE anymore.
- neilmovva 4y agoWhile OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which they seem to have spent a lot of effort cleaning. So: open source models are probably data-constrained, in both quantity and quality.
- mensetmanusman 4y agoIt’s fun to think about a few billion weights being the difference between useless and gold.
- tiborsaas 4y agoLooking at my bank account I can relate :)
- Miraste 4y agoOPT-175B isn't publicly available, sadly. It's available to research institutions, which is much better than "Open"AI, but it doesn't help us hobbyists/indie researchers much.
- JoshCole 4y agoNo, Stable Diffusion isn't the only one to release their weights. OpenAI hasn't been releasing weights for ChatGPT, but Stable Diffusion isn't the only ones releasing weights [1]. [1]: https://huggingface.co/ https://huggingface.co/
- leesec 4y agoNo it isn't. Stable Diffusion is less than 200 grand to train.
- anononaut 4y agoI heard it was $4MM alone in AWS compute time.
- postalrat 4y agoSo about 40k on your own machines.
- inciampati 4y agoIt does make it seem like a box of H100s will easily be able to make an interesting open LLM.
- speedgoose 4y agoThis number seems to match the $200k, if you take into account the cloud margins of our favourite counterfeit products reseller.
- huijzer 4y agoAccording to Christopher Potts (Stanford Professor and Chair, Department of Linguistics, and Professor, by courtesy, Department of Computer Science), training a large language model costs about 50 million [1]. [1]: https://youtu.be/-lnHHWRCDGk?t=637 https://youtu.be/-lnHHWRCDGk?t=637
- nl 4y agoYeah this is way wrong, unless counting the salaries of everyone involved for a few years in the lead up while writing software that ended up being used https://bigscience.huggingface.co/blog/bloom https://bigscience.huggingface.co/blog/bloom was trained with a $7M grant, and that was the first time they'd done it (so there was a lot of failures).
- jejeyyy77 4y agoCrowd source the training costs - leaving some profit for the project owner. Open source the models.
- nl 4y agoThere are current open source projects working on training their own LLMs. I'm aware of one very credible one that has applied for a TPU grant from Google worth under $200K to train the whole model. I think laion.ai will probably get their first with their fleet of A100s though.
- eternalban 4y agoA few million dollars. Kickstart the project, get 100 a head and 100,000 backers. Also check with Uncle Sam and see if there are any grants that can be used for this. Start a campaign and get rich concerned people to donate. Jeff may also want to show AWS can also train AI so maybe even get a break there an Amazon can get some nice PR. The list of possibilities seems extensive given the price tag of $12MM and upside of a fully public GPT.
- pavelstoev 4y agoIt does not have to be. We have optimizations for all kinds of workloads - https://CentML.ai https://CentML.ai
- worldsayshi 4y ago> Cost to train is still astronomical. It sounds like something that could/should/would be crowdfunded?
- moffkalast 4y agoThis may be a flawed approach, but an interesting idea would be to use the current models as a preprocessor to generate a huge "labelled" dataset of inputs and outputs, and then using that more accurate and specific data to train a smaller one that would fit. It likely wouldn't have nearly as much general knowledge since the data just wouldn't be there but the behaviour could be similar?
- zeknife 4y agoLook up knowledge distillation
- moffkalast 4y agoAh TIL, so it's actually a thing and with some cleverer approaches too. Then we do have something to look forward to :D
- anon291 4y agoAs a former AI accelerator employee (laid off), I'm kind of happy I was laid off because I realistically don't see a need for specialized hardware anymore. Large companies can afford Nvidia. Nvidia's software stack is best in class. There's no business need here and the model execution is increasingly becoming possible on single consumer GPUs. The only place where I see specialized chips excelling is on the edge or if they are truly revolutionary (in which case they're only an acquisition target for Nvidia). The truth is... The large language models are likely excessively large.
- foobiekr 4y agoPower is the main reason to do custom ASICs. I’d be curious as to your opinion of Recogni given they are claiming a 10x power reduction per unit compute.
- anon291 4y agoUnfortunately, I've worked at several players which promise power reductions. It doesn't matter though. People don't care about cost at this point. If you are cost-sensitive you're not doing the kind of revolutionary AI work these companies need to create a competitive moat. And once your model works on NVIDIA and is trained, how much are you going to spend on ML engineers to make it work on something else? Because that cost better be less than the marginal cost reduction on electricity. Plus, NVIDIA et al will likely get more and more efficient.
- bitL 4y agoCost sensitive folks post TopTal jobs such as "build a ChatGPT-like service from the scratch for this industry in 2 weeks for $25/hour".
- nl 4y agoThis is exactly right. The only exception is running things on mobile. There is demand for porting models to run natively on mobile, and somewhat reasonable support for doing this.
- permo-w 4y agodo/did you seriously think that the processing requirement was going to “kill” that possibility? the history of computing clearly indicates that either the requirements for running or the bar for owning this technology was always going to drop
- pessimizer 4y agoThe history of computing had Moore's Law.
- permo-w 4y agoexactly