9 ms·
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
by humanistbot 4y ago
This will only happen if "Open"AI or other big orgs release the model weights, which only Stable Diffusion did. Cost to train is still astronomical.
- Dylan16807 4y agoOn the other hand, one techie with a few million dollars... And you could train something like GPT-3 for cheaper than a superbowl commercial. That would get you a lot of publicity.
- celdon25 4y agoI would hope publicity isn’t the motivation for doing it though.
- idiotsecant 4y agoWhat motivation would be sufficiently noble?
- celdon25 4y agoProbably one where there isn't an intrinsic conflict of interest with AI risk. Or from a more traditional angle, one where the author's vanity isn't required to be appeased in order for users/customers to be happy. I'm of the opinion that you should do something with game-changing technology because the world needs it, not because you need an ego boost. All technology brings side effects, and there is no greater example of that than "democratized" AI...
- idiotsecant 4y agoPeople often (usually) do objectively useful things because it's in their selfish interests to do so, ego or otherwise. The surest road to failure is expecting people to act virtuously. Generally systems that assume virtue fail, and systems that assume selfish action and steer that selfish action towards the greater good succeed. In other words, I don't care why people do things, only that they do.
- celdon25 4y agoThat’s fine, as long as publicity isn’t the motivation. It’s safe to assume that isn’t optimal for a projects success (Satoshi understood this). Not sure where you got the idea that the inverse of that was beneficial to such a project. I’ve seen first hand where it becomes a problem. I’m not aware of many examples of starry-eyed divas achieving great results. Usually you hear about them but only because they are exceptional cases, not the norm. It’s a matter of practicality and not virtue (to say otherwise is purely a straw man argument).
- JoshCole 4y ago> to say otherwise is purely a straw man argument This is really overconfident. That publicity isn't causally connected to success is belied by the existence of the advertising industry. While generally refuting across industries, it is worth noting that the most dominant AI company - Google - happens to be in this industry. They are explicitly known - having publicity for - their generous compensation packages. This is because of a causal model of talent attraction. Success is obviously causally connected to publicity and the idea that it isn't isn't well supported by the evidence. Contrary to your assertion, it was not a safe assumption. Your appeal to Satoshi is an appeal to authority, not a causal model of its shielding off from project impacts.
- celdon25 4y ago> That publicity isn't causally connected to success is belied by the existence of the advertising industry. The argument was about publicity as a reward motivator, not publicity itself, as a causal relation to success. To phrase it plainly: Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about? Or those looking for money or fame? Last time I checked the latter case would be seen as a strong negative. The appeal to authority argument doesn’t apply in this situation because the VC portfolio performance is causally related to how accurately they predict future success of a company. On the scale of a smaller project like this, a common failure mode is for a maintainer to stop caring about the project and go to the next thing that motivates them. Someone else may attempt to use the code or project without understanding the theory behind it. And even worse: every time this happens is a signal that this is acceptable. AI is a different beast. Software bugs with big AI systems will become more costly, and eventually deadly. Unfortunately I’m not sure what can be done about it without a global totalitarian regime to ban its use entirely (which is not an idea I support anyway). Eventually the broken clock will be right and some profit-driven AI project will succeed in making the world a not better place, if we are even around to notice :). I would advise deeper thought into these topics when convenient. Read Nick Bostrom’s Superintelligence book or watch his talks, at least one of which was at Google HQ. I think someone should train ChatGPT or similar to argue or teach traditional AGI Philosophy/Ethics and hopefully that will move the needle somewhat more than the OpenAI nannyism we have now.
- IncRnd 4y agoNoble? You're anthropomorphising machine learning. On possible motiviation would be to train a model, instead of training a model in order to create publicity around a model being trained.
- idiotsecant 4y agoI think you're misreading, nobody is anthropomorphizing anything other than the very 'anthro' component of the system we're talking about - the people distributing the funding.
- IncRnd 4y agoI may have misread your comment, then. Either way, thank you for the explanation!
- paxys 4y agoYou can do it once, but probably not every day.
- Ajedi32 4y agoWhy would you want to retrain it from scratch every day? Stable Diffusion doesn't do that either.
- paxys 4y agoBecause things happen every day. If ChatGPT wants to compete with Google, staying up to date with recent events is the minimum bar.
- Ajedi32 4y agoYou wouldn't need to re-train from scratch for that, just fine-tune on the new data sources. I don't think constant re-training is the optimal strategy for that use-case anyway. Bing does it by letting the LLM search a more traditional web index to find the information it needs.
- paxys 4y agoOkay but someone has to do the fine tuning. The code has to be updated. Parts of the training have to be redone. All of this has costs. It isn't a "do it once and forget about it" task that it is being touted as in this thread.
- coldtea 4y ago>The code has to be updated I'm pretty sure this is not how an LLM works. >It isn't a "do it once and forget about it" task that it is being touted as in this thread. That's neither here, nor there. Training the LLM itself is not a "do it multiple times per day if you want to compete with Google" thing as it has been stated in this subthread.
- idiotsecant 4y agoIs there information out there about how much it cost (in time or human-hours) to do the additional training necessary to make chatGPT? I am genuinely curious what the scale of the effort was.
- mensetmanusman 4y agoVCs are funding $100,000,000 AI compute efforts now, so it might be something like that.
- bitL 4y agoAny larger crypto enthusiast with a bunch of 3090s and a solar farm can do it for nearly free (assuming fixed expenses paid by Eth already...)
- JoshCole 4y agoNo, it isn't astronomical. It is smaller than that. Still large, but not astronomical.
- ipsum2 4y agoHave you tried training a large model before? If not, you're probably discounting how difficult and expensive it is.
- coldtea 4y agoWell, for those that trained the largest one atm, it cost them in the order of 10 million dollars (actually less). That's how much some tech companies pay for catering. Hell, that's in the order of a single socialite's wedding costs.
- deleted 4y ago[deleted]
- JoshCole 4y agoAt Voyager's speed it would take approximately 749,000,000 years to reach Canis Major Dwarf. OpenAI was founded in 2015. So it has been eight years. 749,000,000 - 8 = 748,999,992. 1.06809079e-8% of the time of a random astronomical time; rounded, that is about, uh, 0.00%ish. I mean, don't get me wrong. It is a very expensive project. It just isn't astronomical. Anyone reading this and thinking - oh I could never do that even in hundreds of millions of years - that would be wrong. If you won the lottery or just made good financial decisions you could do a project comparable to this instead of getting a very nice house in the Bay Area.
- naillo 4y agoThere are some open source LLM models already such as the one this repo is running and mentioning like OPT-175B
- tarr11 4y agoWonder if someone would be willing to start an open source project where we could crowdsource donations for training, and people could possibly donate their GPU usage for it.
- polishdude20 4y agoThere's gotta be something like this already. Like a SETI @ Home type of thing .
- mryab 4y agoThere is! See https://petals.ml/ https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)
- exebook 4y agoAmazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/ https://aqualxx.github.io/stable-ui/
- efreak 4y agohttps://github.com/aqualxx/stable-horde-notebook https://github.com/aqualxx/stable-horde-notebook My only problem with stable horde is that their anti-cp measure involves checking the prompt for words like small, meaning I can't use a nsfw-capable model with certain prompts (holding a very small bag, etc). That, and seeing great things in the image rating and being unable to reproduce because it doesn't provide the prompt.
- scottmf 4y agoThat’s what Stability AI has been doing… There are already open source LLMs the size of GPT-3 such as OPT and Bloom
- gaogao 4y agoMeta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.
- stavros 4y agoAre there any that perform anywhere close to GPT-3?
- jejeyyy77 4y agoStable Diffusion was behind DALLE, but has since surpassed it. Don't hear anyone talking about DALLE anymore.
- neilmovva 4y agoWhile OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck. And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which they seem to have spent a lot of effort cleaning. So: open source models are probably data-constrained, in both quantity and quality.
- mensetmanusman 4y agoIt’s fun to think about a few billion weights being the difference between useless and gold.
- tiborsaas 4y agoLooking at my bank account I can relate :)
- Miraste 4y agoOPT-175B isn't publicly available, sadly. It's available to research institutions, which is much better than "Open"AI, but it doesn't help us hobbyists/indie researchers much.
- JoshCole 4y agoNo, Stable Diffusion isn't the only one to release their weights. OpenAI hasn't been releasing weights for ChatGPT, but Stable Diffusion isn't the only ones releasing weights [1]. [1]: https://huggingface.co/ https://huggingface.co/
- leesec 4y agoNo it isn't. Stable Diffusion is less than 200 grand to train.
- anononaut 4y agoI heard it was $4MM alone in AWS compute time.
- postalrat 4y agoSo about 40k on your own machines.
- inciampati 4y agoIt does make it seem like a box of H100s will easily be able to make an interesting open LLM.
- speedgoose 4y agoThis number seems to match the $200k, if you take into account the cloud margins of our favourite counterfeit products reseller.
- huijzer 4y agoAccording to Christopher Potts (Stanford Professor and Chair, Department of Linguistics, and Professor, by courtesy, Department of Computer Science), training a large language model costs about 50 million [1]. [1]: https://youtu.be/-lnHHWRCDGk?t=637 https://youtu.be/-lnHHWRCDGk?t=637
- nl 4y agoYeah this is way wrong, unless counting the salaries of everyone involved for a few years in the lead up while writing software that ended up being used https://bigscience.huggingface.co/blog/bloom https://bigscience.huggingface.co/blog/bloom was trained with a $7M grant, and that was the first time they'd done it (so there was a lot of failures).
- jejeyyy77 4y agoCrowd source the training costs - leaving some profit for the project owner. Open source the models.
- nl 4y agoThere are current open source projects working on training their own LLMs. I'm aware of one very credible one that has applied for a TPU grant from Google worth under $200K to train the whole model. I think laion.ai will probably get their first with their fleet of A100s though.
- eternalban 4y agoA few million dollars. Kickstart the project, get 100 a head and 100,000 backers. Also check with Uncle Sam and see if there are any grants that can be used for this. Start a campaign and get rich concerned people to donate. Jeff may also want to show AWS can also train AI so maybe even get a break there an Amazon can get some nice PR. The list of possibilities seems extensive given the price tag of $12MM and upside of a fully public GPT.
- pavelstoev 4y agoIt does not have to be. We have optimizations for all kinds of workloads - https://CentML.ai https://CentML.ai
- worldsayshi 4y ago> Cost to train is still astronomical. It sounds like something that could/should/would be crowdfunded?