35 ms·
GPT-Neo – Building a GPT-3-sized model, open source and free
- newswasboring 6y agoThe intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.
- teruakohatu 6y agoTensorFlow supports distributed training with a client server model.
- newswasboring 6y agoDoes it also solve the problem of everyone having different hardware?
- londons_explore 6y agoIt does. For most models, your home broadband would be far too slow though.
- newswasboring 6y agoIs it because they will have to communicate back errors during training? I forgot that training these models is more of a global task than proteins folding. In that sense this is less parallelizable over the internet.
- londons_explore 6y agoYes, and also activations if your GPU is too small to fit the whole model. The minimum useful bandwidth for that stuff is a few gigabits...
- emteycz 6y agoWhat about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?
- danwills 6y agoYeah! sounds great, if I could easily run a SETI-at-home style thing to contribute to training a (future) model similar to GPT-x, but with the result freely available to play with, I reckon i'd do it. Could even be made into a game I'd say! I am totally aware that gpt3 itself can't be run on a regular workstation, but maybe hosting for instances of models from the best/most-interesting training runs could be worked out by crowd-funding?
- colobas 6y agoI was just gonna propose something like this. Democratizing large ML models.
- chillee 6y agoThe primary issue is that large scale GPU training is primarily dominated by communication costs. Since to some approximation things need to be synchronized after every gradient update, it becomes very quickly quite infeasible to increase the communication cost.
- mryab 6y agoNot directly related, but the Learning@home [1] project aims to achieve precisely that goal of public, volunteer-trained neural networks. The idea is that you can host separate "experts," or parts of your model (akin to Google's recent Switch Transformers paper) on separate computers. This way, you never have to synchronize the weights of the entire model across the participants — you only need to send the gradients/activations to a set of peers. Slow connections are mitigated with asynchronous SGD and unreliable/disconnected experts can be discarded, which makes it more suitable for Internet-like networks. Disclaimer: I work on this project. We're currently implementing a prototype, but it's not yet GPT-3 sized. Some issues like LR scheduling (crucial for Transformer convergence) and shared parameter averaging (for gating etc.) are tricky to implement for decentralized training over the Internet. [1] https://learning-at-home.github.io/ https://learning-at-home.github.io/
- echelon 6y agoDo you have a personal Twitter account I can follow? Your career is one I'd like to follow.
- gabriel_shatana 6y agoYour project looks so interesting. Have u thught of putting the experts on a distributed market where their expertise and work can be exchanged for some token (obvlsy using a blockchain). This would encourage people to host experts in your network and would create value.
- mryab 6y agoThank you! This is definitely something we should look into in the future (hopefully with community help); as of now, training infrastructure and model convergence are the highest priorities. That said, we welcome all ideas of ways to motivate more volunteers to join the experiments, because Learning@home team comes from a distributed DL background with limited volunteer computing expertise. Also, I believe that for some projects (e.g. GPT-3 replication effort) people would want to join the network regardless of the incentive mechanism, as demonstrated by Leela Chess Zero [1]. [1] http://lczero.org/ http://lczero.org/
- jne2356 6y agoThey get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info https://github.com/EleutherAI/info
- rijoja 6y agoYou're a genius! How do we get this started. Wouldn't there be so many bitcoin rigs already that could be used for this already? I'd be very willing to spend time and money on this! Could someone set up a chat room or something?
- anticristi 6y agoI love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows humans to focus on higher level information processing is interesting.
- Erlich_Bachman 6y agoPhotoshop has existed for decades. Is it really that big of a problem for photo news?
- technocratius 6y agoThe difference between Photoshop and generative models is not in what it can technically achieve, but the cost of achieving the desired result. Fake news photo or text generation is possible by humans, but scales poorly compared (more humans) to a algorithmically automated process (some more compute).
- draugadrotten 6y agoYes! https://en.wikipedia.org/wiki/Adnan_Hajj_photographs_controversy https://en.wikipedia.org/wiki/Adnan_Hajj_photographs_controv... https://www.bbc.com/news/world-asia-china-55140848 https://www.bbc.com/news/world-asia-china-55140848 and many more
- Agentlien 6y agoWow, that first Adnan Hajj photograph looks absolutely terrible.
- danielscrubs 6y agoTouch ups where done before photoshop but now it’s ALWAYS done. The issues this has created in society might have a bigger emotional impact than we give it credit for. Regarding photo news there has been quite a lot of scandals to the point that I’d guess the touchups is more or less accepted.
- Erlich_Bachman 6y agoIt's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.
- spiderfarmer 6y agoI don't really care if OpenAI offers commercial licenses as long as the underlying research is truly open. This way alternative options will become available eventually.
- querez 6y agoArguably openAI is one of the most closed industry AI labs (among those that are still participating in the research community), on par only with deep mind (though deepmind at least publishes way more). Funnily enough, FAIR and Google Brain have a vastly better track record wrt. publishing not only papers but also code and models.
- wraptile 6y agoOpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1] Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else: > In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment package of US$1 billion into the company. OpenAI then announced its intention to commercially license its technologies, with Microsoft as its preferred partner [2] 1 - https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill-gates-microsoft-open-ai/index.html https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill... 2 - https://en.wikipedia.org/wiki/OpenAI https://en.wikipedia.org/wiki/OpenAI
- person_of_color 6y agoSo OpenAI employees get Microsoft RSUs?
- nemoniac 6y agoGood initiative but tell us more about the governance. After all OpenAI was "open" until it was bought by Microsoft.
- wizzwizz4 6y agoNo, it wasn't. And iirc, only GPT-3 was.
- 2Gkashmiri 6y agoSo how much money would it take to rebuild this foss alternative ? And distributive power like seti@home? If it can be done and I hope it does, what benefit would the original proprietary one have over this? Licensing?
- astrange 6y agoOpenAI will execute the original one for you. If you can get an account, anyway.
- jne2356 6y agoEleutherAI has already secured the resources necessary. They get the seti@home suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info https://github.com/EleutherAI/info
- Jack000 6y agoHope there will be a distilled or approximate attention version so it can be run on consumer gpus.
- matteocapucci 6y agoDo you know how can one donate?
- techlatest_net 6y agoIs there any real justification behind this fear of close nature of OpenAI or this is just frustration coming out? We had this debate of closed Vs open source 20 years back and eventually opensource won it because of various reasons. Won't those same reasons apply to this situation of close nature of OpenAI? If so then why are people worried about this? What is differnt this time?
- pmontra 6y agoThe cost. Closed source and open source developers use the same $300-3,000 laptops / desktops. Everybody can afford them. Training a large model in a reasonable time costs much more. According to https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/ the cost of training GPT-3 was $4.6 million. Multiply it by the number of trial and errors. Of course we can't expect that something that costs tens or hundreds of millions will be given away for free or to be able to rebuild it without some collective training effort that distributes the cost on at least thousands of volunteers.
- qayxc 6y agoThis. Plus the increasing amount of intransparent results. Training data is private, so it's impossible to even try to recreate results, validate methods, or find biases/failure cases.
- stellaathena 6y agoThis is dead right and very important. The data you train on is much more important than model architecture in terms of validation and compliance, and yet it’s a closely held secret. Producing or obtaining good data is a pain in the ass. For this reason, EAI has made the data we are training on public. I can’t link to it because of anon policies at conferences, but if you look at our website I’m sure you can find a paper detailing it and a link to download it.
- jne2356 6y agoOpenAI only trained the full sized GPT-3 once. Hyperparameter sweep was conducted on significantly smaller models (see: https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361)
- hooande 6y agoIn my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling them?". And honestly, not a lot of productive applications come to mind. This is basically the description of a modern content farm. You give N people a topic, ie "dating advice", and they'll use google to put together different ideas, sentences and paragraphs to produce dozens of articles per day. You could also write very basic code with this, similar to googling a code snippet and combining the results from the first several stackoverflow pages that come up (which, incidentally, is how I program now). After a few more versions, you could probably use GPT to produce fiction that matches the quality of the average self published ebook. And DALL-E can come up with novel images in the same way that a graphic designer can visually merge the google image results for a given query. One limitation of this theoretical "team of automated googlers" is that what they search for content is cached on the date of the last GPT model update. Right now the big news story is the Jan 6th, 20201 insurrection at the US Capitol. GPT-3 can produce infinite bad articles about politics, but won't be able to say anything about current events in real time. I generally think that GPT-3 is awesome, and it's a damn shame that "Open"AI couldn't find a way to actually be open. At this point, it seems like a very interesting technology that is still in desperate need of a Killer App
- dvfjsdhgfv 6y agoIn science, some amazing discoveries are made years or even centuries before some practical applications for them are found. I believe in humanity, sooner or later we'll find some actually useful applications for GPT-X.
- Filligree 6y agoIt's a wonderful co-writer of fiction, for one. Maybe the better authors wouldn't need it, but as for everyone else -- take a look at https://forums.sufficientvelocity.com/threads/generic-pawn-templates-a-gpt-3-quest.70063/ https://forums.sufficientvelocity.com/threads/generic-pawn-t..., and compare the first couple of story posts to the last few. One of the ways in which people get GPT-3 wrong is, they give it a badly worded order and get disappointed when the output is poor. It doesn't work with orders. It takes a lot of practice to work out what it does well with. It always imitates the input, and it's great at matching style -- and it knows good writing as well as bad, but it can't ever write any better than the person using it. If you want to write a good story with it, you need to already be a good writer. But it's wonderful at busting writer's block, and at writing differently than the person using it.
- joshlk 6y agoHow does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.
- tmalsburg2 6y agoI imagine recreating the model will be computationally cheaper because they will not have to sift through the same huge hyperparameter space as the initial GPT-3 team had to.
- thelastwave 6y agoWhy is that?
- jne2356 6y agoThis is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not significantly change the feasibility of the project.
- stellaathena 6y agoHey! One of the lead devs here. A cloud computing company called CoreWeave is giving us the compute for free in exchange for us releasing it. We're currently at the ~10B scale and are working on understanding datacenter scale parallelized training better, but we expect to train the model on 300-500 V100s for 4-6 months.
- jne2356 6y agoThe cloud company CoreWeave has agreed to provide the GPU resources necessary.
- dvfjsdhgfv 6y agoIf they succeed, Eleuther should change their name to ReallyOpenAI.
- stellaathena 6y agoOr for extra irony, ClosedAI
- Havoc 6y agoWould be good if this could decentralized bittorrent/BOINC style somehow. Wouldn't mind contributing some horsepower
- mirekrusin 6y agoI'd gladly contribute (power and) few of idle GTX cards I have to public peer/volunteer/seti@home-like project if result snapshot(s) are available publicly/to registered, active contributors.
- Voloskaya 6y agoSETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.
- mitjam 6y agoMaybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo
- leogao 6y agoHandling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.
- mirekrusin 6y agoThe problem is potentially solvable as generating solutions takes a lot of GPU time and verifying it is very fast. Aquiring input data may be a problem, but should be possible with dedicated models for this type of computation.
- mirekrusin 6y agoAre there any models/research optimised on working on this kind of small, distributed batches that would fit ie. ~10GB of commodity GPU?
- mrfusion 6y agoIt still baffles me that GPT turned out to be more than a glorified markov chain text generator. It seems we’ve actually made it create a model of the world to some degree. And we kind of just stumbled on the design by throwing massive data and neural networks together?
- Filligree 6y agoIt turns out that brute-force works, and the scaling curve is still not bending. I doubt we'll ever see a GPT-4, because there are known improvements they could make besides just upsizing it further, but that's besides the point. If that curve doesn't bend soon then a 10x larger network would be human-level in many ways. (Well, that is to say. It's actually bending. Upwards.)
- mrfusion 6y agoSo fascinating. I’d love to understand why it’s working so well. I guess no one knows. Wouldn’t gpt4 just be more data and more parameters?
- hntrader 6y agoWhat % of all digitized and reasonably easy-to-access text data did they use to train GPT-3? I'm wondering whether the current limits on GPT-n are computation or data.
- kortex 6y ago> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books. It's about 400 B tokens. Library if Congress is about 40M books, let's say 50K tokens per book, or about 2T tokens. Not necessarily unique. I would say it's plausible that it was a decent percent of the indexed text available, and even more of the unique content. GPT2 was 10B tokens. Do we have 20T tokens available for GPT4? Maybe. But the low hanging fruit are definitely plucked.
- nullc 6y ago
- Mizza 6y agoSerious question: is there a warez scene for trained models yet? (I don't know how the model is accessed - are users of mainline GPT-3 given a .pb and a stack of NDAs, or do they have to access it through access-controlled API?) Wherever data is desired by many but held by a few, a pirate crew inevitably emerges.
- exhilaration 6y agoIt's via API https://openai.com/blog/openai-api/ https://openai.com/blog/openai-api/
- jokowueu 6y agoI think this also might be an interest to you https://the-eye.eu/public/AI/pile_preliminary_components/ https://the-eye.eu/public/AI/pile_preliminary_components/
- MasterScrat 6y agoThose are datasets though, not models.
- jokowueu 6y agoI am aware I just thought you might find them interesting.
- notretarded 6y agoNot really
- kordlessagain 6y agoThe model is huge and is currently run in the cloud on many machines.
- mortehu 6y agoIt's only 175 billion parameters, so presumably it can fit on a single computer with 1024 GB RAM.
- wumms 6y agohttps://github.com/EleutherAI/gpt-neo https://github.com/EleutherAI/gpt-neo (Couldn't find it on the website)
- sillysaurusx 6y agoThat’s because the code is inefficient, won’t scale, and doesn’t work too well. The project is hype and no substance. They also pivoted to dall-e to cover up their complete failure to deliver on any of their promises with gpt3, which was an interesting move.
- rexreed 6y agoOnce Microsoft became the sole and exclusive licensee of GPT-3, a credible open source effort was bound to emerge, and I hope it really is an effective alternative.
- fakedang 6y agoEleuther works with Google. I'd rather prefer the Microsoft demon rather than the Google Demon.
- jne2356 6y agoThe resources for the GPT-3 replication will be provided by the cloud company CoreWeave.
- stellaathena 6y ago“Works with Google” and “gets free compute from TFRC” are not really the same thing.
- pjfin123 6y agoWhat does the future of open-source large neural nets look like? My understanding is GPT-3 takes ~600GB of GPU memory to run inference. Does an open source model just allow you a choice of a handful of cloud providers instead of one?
- aabhay 6y agoOpen source doesn’t mean that everyone will be rolling their own. It means that lots of players will start to offer endpoints with GPT-X, perhaps bundled with other services. It is good for the market.
- stellaathena 6y agoRight, this is why a cloud company is sponsoring the replication. They can’t sell GPT-3 as a service until they get a copy of it.
- dmingod666 6y agoWith Open-AI being corporate controlled and not really 'Open'. Is Neo a nod at 'The Matrix'?
- ttctciyf 6y agoI love the name's play on Greek Eleutheria ("ἐλευθερία") - freedom, liberty!
- deleted 6y ago[deleted]
- onenightnine 6y agothis is beautiful. why not? maybe we can make something eventually better than the now closed source version
- habitue 6y agoIs it standard to prune these kinds of large language models once they've been trained to speed them up?
- kashyapc 6y agoIncidentally, over the weekend I listened to a two-hour-long presentation[1] by the cognitive scientist, Mark Turner[2]. The talk's main goal was to explore the question "Does cognitive linguistics offer a way forward for second-language teaching?" During the Q&A, Turner explicitly mentions GPT-3 (that's when I first heard of it) as a "futuristic (but possible) language-learning environment" that is likely to be a great boon for second-language learners. One of the appealing points seems to be "conversation [with GPT-3] is not scripted; it keeps going on any subject you like". Thus allowing you to simulate some bits of the gold standard (immersive language learning in the real world). As an advanced Dutch learner (as my fourth language), I'm curios of these approaches. And glad to see this open source model. (It is beyond ironic that the so-called "Open AI" turned out to have dubious ethics.) [1] https://www.youtube.com/watch?v=A4Q977p8PfQ https://www.youtube.com/watch?v=A4Q977p8PfQ [2] Check out the excellent book he co-authored, "Clear and simple as the truth"—it has valuable insights on improving writing based on some robust research.
- avivo 6y agoI'd love to see an equal amount of the effort put toward initiatives like this, also being put toward mitigating their extremely likely negative societal impacts (and putting in safeguards). Of course, that's not nearly as sexy. Yes, there are lots of incredible positive impacts of such technology, just like there was with fire, or nuclear physics. But that doesn't mean that safeguards aren't absolutely critical if you want it to be net win for society. These negative impacts are not theoretical. They are obvious and already a problem for anyone who works in the right parts of the security and disinformation world. We've been through all this before... https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541bd https://aviv.medium.com/the-path-to-deepfake-harm-da4effb541... Of course, some of the same people who ignored recommendations[1] for harm mitigations in visual deepfake synthesis tools (which ended up being used for espionage and botnets) seem to be working on this. [1] e.g. https://www.technologyreview.com/2019/12/12/131605/ethical-deepfake-tools-a-manifesto/ https://www.technologyreview.com/2019/12/12/131605/ethical-d...
- worldsayshi 6y agoBroken link, broken url fix: https://eleuther.ai/projects/gpt-neo https://eleuther.ai/projects/gpt-neo