15 ms·
GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models
- minimaxir 6y agoPer Twitter, there will be more info about model performance tomorrow: https://twitter.com/arankomatsuzaki/status/1373732645444579331 https://twitter.com/arankomatsuzaki/status/13737326454445793...
- deleted 6y ago[deleted]
- stellaathena 6y agoWe’ve added a table with some evaluation scores to the GitHub repo, and you can see a comparison between our scores, GPT-2, and GPT-3 here: https://twitter.com/BlancheMinerva/status/1373991896616427522?s=20 https://twitter.com/BlancheMinerva/status/137399189661642752... tl;dr we are doing pretty much exactly as well as we expected on LAMBADA and WikiText. Results on more sophisticated tasks will take some time, but HuggingFace is currently working on implementing our model in the transformers library and when they do so we can easily run a lot of analyses very quickly. We actually built an evaluation suite that integrates with HF, but interfacing with the MTF code that GPT-Neo was written in was too much of a pain in the ass because Mesh TensorFlow is the worst. https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness
- deleted 6y ago[deleted]
- ve55 6y agoThis is a nice release, but the title is a bit misleading as the released sizes (1.3B and 2.7B parameters) do not yet compare to the size of GPT-3 (175B), but rather GPT-2 (1.5B) instead (although future releases may have significantly more!). Edit: title improved, thank you!
- nl 6y agoYeah. They say they are doing a 10B release soon[1]. I suspect they have run into training issues since they are moving to a new repo[2] [1] https://twitter.com/arankomatsuzaki/status/1373732646811967490 https://twitter.com/arankomatsuzaki/status/13737326468119674... [2] https://github.com/EleutherAI/gpt-neox/ https://github.com/EleutherAI/gpt-neox/
- chillee 6y agoIt's more about hardware - these models were trained on TPUs, while GPT-NeoX is being trained on GPUs graciously provided by Coreweave.
- orra 6y agoAny idea what the required GPU time would cost (if not donated)? Is GPT-3 just a commodity soon?
- Voloskaya 6y ago~4M$ per full training give or take.
- teruakohatu 6y agoThe number thrown around for gpt-3 is $4.6 million, but I am not sure where that figure originates.
- minimaxir 6y agoIt was a number tossed around by a GPU hosting provider, based on their own costs: https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/ The reality is that GPT-3 was likely "free" to train on Azure, as Microsoft has provided a lot of resources to OpenAI.
- exikyut 6y ago
- graiz 6y agoA great start for a truly open approach. It's ironic that OpenAI isn't particularly open about its tech.
- victor9000 6y agoIt was disappointing to see just how quickly ClosedAI changed its tune once they produced something of value.
- catillac 6y agoIs ClosedAI some counterpart to OpenAI with a clever name?
- qPM9l3XJrF 6y agoMany people (correctly, in my view) criticized OpenAI for the name, saying that openness should be evaluated on a case by case basis. Glad they listened to critics instead of trying to maintain consistency for its own sake.
- FL33TW00D 6y agoWhilst obviously BERT is not the same as GPT-3 in architecture, Amazons recent paper discussing architecture optimizations for BERT seems pretty relevant here (https://arxiv.org/pdf/2010.10499.pdf https://arxiv.org/pdf/2010.10499.pdf) given the chance to improve upon GPT-3s architecture (because it surely isn't the best we can get). Have the Eleuther.ai team been exploring this?
- The_rationalist 6y agoDoes it leverage deepspeed/zero 3?
- minimaxir 6y agoThat’s PyTorch only; the current models are TensorFlow.
- The_rationalist 6y agoOh that's unfortunate, can't the models be exported to pytorch through e.g onnx?
- machinevision 6y agoEven small models can be a headache to export if they have even they use anything custom. I can't even imagine something the size of GPT-3.
- Voloskaya 6y agoLarger models aren't really more complicated than smaller ones though. GPT-2 is already supported, I believe only difference with GPT-3 is sparse attention.
- minimaxir 6y agoThe GPT-2 models included with Transformers can export to ONNX fine w/ a helper included with the Python onnxruntime.
- minimaxir 6y agoTo run in PyTorch the model architecture must be ported. ONNX is slightly different; you could export the model to run in the onnxruntime but that has tradeoffs.
- stellaathena 6y agoThere's a PyTorch + DeepSpeed repository here: https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox
- pjfin123 6y agoGPT Paper: "Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters" README: "1T or bust my dudes [...] An implementation of model & data parallel GPT2 & GPT3 -like models, with the ability to scale up to full GPT3 sizes (and possibly more!)" It seems the largest model they released is 2.7 billion parameters or ~0.01 the size of GPT-3. The most interesting part about GPT-3 was its size and it seems this is only "GPT-3-like" in architecture. I also have a translation library with ~100 million (0.001 GPT-3) parameters: https://github.com/argosopentech/argos-translate https://github.com/argosopentech/argos-translate
- f430 6y agowhat does he mean when he says 1T or bust? Is he referring to 1 trillion parameters? Are you saying that GTP-3 has 2.7 trillion parameters? Does it mean that to get to GPT-3 level it needs 100x more amount of dataset?
- pjfin123 6y agoI interpreted it as aspiring to a trillion paramters but I'm not sure.
- jon_tau 6y agoThe saying comes from a slide by Noam Shazeer (see: https://www.youtube.com/watch?v=HgGyWS40g-g&ab_channel=TensorFlow&t=13m10s https://www.youtube.com/watch?v=HgGyWS40g-g&ab_channel=Tenso...). It just means the current goal should be to have models with 1 trillion parameters.
- Voloskaya 6y agoGPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scaling the numbers of GPUs.
- f430 6y agoI see so that means this GPT Neo is 64 less powerful?
- f430 6y agoCan somebody explain to this beginner how to use this? Where can I load this code and start running it? How can I train it on a dataset and what do I need to prepare? Lots of language here I don't understand like what is he referring to when he says 1.5B or 1T weights? What resources/videos can I watch in order to start tinkering with this?
- vertis 6y agoThe repository readme actually includes a link to a notebook[1] that helps getting started on Google colab. It's as good a place to start as any: [1]: https://colab.research.google.com/github/EleutherAI/GPTNeo/blob/master/GPTNeo_example_notebook.ipynb https://colab.research.google.com/github/EleutherAI/GPTNeo/b...
- f430 6y agothanks, I never used this before. Do I have to add a credit card? How much will it cost to run this?
- zora_goron 6y agoColab is free to use -- you can click Runtime → Run All to run the cells in the notebook free-of-charge. (You may need to be logged in to a Google Account to run it.)
- f430 6y agovery cool! side question but is there a complete guide to learn PyTorch with Colab? I tried to learn ML a few years ago but gave up because I couldn't install CUDA on my machine for some reason. The landscape seems to change dramatically. I am interested in transformers in particular completing incomplete images like what https://openai.com/blog/image-gpt/ https://openai.com/blog/image-gpt/ does, is there a project that implements that and can let me start training? I'm excited but I just get overwhelmed as to where I need to focus my attention on. My goal is to utilize something like image-gpt but for a more narrow domain (ex. only dealing with cats), how can I build my knowledge and skills towards that goal? Much thanks for your answers I'm really looking to learn these stuffs
- leogao 6y agoCould the title of this post be change to emphasize that the model sizes released were 1.3B and 2.7B? Something like "EleutherAI releases 1.3B and 2.7B parameter GPT-like language models". The current title implies that a full sized GPT-3 model is currently available, which is not the case. edit: the title has been changed, seems good enough
- dylanbyte 6y agoCurious to see what parameter size of gpt3 this will end up being equivalent to. Obviously we won't know until they evaluate their models.
- Voloskaya 6y agoIt's trained using the same architecture, and with a very similar dataset, so it should be very close.
- dylanbyte 6y agoMy experience is that replicating papers is actually nontrivial. For example someone announced they had replicated gpt2 some time back but when evals were run it turned about to be the equivalent of a much smaller model.
- deleted 6y ago[deleted]
- joshhart 6y agoI think we need more funding outside of large tech companies and OpenAI for these kinds of things. I wonder if there is a way to crowdsource donations to rent the hardware to train big versions of these things in an open manner.
- monkeydust 6y agoIf I wanted to build an support Q&A system using texts from support logs, training docs, transcribed videos etc etc (basically as much text about my product as I can get) would this model be a good start ?
- malka 6y agoYou should look on hugging faces https://huggingface.co/ https://huggingface.co/ https://huggingface.co/transformers/task_summary.html#extractive-question-answering https://huggingface.co/transformers/task_summary.html#extrac... more precisely
- FL33TW00D 6y agoDepending on how much content you've got, this blog post from HuggingFace might be interesting: https://yjernite.github.io/lfqa.html https://yjernite.github.io/lfqa.html
- ggnore7452 6y agoalso, for some quick and simple Q&A system. Haystack https://github.com/deepset-ai/haystack https://github.com/deepset-ai/haystack (essentially dense vector similarity on Elastic Search) looks pretty promising and supports whole pipeline.
- monkeydust 6y agoThanks that looks decent for quick start.
- kristoo 6y agoThat would be one incredibly expensive Q&A system. As rightly suggested by others, some search + a finetuned Q&A system is what you need. I've been building a library that makes solving tasks like conversational Q&A easy. https://github.com/backprop-ai/backprop https://github.com/backprop-ai/backprop If you need some pointers, we have example notebooks and I'm happy to chat about it as well. Would love to know if it helps!
- choxi 6y agoIs there anything a non-AI researcher can do to help support this project? Is there a way to donate money? Or could a software engineer help with testing, tooling, or other kinds of infrastructure? I was really excited about OpenAI's original plan and still believe that an open source solution is the best way to prevent the potential negative impacts AI might have on society. I can sort of appreciate why OpenAI went the route of going private and trying to monetize their work instead, it might prevent people from using their work nefariously and will probably provide them with way more capital to continue their efforts. But, I trust humanity as a collective more than any particular group of people in the long run. I'm sure there are many others like me who would be eager to help out if they could. Edit: EleutherAI has a whole page on their site about how others can contribute: https://www.eleuther.ai/get-involved/ https://www.eleuther.ai/get-involved/. I didn't see anything about accepting donations though, if anyone involved with the project was interested in setting up a crowdfunding account somewhere I'd be eager to donate.
- zmix 6y agoYou may indirectly support the project by supporting the host, that hosts their data, https://the-eye.eu https://the-eye.eu Right on the front they write: > Hey there fans! We are currently looking for help funding large storage upgrades, > if you want to help us serve more data see our donation options (crypto, etc) > Thanks for reading, happy downloading!
- stellaathena 6y agoThe Eye has been a phenomenal partner and enables a lot of what we do. In addition to providing terabytes of storage for free, they also help us out with CPU from time to time.
- dannyw 6y agoThe Eye stores amazing and important archives. Drives the data hoarding community.
- punnerud 6y agoIndirectly they say you can donate money, in the form of computation that can be rented: “As an independent organization, we are dependent upon donations for our computing costs. We are always interested in speaking with people who can donate compute times.”
- loxias 6y agoI'd love to know the minimum hardware requirements to run something like this locally.
- sillysaurusx 6y agoPlease test their models before you take it at face value. Eleuther has a history of claiming to replicate projects when they haven't. For example, they shipped a DALL-E repo a few days after OpenAI announced it (https://twitter.com/theshawwn/status/1348017515897659392 https://twitter.com/theshawwn/status/1348017515897659392) which was broken, and they've walked back their GPT-3 replication claims to replicating 1.5B due to the fact that their architecture doesn't scale. As far as I can tell, they're generating a large amount of hype with grandiose claims that they can't deliver on. All I care about is whether you like their models and actually use them in practice. If you do, please let me know and I'll pipe down. But so far, I haven't heard of anyone who uses anything they've produced, and that worries me. Has anyone? One specific claim they made: https://twitter.com/BlancheMinerva/status/1347276975547809809 https://twitter.com/BlancheMinerva/status/134727697554780980... "DALL-E is quite straight forward and already coded. We just need data to train it." No, DALL-E is neither straightforward nor was it successfully coded, especially back on January 7th. Anyway, carry on. I really don't like speaking badly of AI projects, and I hope that they succeed. The model release today is a good step forward, assuming it works. But it might be better to have the expectation of "the models don't work" until proven otherwise. I'd also like to point out that there are some capable people doing work at Eleuther. Sid in particular is one of the best TPU hackers in the scene. I just wish they would scale down their claims, release more models, and not claim that they've done X until actually doing X. For example, the readme says they have "the ability to scale up to full GPT3 sizes (and possibly more!), using the mesh-tensorflow library," which they don't.
- natch 6y agoGood perspective but I would like to hear the response from the developers before concluding too much. This is not meant as a goad to you, but more just as more info for everyone, my understanding is it is an open source community of like minded people type project (as opposed to a bigco) and actively solicits contributions (by which I mean code and data) so anyone seeing room for improvement is welcome to step in from what I can tell. I did find your comment helpful and informative; just adding another angle here.
- 6y ago
- ignoranceprior 6y agoAre there any output samples? Would like to get a sense of how close this is to GPT-2 vs. GPT-3 levels of coherency.
- aeroscripts 6y agoI've been having pretty good luck generating all sorts of stuff. There's a large chunk of text generated with the default unicorns prompt here: https://cdn.discordapp.com/attachments/823393531305525288/823399013579358258/message.txt https://cdn.discordapp.com/attachments/823393531305525288/82... It seems really good at generating code too prompt: class ConvBlock(nn.Module): def __init__(self, channels, stride=1): super(ConvBlock, self).__init__() self.convolution = nn.Conv2d(channels//stride, channels, 3, stride=stride, padding=1) self.normalization = nn.BatchNorm2d(channels) #self.activation = nn.LeakyReLU() def forward(self, x): output: x = # x = self.convolution(x) # x = self.normalization(x) # x = self.activation(x) x = self.convolution(x) x = self.normalization(x) x = self.activation(x) return x class Resblock(nn.Module): def __init__(self, stride=1): super(Resblock, self).__init__() self.res = nn.Sequential( nn.Conv2d(9, 9, 3, stride=stride, padding=1), nn.BatchNorm2d(9), nn.LeakyReLU(0.2) ) self.output = nn.MaxPool2d(2) def forward(self, x): return self.res(x) Also, some "tongue twisters": Big ugly stick. Bowl full of chicken peas. New York is universally inadvisable. Long-eared seagull. This morning I have a tumor in me. A dog bark. Invisible dog hides. A crow blind. A cat can’t see if it had a billion eyes. Bumblebee. Sheep herder, sheep herder. A fawns abducts. Two black birds are trapped. Bottle on her finger. Elephant sees another elephant. Bull. Mice in a box in the library. A church swelter. The door of a hotel opens. Bosnian honey melons. Grapes in excess. Cat is on the loose. Soil is shoveled into a glass jar
- krick 6y agoAre there some truly objective benchmarks to compare this to GPT 2/3?
- clircle 6y agoI think this is an important problem. With logistic regression or deep learning, at least one can compare (out of sample) calibration curves or discrimination measures. With a language model, what can we do?
- bigpumpkin 6y agoperplexity score against a corpus such as wikipedia? Basically how well the model predicts the next word.
- rkimb 6y agoThis is a good start, but given the breadth of applications this would hardly give us enough to compare, as the goal of these models isn't to simply recite Wikipedia articles. What about language translation? Content summarization? Code generation? Turing test performance?
- stellaathena 6y agoBoth models were trained on Wikipedia, so that's a particularly bad choice. But yes, in practice this is what people tend to do. Take results with a very large grain of salt though, as the domain of the prompts you feed it make a huge difference.
- ipsum2 6y agoyes, see GLUE or superGLUE benchmarks. It assumes the answers have not been scraped and included in the dataset though.
- flemhans 6y agoSo I would want to include a big corpus like GPT-3 or this newfangled "Neo" thing but still have it trained to respond to our own customers based on 200k email passages. How to create a hybrid?
- stellaathena 6y ago200k emails is not enough to train a model from scratch. If you check out the google colab file in the GPT-Neo repository, it talks about how to fine-tune the model on data which is what you want to do
- minimaxir 6y agoYou fine-tune an existing pretrained model on your proprietary dataset.
- jalammar 6y agoI wouldn't trust any model to generate text for customers yet. Not even the largest GPT3. There are no guarantees on what they will output and could be damaging to your business. You're better off either: 1- Defining common "intents" that a lot of customer queries are categorized into, and having a model map the incoming message to the appropriate canned response. Look at Rasa, for an example of this. 2- if you insist on generating the text, have it be a recommendation to a human agent that either chooses to send it or writes their own response.
- flemhans 6y agoThanks for the advice.
- kristoo 6y agoYou might get some really promising results with finetuning. If anything, you could build writing assistance that almost automates responses. I've been co-authoring a library that lets you finetune such models in a single line of code. https://github.com/backprop-ai/backprop https://github.com/backprop-ai/backprop In specific the text generation finetuning example should be what you are looking for: https://github.com/backprop-ai/backprop/blob/main/examples/Finetuning_GettingStarted.ipynb https://github.com/backprop-ai/backprop/blob/main/examples/F... Hope this helps, happy to chat more about it. Pretty curious about the results.
- godmode2019 6y agoFollowing
- Bostonian 6y agoIs something with billions of parameters actually a "model"? I guess the answer is yes if the data set is even larger than that?
- PufPufPuf 6y agoDid anyone manage to successfully run inference in the provided Google Colab (https://colab.research.google.com/github/EleutherAI/GPTNeo/blob/master/GPTNeo_example_notebook.ipynb https://colab.research.google.com/github/EleutherAI/GPTNeo/b...)? I can run training, but can't manage to make the inference (even from a pre-trained model) work.
- MasterScrat 6y agoSame here. I managed to make it "work" in the sense that it wouldn't crash during inference, but then it generated gibberish. Has anyone managed to make it work reliably?
- aeroscripts 6y agoThe problem in my case was "train_steps" in the model json file. Default is 0. The notebook sets it to 401000 which works.
- stellaathena 6y agoHi! Thanks for trying it out. There was a bug that should now be fixed. When I run the example unicorn prompt I get the follow. Don't hesitate to open an issue if you're still having trouble. "In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. Bebek Uranzoglu, another member of the research team from the University of British Columbia, was working on a project the Latino-Canadian rodeo competition equipos to document a rare and remarkable ecosystem in the Andes Mountains. His curiosity was piqued when he spotted an adolescent herd of about 10 unicorns foraging in a forest near the valley of the Jumbo Flu Group. The unicorns — whose numbers once swelled to 46,000 — were perched on the forest floor and watched the researchers work. Urizoglu grew excited when he spotted another group that seemed to be thriving in an area below the herd. The team hoped the apparent population growth would indicate a human presence. But when a team of researchers set up a camera trap, they were surprised to find the unicorns in the first place, and in a forest near a lake — in fact the forest was almost entirely made up of the animals. Despite their own magical presence, the team could not see the herd was populated by humans. “The whole place almost smelled like animals,” says Bebek. “We were never able to find human footprints at any of the points we stood at. The trees were so large, you wouldn’t have been able to walk 40 meters through them. We assumed that the truth of the matter was, ‘Well the deer didn’t like this forest at all.’”
- scottisbrave84 6y agoAmen
- vincentmarle 6y agoDoes anyone know if there's a hosted version of this kind of GPT model somewhere? All I want to do is just call a GPT-2 API and get a response back, I'm not interested in setting up the entire infrastructure by myself.
- jalammar 6y agoHugging Face has that service https://huggingface.co/pricing https://huggingface.co/pricing
- kristoo 6y agoWe have exposed the large GPT 2 model as an API. https://backprop.co https://backprop.co Anyone can access it and our pricing is usage based. If you use less than 1000 seconds of inference a month then it's completely free.
- vincentmarle 6y agoGreat will try this!
- FeepingCreature 6y agoIs there something like chattingtransformer ( https://pypi.org/project/chattingtransformer/ https://pypi.org/project/chattingtransformer/ ) for gpt-neo? Ie. a trivial way to get text completion on a sample with sane defaults from the commandline. edit: Oh, I see the "generating text" section. Any way to run it on CPU, even if it takes an hour?
- victor9000 6y agoStella mentioned elsewhere in this thread that HuggingFace is adding support for the Eleuther model, so text generation should become trivial once this work is complete.
- stellaathena 6y agoWe’ve added a table with some evaluation scores to the GitHub repo, and you can see a comparison between our scores, GPT-2, and GPT-3 here: https://twitter.com/BlancheMinerva/status/1373991896616427522?s=20 https://twitter.com/BlancheMinerva/status/137399189661642752... tl;dr we are doing pretty much exactly as well as we expected on LAMBADA and WikiText. Results on more sophisticated tasks will take some time, but HuggingFace is currently working on implementing our model in the transformers library and when they do so we can easily run a lot of analyses very quickly. We actually built an evaluation suite that integrates with HF, but interfacing with the MTF code that GPT-Neo was written in was too much of a pain in the ass because Mesh TensorFlow is the worst. https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness