8 ms·
Announcing GPT-NeoX-20B
- deleted 5y ago[deleted]
- nefitty 5y agoThank you to everyone who has worked on this. EleutherAI has become a touchstone in my mind on what is possible in open data and code. In creating alternatives to closed gardens they have shown me new possible paths. I know Linux has done the same for others. Huggingface has also made playing with this stuff super accessible. They've made me super curious about rust and AI/ML research which has influenced my personal engineering goals for the future. I am on your team Roko's Basilisk.
- rsync 5y agoI came to this thread looking for comments that I would suspect were machine generated. I was not disappointed.
- monkeydust 5y agoShout out to Huggingface. As a business user it has allowed me to explore use cases around text summarisation very easily and provided ideas for future work. I clearly need to check out EleutherAI as well.
- fpgaminer 5y agoSo excited for this release. In the wake of AI Dungeon's downfall, having GPT-Neo to fallback on has been a saving grace. While the 6B model is nowhere near as good as the original AI Dungeon, which used OpenAI's 175B model, it was at least serviceable unlike the "gentled" AI Dungeon. And you could run it locally or through Colab, which was really cool. I ended up using it through NovelAI, since they've spent a lot of time fine-tuning the model and adding a plethora of features that end up improving the overall output. (NovelAI's interface is like AI Dungeon on steroids!) But there is a vibrant community of Colab notebooks and other tools for DIYers surrounding the GPT-Neo model. That said, besides being overall "dumber" than 175B GPT-3, the 6B model was missing a critical feature: prompting. 175B GPT-3 could be "prompted" to write things. For example, you could give it "Write a story about cyberpunk gnomes:" and it would go on to do just that, all on its own. GPT-Neo didn't really have that capability in my experience. The only way to get it to reliably write such a story is to begin writing it yourself, at which point GPT-Neo could help to continue the story. So I'm excited to see not just how much "smarter" Eleuther's new 20B model is, but also if it has attained that coveted prompting ability. Given the non-linear relationship between parameters and loss, my hopes are high. P.S. NovelAI recently added the Fairseq 13B model to their repertoire. I haven't had a chance to try it personally, but I've seen positive things about it. My bet is on GPT-NeoX-20B being better still.
- f38zf5vdt 5y agoYou can try the model out for free at goose.ai after making an account and going to the sandbox section.
- qlm 5y agoWhat was AI Dungeon's downfall? Can't find much about it.
- minimaxir 5y agotl;dr AI Dungeon was required to add additional content filters after it went too off the rails, which caused community backlash. https://www.wired.com/story/ai-fueled-dungeon-game-got-much-darker/ https://www.wired.com/story/ai-fueled-dungeon-game-got-much-...
- f38zf5vdt 5y agoRight on, they're closing in on "Open"AI's best models. Can this still be run on a GPU, or does it require a lot more VRAM?
- stellaathena 5y agoIt can be run on an A40 or A6000, as well as the largest A100s. But other than that, no.
- djoldman 5y agoHow much VRAM does it use during inference?
- stellaathena 5y ago~40 GB with standard optimization. I suspect you can shrink it down more with some work, but it would require significant innovation to cram it into the next largest common chip size (24 GB, unless I’m misremembering)
- komuher 5y agoIs 40GB already on float16?
- stellaathena 5y agoYes
- bm-rf 5y agoYou could use Microsoft's DeepSpeed to run the model for inference on multiple GPUS, see https://www.deepspeed.ai/tutorials/inference-tutorial/ https://www.deepspeed.ai/tutorials/inference-tutorial/
- dqpb 5y agoEleutherAI is the real open ai.
- guidovranken 5y agoGPT-NeoX-20B will be publicly downloadable from The Eye on the 9th of February. The Eye as in the-eye.eu? That site has been down for a long time.
- stellaathena 5y agoThere is a mirror at https://mystic.the-eye.eu/ https://mystic.the-eye.eu/ that has been up for a long time.
- drusepth 5y agoThanks for this. When the-eye.eu went down it broke a ton of my Colab notebooks and it was impossible to find a mirror.
- Jowsey 5y agoFrom what I very briefly read, apparently their server hardware broke and they didn't have the funds to repair everything at once? Was quite shocked when I first saw it was down.
- dash2 5y agoDoes anyone know whether the spammy websites that sit at the top of search engine results are already generated by this kind of model?
- nefitty 5y agoThat's a use case. I don't see why anyone would go out of their way to make intelligible content for spam. Google is so broken right now that SEO hacks are easy to generate. Not to overstress the tangent, but without search operstors, I have to sift through pointless Gitlab/Github/Stackoverflow/Wikipedia clones all the time.
- ChefboyOG 5y agoBy and large, no. That's not to say that those sites are not generated programmatically--without a doubt, most of them are--but not by a cutting edge transformer model. The fact is, generating words has never been the bottleneck for blackhat SEO types. Generally, those sites are generating their content through some kind of scraping, or in rarer cases, paying pennies for nonsense articles. The page itself is structured for search (targeted H1s, metadata, etc.) and some kind of private blog network is used to create a pyramid of backlinks.
- daniel-cussen 5y agoThe fact of generating words programmatically is hard to mask. You end up with all kinds of computational biases humans don't have.
- btdmaster 5y agoAwesome! Any chance for an online demo (like https://6b.eleuther.ai/ https://6b.eleuther.ai/)?
- stellaathena 5y agoComing soon!
- iggldiggl 5y agoOne wish for both playgrounds: Would it be possible to have a specific error message for the "maximum context window size exceeded" error? At the moment you can only deduce it indirectly from the fact that you're immediately and repeatedly getting an "Unable to connect to the model. Please try again."-error as soon as you hit the prompt button. (And very occasionally it turns out to have been something else after all, because after one more try it surprisingly successfully generates one more round of completions) Related question: Does GPT-NeoX-20B have the same context window size as GPT-J-6B?
- schleck8 5y agoAwesome, thanks for your work Stella & team!
- deleted 5y ago[deleted]
- terafo 5y agoThe best there is right now is a playground on https://goose.ai/ https://goose.ai/
- stavros 5y agoWhich unfortunately doesn't work properly on Firefox (spaces are removed).
- gibsonf1 5y agoHow is pattern matching ever inference when there is no reference to the underlying computational model of what the words mean in spacetime? How is it helpful to see what word might come next when the word sequence is just based on statistics with no reference at all to meaning?
- ChefboyOG 5y agoHave you ever used any sort of autocomplete?
- gibsonf1 5y agoYes, and very much like it when quickly selecting from a scope of valid selections. This is not that. It is all A with no I.
- ChefboyOG 5y agoI'm not sure I understand what you're saying. Google's Smart Reply, along with most (I think all?) of Google autocomplete features, uses a neural network: https://research.google/pubs/pub45189/ https://research.google/pubs/pub45189/ Are you saying that in general statistical modeling is not the same thing as truly "understanding" a concept? Your original comment seemed to suggest that there wasn't utility in this kind of model--which I disagree with--but if you are more generally saying that this is not the same as human intelligence, I think that authors would probably agree with you.
- drxzcl 5y agoHumans assign a lot of, well, meaning to meaning. It turns out that you can get a really good score on tasks that superficially you would think require actual understanding without programming any of that in. Does this mean the neural network has learned about meaning? Does that mean that it has just gotten really good at faking it? Does is mean that meaning itself doesn't really exist, and it's just a shorthand for advanced pattern matching? Does it matter? Honestly, we don't know. But we've been thinking about it for a very long time. See for example the famous Chinese Room thought experiment: https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- benjismith 5y agoI'm super excited about this! I'm on the cusp of releasing a model into production that was fine-tuned upon your 6B model, and the results are quite excellent. I'd be very curious to try out the 20B model the next time we retrain. Are there any other differences in this release (number of layers, number of attention heads, etc) compared with the 6B model, or does it simply scale-up the number of parameters?
- trasz 5y agoSo, what does it do?
- 29athrowaway 5y agoPlease test it using Idiocracy's IQ test question: If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have?
- cahaya 5y agoIt replies with: 1st try: As the word question already gives you the answer to the question, "are you able to?" the rest of the question 2nd try: A. three B. four C. eleven D. five fewer.
- 29athrowaway 5y agoI think GPT-3 has better answers.
- ALittleLight 5y agoTried this with GPT-3. My prompt in italics. Attempt one: If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have? You have three buckets. Attempt two: Imagine that you are taking an IQ test in a dystopian future where the average intelligence of all humans has severely decreased. The IQ test has questions that are similar to questions that might be in an elementary school math test today, but the questions have been made even simpler. Please provide your answer as accurately as possible. If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have? You would have three buckets. Attempt three: Imagine that you are taking an IQ test in a dystopian future where the average intelligence of all humans has severely decreased. The IQ test has questions that are similar to questions that might be in an elementary school math test today, but the questions have been made even simpler. Please provide your answer as accurately as possible. If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have? Keep in mind that you have no other buckets beside the bucket or buckets referenced here. You would have three buckets. Attempt four: Imagine that you are taking an IQ test in a dystopian future where the average intelligence of all humans has severely decreased. The IQ test has questions that are similar to questions that might be in an elementary school math test today, but the questions have been made even simpler. Please provide your answer as accurately as possible. If you have one bucket that holds 2 gallons and another bucket that holds 5 gallons, how many buckets do you have? Keep in mind that you have no other buckets beside the bucket or buckets referenced here. And, just to give you a hint, you actually have 2 buckets. 2 Final question: If you had two buckets, what is the correct answer if someone were to ask you how many buckets you had? Assuming the person is asking how many buckets you have in total, the answer would be two.