4 ms·

So excited for this release. In the wake of AI Dungeon's downfall, having GPT-Neo to fallback on has been a saving grace. While the 6B model is nowhere near a

So excited for this release. In the wake of AI Dungeon's downfall, having GPT-Neo to fallback on has been a saving grace. While the 6B model is nowhere near as good as the original AI Dungeon, which used OpenAI's 175B model, it was at least serviceable unlike the "gentled" AI Dungeon. And you could run it locally or through Colab, which was really cool. I ended up using it through NovelAI, since they've spent a lot of time fine-tuning the model and adding a plethora of features that end up improving the overall output. (NovelAI's interface is like AI Dungeon on steroids!) But there is a vibrant community of Colab notebooks and other tools for DIYers surrounding the GPT-Neo model. That said, besides being overall "dumber" than 175B GPT-3, the 6B model was missing a critical feature: prompting. 175B GPT-3 could be "prompted" to write things. For example, you could give it "Write a story about cyberpunk gnomes:" and it would go on to do just that, all on its own. GPT-Neo didn't really have that capability in my experience. The only way to get it to reliably write such a story is to begin writing it yourself, at which point GPT-Neo could help to continue the story. So I'm excited to see not just how much "smarter" Eleuther's new 20B model is, but also if it has attained that coveted prompting ability. Given the non-linear relationship between parameters and loss, my hopes are high. P.S. NovelAI recently added the Fairseq 13B model to their repertoire. I haven't had a chance to try it personally, but I've seen positive things about it. My bet is on GPT-NeoX-20B being better still.

f38zf5vdt 5y ago
You can try the model out for free at goose.ai after making an account and going to the sandbox section.
qlm 5y ago
What was AI Dungeon's downfall? Can't find much about it.
- minimaxir 5y ago
  tl;dr AI Dungeon was required to add additional content filters after it went too off the rails, which caused community backlash. https://www.wired.com/story/ai-fueled-dungeon-game-got-much-darker/ https://www.wired.com/story/ai-fueled-dungeon-game-got-much-...
  - fpgaminer 5y ago
    It was more than that. They also significantly downgraded the model. I didn't follow the details, but IIUC Dragon used the 175B directly initially, then I think they went down a model size at Open AI's behest. Finally, when Open AI announced pricing, AI Dungeon had to downgrade the model further. But yes, the content filtering got out of hand too. I was initially fine with it, as its proposed intention was to filter out really illegal stuff, like underage content. I rarely hit the filter. But then they tweaked it at some point and I was triggering it constantly on otherwise benign stuff. And they broke features constantly. When I unsubbed the state of AID was broken features, micro-transactions, terrible AI model, and a glitchy, puritanical content filter. The plus side is that it made the puny GPT-Neo model look like a godsend.
    - bitforger 5y ago
      I believe they're currently using AI21's 178B Jumbo model for Dragon. Since they're completely off of OpenAI now, the content filter is much more lax.
    - causi 5y ago
      really illegal stuff, like underage content Wait, isn't this output just text? How is a text AI generating illegal content?
      - NovemberWhiskey 5y ago
        The content may not be illegal to possess, but if it's obscene, then it can be illegal to sell it, produce it with the intention of selling it, transport it, distribute it, and so on.
        capableweb 5y ago
        Could it really? I was under the impression that unless you incite someone to commit crimes (or confess to crimes), the story would be covered under "art" and therefore protected. It's just text after all. Where does the limit for "obscene" go?
benjismith 5y ago
The "prompting" ability you're referring to is called "instruction following", and here are some descriptions of it. https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/ I think the differences are more in the training data used, than in the nature of the model itself. So you could probably train your own instruction-following model on top of this raw 20B model.

f38zf5vdt 5y ago

You can try the model out for free at goose.ai after making an account and going to the sandbox section.

qlm 5y ago

What was AI Dungeon's downfall? Can't find much about it.

minimaxir 5y ago
tl;dr AI Dungeon was required to add additional content filters after it went too off the rails, which caused community backlash. https://www.wired.com/story/ai-fueled-dungeon-game-got-much-darker/ https://www.wired.com/story/ai-fueled-dungeon-game-got-much-...
- fpgaminer 5y ago
  It was more than that. They also significantly downgraded the model. I didn't follow the details, but IIUC Dragon used the 175B directly initially, then I think they went down a model size at Open AI's behest. Finally, when Open AI announced pricing, AI Dungeon had to downgrade the model further. But yes, the content filtering got out of hand too. I was initially fine with it, as its proposed intention was to filter out really illegal stuff, like underage content. I rarely hit the filter. But then they tweaked it at some point and I was triggering it constantly on otherwise benign stuff. And they broke features constantly. When I unsubbed the state of AID was broken features, micro-transactions, terrible AI model, and a glitchy, puritanical content filter. The plus side is that it made the puny GPT-Neo model look like a godsend.
  - bitforger 5y ago
    I believe they're currently using AI21's 178B Jumbo model for Dragon. Since they're completely off of OpenAI now, the content filter is much more lax.
  - causi 5y ago
    really illegal stuff, like underage content Wait, isn't this output just text? How is a text AI generating illegal content?
    - NovemberWhiskey 5y ago
      The content may not be illegal to possess, but if it's obscene, then it can be illegal to sell it, produce it with the intention of selling it, transport it, distribute it, and so on.
      - capableweb 5y ago
        Could it really? I was under the impression that unless you incite someone to commit crimes (or confess to crimes), the story would be covered under "art" and therefore protected. It's just text after all. Where does the limit for "obscene" go?

benjismith 5y ago

The "prompting" ability you're referring to is called "instruction following", and here are some descriptions of it. https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/ I think the differences are more in the training data used, than in the nature of the model itself. So you could probably train your own instruction-following model on top of this raw 20B model.