7 ms·
Wow. GPT2 is so, so, so much better than Markov chains. I'm reading these definitions, and the fact that the last few words of the sentence match the first few
by crazygringo 7y ago
Wow. GPT2 is so, so, so much better than Markov chains. I'm reading these definitions, and the fact that the last few words of the sentence match the first few words subject-wise is pretty amazing. Just some random ones:
> denoting or relating to a word (e.g., al-Qadri), the first letter of which is preceded or followed by another letter
> a synthetic compound used in perfumery and cosmetic surgery to improve the appearance of skin tone and irritation
> a type of cookie made with dough, jelly, butter, or chocolate, often filled with extra flour
Pretty impressive. I've never seen fake text so real. (I mean none of these seem to quite make 100% logical sense, but if you were just skimming the sentence nothing would stand out as a red flag.)
- Klathmon 7y agoI always like to point people to /r/SubSimulatorGPT2 [1] as a good example of what GPT2 is able to accomplish. It's a subreddit filled entirely with bots, each user is trained on a specific subreddit's comments matching it's username (so politicsGPT2Bot is trained on comments from the politics subreddit). Go click through a few comment sections and see how mind-bendingly real some comment chains seem. They reply quoting other comments, they generate links (they almost always go to a 404 page, but they look real and are in format that makes me think it's real every time I hover over it) entirely on their own, they have full conversations back and forth, they make jokes, they argue "opinions" (often across multiple comments back and forth keeping the context of which "side" each comment is on), and they vary from single word comments to multi-paragraph comments. Take a look at this thread [2] specifically. The headline is made up, the link it goes to is made up, but the comments look insanely real at first glance. Some of them even seem to be quoting the contents of the article (which again, doesn't exist) in it's comments! If you threw something like 50% "real humans" in the mix, I genuinely don't think I'd be able to pick out the bots on my own. [1] https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/ [2] https://www.reddit.com/r/SubSimulatorGPT2/comments/fzwso5/nra_endorses_biden_for_president/ https://www.reddit.com/r/SubSimulatorGPT2/comments/fzwso5/nr...
- whymauri 7y agoWow, this is so much better than the original /r/subredditsimulator (which was Markov Chains). That's a very fun thread you linked - it's very believable!
- gwern 7y agoOne fun part - we used the inline metadata trick to train a single GPT-2-1.5b to do all the different subreddits. It allows mutual transfer learning and saves an enormous amount of space & complexity compared to training separate models, and it's easy to add in any new subreddits one might want (just define a new keyword prefix and train some more). Not sure that trick is meaningful for Markov chains at all!
- jessehu 7y agoWhat is the inline metadata trick?
- gwern 7y agoIt's an old trick in generative models, I've been using it since 2015: https://www.gwern.net/RNN-metadata https://www.gwern.net/RNN-metadata When you have categorical or other metadata, instead of trying to find some way to hardwire it into the NN by having a special one-hot vector or something, you simply inline it into the dataset itself, as a text prefix, and then let the model figure it out. If it's at all good, like a char-RNN, it'll learn what the metadata is and how to use it. So you get a very easy generic approach to encoding any metadata, which lets you extend it indefinitely without retraining from scratch (reusing models not trained with it in the first place, like OA's GPT-2-1.5b), while still controlling generation. Particularly with GPT-2, you see this used for (among others) Grover and CTRL, in addition to my own poetry/music/SubSim models.
- Diederich 7y agoSome of this stuff is just gold: https://old.reddit.com/r/SubSimulatorGPT2/comments/ggf1mq/i_have_a_cat_it_is_amazing_and_i_wish_i_had_it/fq0a1e1/ https://old.reddit.com/r/SubSimulatorGPT2/comments/ggf1mq/i_...
- minimaxir 7y agoOne of the recurring tropes on Hacker News whenever a text generator (either RNN or GPT-2 based) project is posted, there is inevitably a comment saying "this is indistinguishable from a Markov chain." In this case, it's impossible to do so.
- mycall 7y agoIf someone could make a billion word dictionary of these words, you could get excellent sentence compression rates out of standard English.