5 ms·

How to Make Custom AI-Generated Text with GPT-2

95 points by Impossible 7y ago

tyingq 7y ago
I'm curious what Google might be doing to combat content/link farms created for SEO purposes using this.
- conse_lad 7y ago
  There's a tool for that: http://gltr.io/dist/index.html http://gltr.io/dist/index.html "Each text is analyzed by how likely each word would be the predicted word given the context to the left. If the actual used word would be in the Top 10 predicted words the background is colored green, for Top 100 in yellow, Top 1000 red, otherwise violet."
  - Viliam1234 7y ago
    What if I find out what is the supposedly "non-fake" distribution of green/yellow/red/violet words, and generate my text accordingly? Generally, whenever your detection strategy is "a spam generator would never do X", I simply update my spam generator to do X. (Note that "X" must be something that is relatively easy to calculate, not things like "this actually makes sense", because it must be something your detector can recognize.) Also, if Google suddenly started penalizing all text that doesn't "seem natural", there would be tons of false positives: languages other than English, jargon-heavy websites, dyslectic people, etc.
    - Avamander 7y ago
      Non-native speakers writing text also comes to mind.
  - minimaxir 7y ago
    Since we're on the topic, this detector assumes generated text is from the default GPT-2 models; it won't work as a well on finetuned GPT-2.
  - jcims 7y ago
    I need something like this for audit logs.
amrrs 7y ago
minimaxir's gpt-2-simple is a very nice work to get anyone start with Text Generation from the models released by OpenAI. I often wonder how can one keep up so much with the fast-paced NLP and produce things that abstracts the pain and exposes simple functions for developers to on it. Hugging Face is another such company making massive impact among non-NLP developers to use such resources. Kudos to these guys!
- minimaxir 7y ago
  tl;dr I just keep an eye on GPT-2-related news on Twitter. I have no advance knowledge of any GPT-2 related happenings, which has become inconvenient as OpenAI likes to release things when I am on vacation. :P
yowlingcat 7y ago
Very nice. I'm going to have some fun taking a crack using this with some of the funnier stuff I've seen on twitter.
bigmit37 7y ago
Thanks for sharing. I have been hearing a lot of buzzword regarding this archtecture, so I checked out the blog. I quickly glanced through it but will do a deeper dive later on. The outputted paragraphs don’t seem to realistic. However you point out that the text needs to be longer for a more a natural output. I am genuinely curious how relalistic it can get.
- Avamander 7y ago
  Check out https://www.reddit.com/r/SubSimulatorGPT2Meta/top/ https://www.reddit.com/r/SubSimulatorGPT2Meta/top/ In my personal experience I get fooled by the bots far too often, it's actually really scary, I can imagine it being used nefariously to push a specific political agenda, spam places, do SEO fraud etc. etc.
minimaxir 7y ago
If you want examples of AI-generated Hacker News titles (made pretty much the same way as the Reddit example in this post), check out this repo: https://github.com/minimaxir/hacker-news-gpt-2 https://github.com/minimaxir/hacker-news-gpt-2
onemoresoop 7y ago
With the risk of being downvoted, what’s the point of ai generating text knowing that it will never understand it like we do and having in mind that poor non AI generated text is already being used for mostly nefarious reasons? As a problem I understand the fascination with it, human languages are extremely complex things and mimmicking it has been proven complicated, but beyond that, what is a practical purpose for this?
- minimaxir 7y ago
  Because it's fun. Which is practical enough! I do have ideas for more practical content generation but those do not currently have plans to be made public.
  - onemoresoop 7y ago
    Enjoy if it is fun. I simply hope you won’t use it or enable others to use it for any nefarious means. I once entertained the thought that one day we may be able to ai generate novels. And the more i thought about it the more pointless it seemed.
- ganeshkrishnan 7y ago
  We tried this for e-commerce products description. The model asked us to "dip the laundry nuts in chocolate and feed it to the kids" They are grammatically correct but logically nonsense. The examples shown are also curated from thousands of generated text. The only use for these is the seo landfill to get in top of Google
- dlphn___xyz 7y ago
  the bulk of ML research is content generation which will eventually be used for nefarious reasons
- darawk 7y ago
  > With the risk of being downvoted, what’s the point of ai generating text knowing that it will never understand it like we do In part because we're trying to probe what it even means to "understand" something. If an AI is capable of carrying on a cogent conversation with you, does it understand what it's saying? How do you know?
georgewsinger 7y ago
Wolfram has a GPT-2 interface that can be beckoned with two lines of code: https://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18e-353eff4d0b33/1143236222823602259835271068650786074616893186089.png https://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18...

95 points by Impossible 7y ago

tyingq 7y ago

I'm curious what Google might be doing to combat content/link farms created for SEO purposes using this.

conse_lad 7y ago
There's a tool for that: http://gltr.io/dist/index.html http://gltr.io/dist/index.html "Each text is analyzed by how likely each word would be the predicted word given the context to the left. If the actual used word would be in the Top 10 predicted words the background is colored green, for Top 100 in yellow, Top 1000 red, otherwise violet."
- Viliam1234 7y ago
  What if I find out what is the supposedly "non-fake" distribution of green/yellow/red/violet words, and generate my text accordingly? Generally, whenever your detection strategy is "a spam generator would never do X", I simply update my spam generator to do X. (Note that "X" must be something that is relatively easy to calculate, not things like "this actually makes sense", because it must be something your detector can recognize.) Also, if Google suddenly started penalizing all text that doesn't "seem natural", there would be tons of false positives: languages other than English, jargon-heavy websites, dyslectic people, etc.
  - Avamander 7y ago
    Non-native speakers writing text also comes to mind.
- minimaxir 7y ago
  Since we're on the topic, this detector assumes generated text is from the default GPT-2 models; it won't work as a well on finetuned GPT-2.
- jcims 7y ago
  I need something like this for audit logs.

amrrs 7y ago

minimaxir's gpt-2-simple is a very nice work to get anyone start with Text Generation from the models released by OpenAI. I often wonder how can one keep up so much with the fast-paced NLP and produce things that abstracts the pain and exposes simple functions for developers to on it. Hugging Face is another such company making massive impact among non-NLP developers to use such resources. Kudos to these guys!

minimaxir 7y ago
tl;dr I just keep an eye on GPT-2-related news on Twitter. I have no advance knowledge of any GPT-2 related happenings, which has become inconvenient as OpenAI likes to release things when I am on vacation. :P

yowlingcat 7y ago

Very nice. I'm going to have some fun taking a crack using this with some of the funnier stuff I've seen on twitter.

bigmit37 7y ago

Thanks for sharing. I have been hearing a lot of buzzword regarding this archtecture, so I checked out the blog. I quickly glanced through it but will do a deeper dive later on. The outputted paragraphs don’t seem to realistic. However you point out that the text needs to be longer for a more a natural output. I am genuinely curious how relalistic it can get.

Avamander 7y ago
Check out https://www.reddit.com/r/SubSimulatorGPT2Meta/top/ https://www.reddit.com/r/SubSimulatorGPT2Meta/top/ In my personal experience I get fooled by the bots far too often, it's actually really scary, I can imagine it being used nefariously to push a specific political agenda, spam places, do SEO fraud etc. etc.

minimaxir 7y ago

If you want examples of AI-generated Hacker News titles (made pretty much the same way as the Reddit example in this post), check out this repo: https://github.com/minimaxir/hacker-news-gpt-2 https://github.com/minimaxir/hacker-news-gpt-2

onemoresoop 7y ago

With the risk of being downvoted, what’s the point of ai generating text knowing that it will never understand it like we do and having in mind that poor non AI generated text is already being used for mostly nefarious reasons? As a problem I understand the fascination with it, human languages are extremely complex things and mimmicking it has been proven complicated, but beyond that, what is a practical purpose for this?

minimaxir 7y ago
Because it's fun. Which is practical enough! I do have ideas for more practical content generation but those do not currently have plans to be made public.
- onemoresoop 7y ago
  Enjoy if it is fun. I simply hope you won’t use it or enable others to use it for any nefarious means. I once entertained the thought that one day we may be able to ai generate novels. And the more i thought about it the more pointless it seemed.
ganeshkrishnan 7y ago
We tried this for e-commerce products description. The model asked us to "dip the laundry nuts in chocolate and feed it to the kids" They are grammatically correct but logically nonsense. The examples shown are also curated from thousands of generated text. The only use for these is the seo landfill to get in top of Google
dlphn___xyz 7y ago
the bulk of ML research is content generation which will eventually be used for nefarious reasons
darawk 7y ago
> With the risk of being downvoted, what’s the point of ai generating text knowing that it will never understand it like we do In part because we're trying to probe what it even means to "understand" something. If an AI is capable of carrying on a cogent conversation with you, does it understand what it's saying? How do you know?

georgewsinger 7y ago

Wolfram has a GPT-2 interface that can be beckoned with two lines of code: https://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18e-353eff4d0b33/1143236222823602259835271068650786074616893186089.png https://www.wolframcloud.com/obj/user-900a994f-78ab-4931-b18...