5 ms·

I'm probably not qualified to say this so take it with a grain of salt, but I'd be surprised if OpenAI is concerned about a lone hobbyist, and is much more conc

I'm probably not qualified to say this so take it with a grain of salt, but I'd be surprised if OpenAI is concerned about a lone hobbyist, and is much more concerned about nation states with access to much more hardware, software, and experts who can do things that no single person or small group of people could do.

comex 7y ago
Sounds like exactly the kind of actor that would have the budget and expertise to replicate GPT-2 itself, rather than having to rely on a pretrained model.
mattnewport 7y ago
If you have the resources of a nation state then you already have the resources to push your agenda through traditional media. It's not clear what benefit this type of AI generated content would have over the traditional approach.
- rm_-rf_slash 7y ago
  Asymmetric information warfare. Flood every channel with misleading and contradicting information beyond the capacity for fact-checking. Make entirely fake social media threads to steer perceptions towards a predetermined direction. Create exaggerated stereotypes of both sides of a debate to inflame and harden opinions. All of this is already being done right now. The difference is that a robust language model would allow this to be done with far fewer humans involved. Fewer people to raise their hands about how far down the ethical rabbit hole they are willing to go. And a single AI model could react to news much more quickly than a team of people who have to sleep at some point.
  - whatshisface 7y ago
    That's not asymmetric, any nation could do it. Also, the Reddit farms wouldn't have to sleep, because the writers could work in shifts.
    - cardiffspaceman 7y ago
      The human writers are constrained by what they think they can get away with. The automated writers don't even work that way. You can see from the examples used in discussions here, the more limited the domain of discourse the more convincing the samples are (except the hello world one that looked like a mash-up of every programming tutorial ever written). I think this thing could do a great job making fake announcements to air travelers about flight delays. I think the synthetic texts are already convincing enough that the natural desire to try to impart meaning to writings/utterances will go a long way to convincing forum user "victims" that something is being said that needs to be thought about. And here in this discussion, we know that the fakes are fakes (except for one comment which is unlabeled and I'm not sure about), in an aggression against a forum the fakes won't be labeled.
      - rm_-rf_slash 7y ago
        The less said, the easier it is for a language model to approximate a useful thread comment for the purposes of mass propaganda. “These graphics look terrible. I will never play this game.” “$Candidate is a corporate shill and everyone knows it.” “I can’t wait for $Artist’s next album! They’re sooooo good!” Doesn’t need to be an extensive, well thought out comment to drive thought and discourse. GPT2 is good enough for that.
Bartweiss 7y ago
This is certainly a reasonable idea, and the OpenAI release statement did include a risk assessment attempting to determine who would be able to replicate their work. It's sort of hard for me to understand how this lead to the decision they made, though. OpenAI's original aim was democratizing access to AI tools so they wouldn't be abused by a handful of players, but releasing a scaled-down model seemed to primarily limit its utility for hobbyists. Assembling and training on a larger corpus is precisely the sort of step that's out of reach for individuals, but extremely approachable for governments or corporations with lots of computing power and expert-hours. That said, OpenAI also noted that their staged-release model was something of an experiment, so it's not entirely clear to me how much danger they thought GPT-2 actually posed. It seems to have been a useful experiment in gradual release and collaboration with other researchers, and a test of 'inoculating' people via a weak model. GPT-2 isn't a great tool for tricking people or disrupting conversations, but publicizing awareness of AI text generation before turning it loose has probably further decreased that risk.
ryanmercer 7y ago
You don't even need a nation state. 1 person could sufficiently weaponize this against multiple individuals by using it to mimic their writing style by training it on their social media/blog/articles they've written and then using it to start various accounts online as the target and slowly building up followers with content and at some point switching to saying highly controversial, or downright offensive, things to immediately cause people to change their opinion of the individual. Racist/sexist/etc content, allegations against someone else, fake content about a company/product/service that could cause varying levels of financial damages to the company etc. It also is a new tool for blackhat SEO, instead of paying people to write crappy content, or running relevant articles through any number of commercially available software packages that will rewrite the article enough to fool search engines as being unique for weeks or even years, you can simply train the software on stacks of books of material about a given subject and have it churn out semi-intelligible-to-humans articles about the subject that could allow you to build a rather large amount of content to manipulate search engine rankings or even to just create contextual/affiliate ad farms. If you have the software churn out smaller sets of content, then quickly edit it yourself, a single individual could churn out 10-20,000 words of content a day with banners/images/contextual ads inserted. A small blackhat SEO team in a country like Bangladesh or India could offer prices far cheaper than competition using more traditional methods, have articles that are more 'natural' in the customer's language, and be of a likely higher quality in a much higher volume. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states?

comex 7y ago

Sounds like exactly the kind of actor that would have the budget and expertise to replicate GPT-2 itself, rather than having to rely on a pretrained model.

mattnewport 7y ago

If you have the resources of a nation state then you already have the resources to push your agenda through traditional media. It's not clear what benefit this type of AI generated content would have over the traditional approach.

rm_-rf_slash 7y ago
Asymmetric information warfare. Flood every channel with misleading and contradicting information beyond the capacity for fact-checking. Make entirely fake social media threads to steer perceptions towards a predetermined direction. Create exaggerated stereotypes of both sides of a debate to inflame and harden opinions. All of this is already being done right now. The difference is that a robust language model would allow this to be done with far fewer humans involved. Fewer people to raise their hands about how far down the ethical rabbit hole they are willing to go. And a single AI model could react to news much more quickly than a team of people who have to sleep at some point.
- whatshisface 7y ago
  That's not asymmetric, any nation could do it. Also, the Reddit farms wouldn't have to sleep, because the writers could work in shifts.
  - cardiffspaceman 7y ago
    The human writers are constrained by what they think they can get away with. The automated writers don't even work that way. You can see from the examples used in discussions here, the more limited the domain of discourse the more convincing the samples are (except the hello world one that looked like a mash-up of every programming tutorial ever written). I think this thing could do a great job making fake announcements to air travelers about flight delays. I think the synthetic texts are already convincing enough that the natural desire to try to impart meaning to writings/utterances will go a long way to convincing forum user "victims" that something is being said that needs to be thought about. And here in this discussion, we know that the fakes are fakes (except for one comment which is unlabeled and I'm not sure about), in an aggression against a forum the fakes won't be labeled.
    - rm_-rf_slash 7y ago
      The less said, the easier it is for a language model to approximate a useful thread comment for the purposes of mass propaganda. “These graphics look terrible. I will never play this game.” “$Candidate is a corporate shill and everyone knows it.” “I can’t wait for $Artist’s next album! They’re sooooo good!” Doesn’t need to be an extensive, well thought out comment to drive thought and discourse. GPT2 is good enough for that.

Bartweiss 7y ago

This is certainly a reasonable idea, and the OpenAI release statement did include a risk assessment attempting to determine who would be able to replicate their work. It's sort of hard for me to understand how this lead to the decision they made, though. OpenAI's original aim was democratizing access to AI tools so they wouldn't be abused by a handful of players, but releasing a scaled-down model seemed to primarily limit its utility for hobbyists. Assembling and training on a larger corpus is precisely the sort of step that's out of reach for individuals, but extremely approachable for governments or corporations with lots of computing power and expert-hours. That said, OpenAI also noted that their staged-release model was something of an experiment, so it's not entirely clear to me how much danger they thought GPT-2 actually posed. It seems to have been a useful experiment in gradual release and collaboration with other researchers, and a test of 'inoculating' people via a weak model. GPT-2 isn't a great tool for tricking people or disrupting conversations, but publicizing awareness of AI text generation before turning it loose has probably further decreased that risk.

ryanmercer 7y ago

You don't even need a nation state. 1 person could sufficiently weaponize this against multiple individuals by using it to mimic their writing style by training it on their social media/blog/articles they've written and then using it to start various accounts online as the target and slowly building up followers with content and at some point switching to saying highly controversial, or downright offensive, things to immediately cause people to change their opinion of the individual. Racist/sexist/etc content, allegations against someone else, fake content about a company/product/service that could cause varying levels of financial damages to the company etc. It also is a new tool for blackhat SEO, instead of paying people to write crappy content, or running relevant articles through any number of commercially available software packages that will rewrite the article enough to fool search engines as being unique for weeks or even years, you can simply train the software on stacks of books of material about a given subject and have it churn out semi-intelligible-to-humans articles about the subject that could allow you to build a rather large amount of content to manipulate search engine rankings or even to just create contextual/affiliate ad farms. If you have the software churn out smaller sets of content, then quickly edit it yourself, a single individual could churn out 10-20,000 words of content a day with banners/images/contextual ads inserted. A small blackhat SEO team in a country like Bangladesh or India could offer prices far cheaper than competition using more traditional methods, have articles that are more 'natural' in the customer's language, and be of a likely higher quality in a much higher volume. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states?