11 ms·

GPT-2 is not as dangerous as OpenAI thought it might be

161 points by luu 7y ago

codezero 7y ago
I'm probably not qualified to say this so take it with a grain of salt, but I'd be surprised if OpenAI is concerned about a lone hobbyist, and is much more concerned about nation states with access to much more hardware, software, and experts who can do things that no single person or small group of people could do.
- comex 7y ago
  Sounds like exactly the kind of actor that would have the budget and expertise to replicate GPT-2 itself, rather than having to rely on a pretrained model.
- mattnewport 7y ago
  If you have the resources of a nation state then you already have the resources to push your agenda through traditional media. It's not clear what benefit this type of AI generated content would have over the traditional approach.
  - rm_-rf_slash 7y ago
    Asymmetric information warfare. Flood every channel with misleading and contradicting information beyond the capacity for fact-checking. Make entirely fake social media threads to steer perceptions towards a predetermined direction. Create exaggerated stereotypes of both sides of a debate to inflame and harden opinions. All of this is already being done right now. The difference is that a robust language model would allow this to be done with far fewer humans involved. Fewer people to raise their hands about how far down the ethical rabbit hole they are willing to go. And a single AI model could react to news much more quickly than a team of people who have to sleep at some point.
    - whatshisface 7y ago
      That's not asymmetric, any nation could do it. Also, the Reddit farms wouldn't have to sleep, because the writers could work in shifts.
      - cardiffspaceman 7y ago
        The human writers are constrained by what they think they can get away with. The automated writers don't even work that way. You can see from the examples used in discussions here, the more limited the domain of discourse the more convincing the samples are (except the hello world one that looked like a mash-up of every programming tutorial ever written). I think this thing could do a great job making fake announcements to air travelers about flight delays. I think the synthetic texts are already convincing enough that the natural desire to try to impart meaning to writings/utterances will go a long way to convincing forum user "victims" that something is being said that needs to be thought about. And here in this discussion, we know that the fakes are fakes (except for one comment which is unlabeled and I'm not sure about), in an aggression against a forum the fakes won't be labeled.
        rm_-rf_slash 7y ago
        The less said, the easier it is for a language model to approximate a useful thread comment for the purposes of mass propaganda. “These graphics look terrible. I will never play this game.” “$Candidate is a corporate shill and everyone knows it.” “I can’t wait for $Artist’s next album! They’re sooooo good!” Doesn’t need to be an extensive, well thought out comment to drive thought and discourse. GPT2 is good enough for that.
- Bartweiss 7y ago
  This is certainly a reasonable idea, and the OpenAI release statement did include a risk assessment attempting to determine who would be able to replicate their work. It's sort of hard for me to understand how this lead to the decision they made, though. OpenAI's original aim was democratizing access to AI tools so they wouldn't be abused by a handful of players, but releasing a scaled-down model seemed to primarily limit its utility for hobbyists. Assembling and training on a larger corpus is precisely the sort of step that's out of reach for individuals, but extremely approachable for governments or corporations with lots of computing power and expert-hours. That said, OpenAI also noted that their staged-release model was something of an experiment, so it's not entirely clear to me how much danger they thought GPT-2 actually posed. It seems to have been a useful experiment in gradual release and collaboration with other researchers, and a test of 'inoculating' people via a weak model. GPT-2 isn't a great tool for tricking people or disrupting conversations, but publicizing awareness of AI text generation before turning it loose has probably further decreased that risk.
- ryanmercer 7y ago
  You don't even need a nation state. 1 person could sufficiently weaponize this against multiple individuals by using it to mimic their writing style by training it on their social media/blog/articles they've written and then using it to start various accounts online as the target and slowly building up followers with content and at some point switching to saying highly controversial, or downright offensive, things to immediately cause people to change their opinion of the individual. Racist/sexist/etc content, allegations against someone else, fake content about a company/product/service that could cause varying levels of financial damages to the company etc. It also is a new tool for blackhat SEO, instead of paying people to write crappy content, or running relevant articles through any number of commercially available software packages that will rewrite the article enough to fool search engines as being unique for weeks or even years, you can simply train the software on stacks of books of material about a given subject and have it churn out semi-intelligible-to-humans articles about the subject that could allow you to build a rather large amount of content to manipulate search engine rankings or even to just create contextual/affiliate ad farms. If you have the software churn out smaller sets of content, then quickly edit it yourself, a single individual could churn out 10-20,000 words of content a day with banners/images/contextual ads inserted. A small blackhat SEO team in a country like Bangladesh or India could offer prices far cheaper than competition using more traditional methods, have articles that are more 'natural' in the customer's language, and be of a likely higher quality in a much higher volume. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states?
kdavis 7y ago
Plot twist, the article was written by GPT-2
- hash872 7y ago
  You read my mind
- snazz 7y ago
  I gave the first sentence to https://talktotransformer.com https://talktotransformer.com and it returned this: It seems pretty clear to me by now that GPT-2 is not as dangerous as OpenAI thought (or claimed to think) it might be. Generating fake news using GPT-2 is more difficult than the original "propaganda" model, but as we have seen it can be very effective. The problem with GPT-1, and all AI projects that rely on it, is that they have completely misunderstood what human beings are like. Humans like to look and feel like they are human. It's how we have evolved. While the OpenAI AI team (and probably the GPT-1 team as well) believe they understand the human brain, they completely lack empathy when it comes to real humans. A fake news system that gets to the truth of a story based solely on "what it would feel like to be" a human being is no more human than a machine trying to "feel" what it is like. As such, the idea that the OpenAI AI team believes their AI will be able to "see" what a human might see is a complete and utter failure. The truth is that we don't understand what
  - bitL 7y ago
    ...and that's just GPT-2_medium.
    - slavik81 7y ago
      I actually thought snazz had forgotten to include the generated snippet. However, I didn't understand the reasoning in the second generated paragraph and stopped reading. At that point, I still thought they were human (and wrong). I suppose my question is why it matters who wrote it. I've always been taught that an argument should be judged on its own merits, and from that perspective nothing changes.
nradov 7y ago
The whole notion of GPT-2 being "dangerous" was ridiculous to begin with. OpenAI does some impressive technical work but they're a little too impressed with themselves and appear somewhat detached from reality.
- randallsquared 7y ago
  Specifically, they believed it could be dangerous as a tool for generating really good fake news and related things, not necessarily AGI.
  - swiley 7y ago
    You don't need AI to do that, NI does it well enough.
  - repolfx 7y ago
    So what? Lots of news is fake already, written by journalists who already have an agenda. You don't need any AI breakthrough to have that problem.
- throwaway2048 7y ago
  I think the danger of neural network bots overrunning social media with garbage is entirely valid.
  - TaylorAlexander 7y ago
    Social media is already pretty full of garbage. What is the actual threat here?
    - inflatableDodo 7y ago
      The actual threat is the danger of neural network bots eventually overrunning social media with far more entertaining content.
  - PeCaN 7y ago
Bartweiss 7y ago
I think this is a pretty compelling claim. The published full GPT-2 samples are significantly better than the initial release, but pretty similar to the 774M corpus output. They're disconcerting in as much as they're a leap forward for AI writing, and they have remarkably strong tone consistency. But I see two major weaknesses with GPT-2 that make it hard to imagine effective malicious use. First, the output is inconsistent. The scary-good examples are scattered among a lot of duds, and it semi-regularly breaks down into either general incoherence or distinctively robotic loops. You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. Second, it's absolutely crippled by length. Antecedents often go missing after about a paragraph. Nouns (especially proper nouns) that are likely once are likely in numerous roles, so stories will pull in figures like Obama as subject, object, and commentator on the same issue. Even stylistically, the tight guidelines of news ledes slacken within a few paragraphs, which results in a loss of focus and a rising likelihood of loops and gibberish. GPT-2 is absolutely an impressive breakthrough in machine writing, I don't mean to disparage that. But as far as it's potential for deceit or trolling, it's not particularly threatening. Quantity of output is rarely the limiting factor on impact for that, and GPT-2 doesn't offer enough in sophisticated tone to make up for what it loses in basic coherence. (For anyone wondering "why is this sourced to a random tumblr?", nostalgebraist is some flavor of AI professional who's played with GPT-2 to produce some pretty interesting results, and produced some other useful essays like an explanation of Google's Transformer architecture (https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained https://nostalgebraist.tumblr.com/post/185326092369/the-tran...).)
- jcims 7y ago
  >You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. It might be getting closer than you think: https://old.reddit.com/r/SubSimulatorGPT2/ https://old.reddit.com/r/SubSimulatorGPT2/ (Note can be nsfw, meta sub has best-of highlights) Still a lot of complete nonsense but this is just a hobby project (not mine)
  - 7y ago
atemerev 7y ago
I have played with OpenGPT-2 for a few hours, testing its capabilities for generating politically-motivated texts in particular. Indeed, it fails at large sample sizes, but it is _quite_ capable of replying to tweets and writing e.g. comments on Hacker News, Reddit or elsewhere. With some human assistance, it can produce believable text fragments that can be later copypasted and reassembled into genuine articles or blog posts. If your full-time job is shitposting and/or generating politically divisive content by megabytes, it increases your productivity by orders of magnitude.
- jamesrom 7y ago
  This, 100% this. It's the massive productivity gains, multiplied by the ever increasing ability to measure virility/persuasiveness that makes this combination dangerous.
  - rahidz 7y ago
    The biggest problem here is that, like so many of these "smart machines", OpenGPT-2 is a self-correcting machine that does not care about real-world context. ... That, of course, is exactly what GPT-2 wants you to think ;) https://i.imgur.com/lFkB3gI.png https://i.imgur.com/lFkB3gI.png
    - atemerev 7y ago
      You have officially won this thread.
- Erlich_Bachman 7y ago
  To illustrate your point, here is a twitter by a model trained on Donald Trump's tweets: https://twitter.com/botustrump https://twitter.com/botustrump
minimaxir 7y ago
In my opinion, the rationale that the original GPT-2 is too dangerous to release is not good for two reasons: a finetuned small GPT-2 model on a targeted domain dataset (https://minimaxir.com/2019/09/howto-gpt2/ https://minimaxir.com/2019/09/howto-gpt2/) already gives better results than a large default GPT-2 model for targeted news, and despite independent researchers creating GPT-2 clones at a fraction of the cost of adversarial organizations, there hasn't been any evidence of mass-produced fake news in the six months since GPT-2 was open sourced and the paper was released. GPT-2 will likely be used more for mass-producing crazy erotica (https://twitter.com/Fred_Delicious/status/1166783214750445573 https://twitter.com/Fred_Delicious/status/116678321475044557... [NSFW text]) than fake news.
- HNLurker2 7y ago
  >GPT-2 will likely be used more for mass-producing crazy erotica We should, at any cost, do not release this to the masses
tw1010 7y ago
Meh, all this "we can't release it because it's dangerous" business is just marketing
paggle 7y ago
“It is less good at larger-scale structure, like maintaining a consistent topic or (especially) making a structured argument with sub-parts larger than a few sentences.” Have you surfed Facebook recently? Shockingly few people are good at these either. The quality of actual human writing on social media is so bad that today’s neural networks can be of equal quality, which is very dangerous if used to make certain ideas seem more widely held than they are. After all, our morals/ethics are basically set by what we observe in the world around us (compare your view on torture, public execution, slavery, etc) to what your view would be if a baby with your exact DNA had been born in Ancient Babylon.
dcdulin 7y ago
Its good practice to prematurely assume threat. then in the future they can fall back on a more battle-tested policy
jamesrom 7y ago
Anyone claiming GPT-2 not to be dangerous is very uninformed about humans. If you can generate text, and find a way to measure it's persuasive effectiveness (which is not hard in 2019).. It will be used to push an agenda. And given enough time, it will do so with hypnotic levels of persuasion. The claim it's not dangerous is absolutely missing the mark.
awinter-py 7y ago
fwiw I pumped 'GPT-2 is not as dangerous as' into writeup.ai and got: > GPT-2 is not as dangerous as the older version. The second issue is that the newer version of the MSA- 1 does not support the new EFI bootloader. This means that the old MSA- 1 cannot be used to boot the newer MSA-2. The MSA-2 is a newer version of the MSA-1. Talktotransformer.com thinks: > GPT-2 is not as dangerous as erythromycin and sulfamethoxazole and should be used with caution during pregnancy. While this tool is not yet smart enough to comment on this debate, a future version of it might be. Other good prompts are 'A good pasta sauce begins with'
- jonathankoren 7y ago
  "hello" generates... hello_world: from funcs import funcs_builder from funcs_spec import funcs_spec from funcs_unit import Unit import ui import os import sys import sys.stdin import time import json from scopes import Context , QueryBuilder , StructContext , ContextHandler , QueryBuilderType , ContextType , StringContext def add_from ( builder , context_id ): """ Create a query builder that expects a builder object. """ # Create the context that contains the custom query builder if Context ( builders_context ) is None : context_type = StructContext ( builder ) . type . get () # Define the builder for this context in context_builder ( context_id ): context = builder . get_context ( context_id ) # Create the query builder for the query builder in context . query_builder ( context , context_id ): return builder return "hello_world" class CustomSchema ( QueryBuilder ): querysource = { 'base' : 'world' , 'world_scopes' : [], 'query_builder' : 'hello_world' , } query_builder = CustomSchema () def add_to
aaron695 7y ago
We knew this at the time, but no one in the media, or even the tech world called them on it. It's great to have a blog with empirical info, but the damage has been done. It was never better than a Markov chain. Yes it was different to a Markov chain, but for any practical use like creating blog spam it improved nothing. Given OpenAI never said how it was dangerous, I'm not sure why people couldn't see the lie a mile off.
Animats 7y ago
Like other chatterbots that don't really know what they're talking about, coherence between sentences is poor. There's no internal model beyond likely words. The next stage is something that takes an outline and cranks out a paper or speech to fit. There are specialized systems like that for sports reporting.
WhitneyLand 7y ago
What's eye opening is the need for articles like these to seriously investigate and characterize the distinction with humans. Then comments here discussing it. Beyond all the AI hype, there are notably more scenarios where the question has to be seriously asked and a detailed answer is informative. It reminds me of the .com bubble in a way. There is too much hype, misunderstanding of the real current state of things, and bad predictions about the future. However only flip side simultaneously something huge is happening, it's real. The real non-academic milestones of progress are just smaller, more numerous and subtle than the headlines. In this case, it's not whether it's dangerous that's interesting as much as the subtle improvements made and how the march continues forward by degree.
buboard 7y ago
My understanding was that the main issue is this would make spam too hard to detect. There is also the question of whether we are overanalyzing the output of the model. The model pretty much spits out garbage, but our brains are hellbent to find patterns in that stream, because thats what brains do. A simple analysis would probably show that it s no more meaningful than random words.
tomweingarten 7y ago
Did anyone else read that whole article waiting for the reveal that it was generated by GPT-2? Ok fine... I skimmed it
designium 7y ago
Just quick question. Which graphic card should I use to run 774M version? I tried running with 2070 RTX in the 355M and it went out of memory.
- bitL 7y ago
  Haven't tested myself, but the minimum for current SOTA NLP models is 1-2x Titan RTX.
vessenes 7y ago
This is an article that the author is going to regret in five to ten years, sadly. Basic claims: 1. Long form text coherence problems in GPT-2 mean it's not useful for creating propaganda. 2. No propaganda has been noticed in the wild since GPT-2 announced, hence GPT-2 or its near successors are not dangerous. 3. Existing alt-right propaganda (presumed written by humans) is already plenty effective, who needs better written propaganda? Maybe it's enough just to write down the logical premises, but I'll say what I think, (and we can check back here in 10 years and see who's more correct, I hope it's the author) -- anything that changes the cost of information creation and dissemination is fundamentally a highly powerful force. In A16Z terms, software looks like it's going to eat writing, at least certain forms of writing. We know from numerous examples that this means radically faster innovation time, cheaper scale and wealth creation and destruction are likely not far behind. Compounding the problem for threat assessment is the toupeé fallacy -- only obviously poorly generated text is noticeable as "AI generated". I would urge anyone thinking about these things to flat out disregard any statements that AI-generated text is not in the wild or if it is that it is not effective. You literally have no way of knowing if you have read AI generated text -- in fact, the examples curated from existing models suggest curated text can be high quality enough that it is likely being used online in some way today. It's not going to get harder to generate text that reads well. It's not going to get more expensive. It's not going to get slower. Architectures that tune text creation to create clickbait titles with underlying goals are going to get worked on and thought about and tested. Whether it will be more typically a nation state tool, corporate tool or instead the equivalent of a molotov cocktail - cheap digital force extension for renegades or infoterrorists -- this isn't clear yet. But my money would be on all three, sadly. To solutions - Comparatively little science fiction thought work has been done about this that I'm aware of. Vernor Vinge speculated about broadscale disinformation as a public service in Rainbow's End -- The Friends of Privacy -- and at some level projects like Xanadu and more recently advertisement attribution (blockchain or not) tech startups are all working on this from different angles. To my mind, these questions come from a part of the Internet's architecture - text isn't generally signed or strongly attributed, and if it were, we don't really have a solid global identity infrastructure - hence a lot of worrying and hand wringing. I'm trying to invest in stuff that works on this identity layer specifically, but honestly it's an immense problem with few compelling stories about how things could change.
anilakar 7y ago
I know I'm treading on dangerous waters here considering Sam Altman's involvement, but I'm gonna burn some karma regardless: Is there anything open about OpenAI or is it just a name?
- juped 7y ago
  The mission statement reads, in part: > We’re hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. I have no idea if they live up to this but "GPT-2 is too dangerous to release" crackpottery suggests they at least sometimes don't.
  - Miraste 7y ago
    That mission statement has been abandoned. OpenAI restructured as a for-profit corporation several months ago.
thuylun2412 7y ago
godđ
alexmlamb 7y ago
Fake news concern isn't the right place to focus. News type content is low-volume, has a structured sharing model (i.e. people usually share the articles), is long-form, and requires fairly high precision. This is where humans excel and where automated language models are the weakest. On the other hand, comments on sites like Facebook/Reddit/4chan are perfect for language model bots. The content is high-volume, semi-anonymous, usually shared without an explicit network, and can be extremely low precision. So if you had a bot that could get on any discussion network for planning protests for example, and spammed it with destructive and divisive fake comments, it could actually make organizing pretty hard. And while there's some demand on the language model, many comments are just a few sentences long. And the content doesn't need to be very precise, it just needs to be convincing enough to be distracting. I also think that the worst abusers are likely to be current power-players like Google/Facebook/Chinese Gov rather than small actors.
Improvotter 7y ago
Could we talk about how horrible Tumblr's "privacy form" is? It is literally impossible to decline.
Havoc 7y ago
It was a great PR stunt though
ryanmercer 7y ago
I disagree, you can have software spit out these meh articles then pay a room full of people with the writing ability of an average 15 year old to go line by line rewriting the content with their own 'voice' and you can easily churn on massive amounts of content based on whatever you trained it on. For over a decade individuals have used software to take already written content and change it just enough to fool search engines for SEO purposes, it's been an effective tactic despite the articles often being largely unintelligible to a human. If you can make something moderately intelligible to a human, about a given area and have some minimum wage employee 'personalize' it and BAM you can turn a room of a 10, 20, 100 people into a hardcore content machine. I've suggested to Altman before that, given a sufficient body of work, you could churn out fiction in the style of famous deceased authors by having the machine do the bulk of the work then having a small team go in and edit the work to make it fully coherent and an enjoyable read. With someone like me, that uses their name as their username virtually everywhere, you could sufficiently train the machine on my reddit and blog alone to imitate me on social media platforms. It could learn my writing style, my habits of using 'heh' and 'haha' way too much on reddit/twitter/facebook and you suddenly create Bizarro Ryan that you can create new social media accounts for and start tossing in some hate speech in an anti-me campaign. While this wouldn't do much to me, to a celebrity/politician/expert in a field it could absolutely ruin their career, even if later proven to have been faked because popular opinion will still associate that person with that undesirable behavior. While pursuing this technology would be amazing for creating new literary works from people like Verne, Heinlein, Burroughs (your favorite authors here), I can weaponize it RIGHT NOW. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states? Maybe they have someone but, methinks they don't.
- retsibsi 7y ago
spyder 7y ago
You can see how GPT-2 works on a social media site in the "SubSimulatorGPT2" subreddit, where every post and comment generated by it (but it's probably just the smaller model) sometimes it's funny sometimes it's scary: https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/

161 points by luu 7y ago

I'm probably not qualified to say this so take it with a grain of salt, but I'd be surprised if OpenAI is concerned about a lone hobbyist, and is much more concerned about nation states with access to much more hardware, software, and experts who can do things that no single person or small group of people could do.

comex 7y ago
Sounds like exactly the kind of actor that would have the budget and expertise to replicate GPT-2 itself, rather than having to rely on a pretrained model.
mattnewport 7y ago
If you have the resources of a nation state then you already have the resources to push your agenda through traditional media. It's not clear what benefit this type of AI generated content would have over the traditional approach.
- rm_-rf_slash 7y ago
  Asymmetric information warfare. Flood every channel with misleading and contradicting information beyond the capacity for fact-checking. Make entirely fake social media threads to steer perceptions towards a predetermined direction. Create exaggerated stereotypes of both sides of a debate to inflame and harden opinions. All of this is already being done right now. The difference is that a robust language model would allow this to be done with far fewer humans involved. Fewer people to raise their hands about how far down the ethical rabbit hole they are willing to go. And a single AI model could react to news much more quickly than a team of people who have to sleep at some point.
  - whatshisface 7y ago
    That's not asymmetric, any nation could do it. Also, the Reddit farms wouldn't have to sleep, because the writers could work in shifts.
    - cardiffspaceman 7y ago
      The human writers are constrained by what they think they can get away with. The automated writers don't even work that way. You can see from the examples used in discussions here, the more limited the domain of discourse the more convincing the samples are (except the hello world one that looked like a mash-up of every programming tutorial ever written). I think this thing could do a great job making fake announcements to air travelers about flight delays. I think the synthetic texts are already convincing enough that the natural desire to try to impart meaning to writings/utterances will go a long way to convincing forum user "victims" that something is being said that needs to be thought about. And here in this discussion, we know that the fakes are fakes (except for one comment which is unlabeled and I'm not sure about), in an aggression against a forum the fakes won't be labeled.
      - rm_-rf_slash 7y ago
        The less said, the easier it is for a language model to approximate a useful thread comment for the purposes of mass propaganda. “These graphics look terrible. I will never play this game.” “$Candidate is a corporate shill and everyone knows it.” “I can’t wait for $Artist’s next album! They’re sooooo good!” Doesn’t need to be an extensive, well thought out comment to drive thought and discourse. GPT2 is good enough for that.
Bartweiss 7y ago
This is certainly a reasonable idea, and the OpenAI release statement did include a risk assessment attempting to determine who would be able to replicate their work. It's sort of hard for me to understand how this lead to the decision they made, though. OpenAI's original aim was democratizing access to AI tools so they wouldn't be abused by a handful of players, but releasing a scaled-down model seemed to primarily limit its utility for hobbyists. Assembling and training on a larger corpus is precisely the sort of step that's out of reach for individuals, but extremely approachable for governments or corporations with lots of computing power and expert-hours. That said, OpenAI also noted that their staged-release model was something of an experiment, so it's not entirely clear to me how much danger they thought GPT-2 actually posed. It seems to have been a useful experiment in gradual release and collaboration with other researchers, and a test of 'inoculating' people via a weak model. GPT-2 isn't a great tool for tricking people or disrupting conversations, but publicizing awareness of AI text generation before turning it loose has probably further decreased that risk.
ryanmercer 7y ago
You don't even need a nation state. 1 person could sufficiently weaponize this against multiple individuals by using it to mimic their writing style by training it on their social media/blog/articles they've written and then using it to start various accounts online as the target and slowly building up followers with content and at some point switching to saying highly controversial, or downright offensive, things to immediately cause people to change their opinion of the individual. Racist/sexist/etc content, allegations against someone else, fake content about a company/product/service that could cause varying levels of financial damages to the company etc. It also is a new tool for blackhat SEO, instead of paying people to write crappy content, or running relevant articles through any number of commercially available software packages that will rewrite the article enough to fool search engines as being unique for weeks or even years, you can simply train the software on stacks of books of material about a given subject and have it churn out semi-intelligible-to-humans articles about the subject that could allow you to build a rather large amount of content to manipulate search engine rankings or even to just create contextual/affiliate ad farms. If you have the software churn out smaller sets of content, then quickly edit it yourself, a single individual could churn out 10-20,000 words of content a day with banners/images/contextual ads inserted. A small blackhat SEO team in a country like Bangladesh or India could offer prices far cheaper than competition using more traditional methods, have articles that are more 'natural' in the customer's language, and be of a likely higher quality in a much higher volume. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states?

kdavis 7y ago

Plot twist, the article was written by GPT-2

hash872 7y ago
You read my mind
snazz 7y ago
I gave the first sentence to https://talktotransformer.com https://talktotransformer.com and it returned this: It seems pretty clear to me by now that GPT-2 is not as dangerous as OpenAI thought (or claimed to think) it might be. Generating fake news using GPT-2 is more difficult than the original "propaganda" model, but as we have seen it can be very effective. The problem with GPT-1, and all AI projects that rely on it, is that they have completely misunderstood what human beings are like. Humans like to look and feel like they are human. It's how we have evolved. While the OpenAI AI team (and probably the GPT-1 team as well) believe they understand the human brain, they completely lack empathy when it comes to real humans. A fake news system that gets to the truth of a story based solely on "what it would feel like to be" a human being is no more human than a machine trying to "feel" what it is like. As such, the idea that the OpenAI AI team believes their AI will be able to "see" what a human might see is a complete and utter failure. The truth is that we don't understand what
- bitL 7y ago
  ...and that's just GPT-2_medium.
  - slavik81 7y ago
    I actually thought snazz had forgotten to include the generated snippet. However, I didn't understand the reasoning in the second generated paragraph and stopped reading. At that point, I still thought they were human (and wrong). I suppose my question is why it matters who wrote it. I've always been taught that an argument should be judged on its own merits, and from that perspective nothing changes.

nradov 7y ago

The whole notion of GPT-2 being "dangerous" was ridiculous to begin with. OpenAI does some impressive technical work but they're a little too impressed with themselves and appear somewhat detached from reality.

randallsquared 7y ago
Specifically, they believed it could be dangerous as a tool for generating really good fake news and related things, not necessarily AGI.
- swiley 7y ago
  You don't need AI to do that, NI does it well enough.
- repolfx 7y ago
  So what? Lots of news is fake already, written by journalists who already have an agenda. You don't need any AI breakthrough to have that problem.
throwaway2048 7y ago
I think the danger of neural network bots overrunning social media with garbage is entirely valid.
- TaylorAlexander 7y ago
  Social media is already pretty full of garbage. What is the actual threat here?
  - inflatableDodo 7y ago
    The actual threat is the danger of neural network bots eventually overrunning social media with far more entertaining content.
- PeCaN 7y ago

Bartweiss 7y ago

I think this is a pretty compelling claim. The published full GPT-2 samples are significantly better than the initial release, but pretty similar to the 774M corpus output. They're disconcerting in as much as they're a leap forward for AI writing, and they have remarkably strong tone consistency. But I see two major weaknesses with GPT-2 that make it hard to imagine effective malicious use. First, the output is inconsistent. The scary-good examples are scattered among a lot of duds, and it semi-regularly breaks down into either general incoherence or distinctively robotic loops. You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. Second, it's absolutely crippled by length. Antecedents often go missing after about a paragraph. Nouns (especially proper nouns) that are likely once are likely in numerous roles, so stories will pull in figures like Obama as subject, object, and commentator on the same issue. Even stylistically, the tight guidelines of news ledes slacken within a few paragraphs, which results in a loss of focus and a rising likelihood of loops and gibberish. GPT-2 is absolutely an impressive breakthrough in machine writing, I don't mean to disparage that. But as far as it's potential for deceit or trolling, it's not particularly threatening. Quantity of output is rarely the limiting factor on impact for that, and GPT-2 doesn't offer enough in sophisticated tone to make up for what it loses in basic coherence. (For anyone wondering "why is this sourced to a random tumblr?", nostalgebraist is some flavor of AI professional who's played with GPT-2 to produce some pretty interesting results, and produced some other useful essays like an explanation of Google's Transformer architecture (https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained https://nostalgebraist.tumblr.com/post/185326092369/the-tran...).)

jcims 7y ago
>You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. It might be getting closer than you think: https://old.reddit.com/r/SubSimulatorGPT2/ https://old.reddit.com/r/SubSimulatorGPT2/ (Note can be nsfw, meta sub has best-of highlights) Still a lot of complete nonsense but this is just a hobby project (not mine)
- 7y ago

atemerev 7y ago

I have played with OpenGPT-2 for a few hours, testing its capabilities for generating politically-motivated texts in particular. Indeed, it fails at large sample sizes, but it is _quite_ capable of replying to tweets and writing e.g. comments on Hacker News, Reddit or elsewhere. With some human assistance, it can produce believable text fragments that can be later copypasted and reassembled into genuine articles or blog posts. If your full-time job is shitposting and/or generating politically divisive content by megabytes, it increases your productivity by orders of magnitude.

jamesrom 7y ago
This, 100% this. It's the massive productivity gains, multiplied by the ever increasing ability to measure virility/persuasiveness that makes this combination dangerous.
- rahidz 7y ago
  The biggest problem here is that, like so many of these "smart machines", OpenGPT-2 is a self-correcting machine that does not care about real-world context. ... That, of course, is exactly what GPT-2 wants you to think ;) https://i.imgur.com/lFkB3gI.png https://i.imgur.com/lFkB3gI.png
  - atemerev 7y ago
    You have officially won this thread.
Erlich_Bachman 7y ago
To illustrate your point, here is a twitter by a model trained on Donald Trump's tweets: https://twitter.com/botustrump https://twitter.com/botustrump

minimaxir 7y ago

In my opinion, the rationale that the original GPT-2 is too dangerous to release is not good for two reasons: a finetuned small GPT-2 model on a targeted domain dataset (https://minimaxir.com/2019/09/howto-gpt2/ https://minimaxir.com/2019/09/howto-gpt2/) already gives better results than a large default GPT-2 model for targeted news, and despite independent researchers creating GPT-2 clones at a fraction of the cost of adversarial organizations, there hasn't been any evidence of mass-produced fake news in the six months since GPT-2 was open sourced and the paper was released. GPT-2 will likely be used more for mass-producing crazy erotica (https://twitter.com/Fred_Delicious/status/1166783214750445573 https://twitter.com/Fred_Delicious/status/116678321475044557... [NSFW text]) than fake news.

HNLurker2 7y ago
>GPT-2 will likely be used more for mass-producing crazy erotica We should, at any cost, do not release this to the masses

tw1010 7y ago

Meh, all this "we can't release it because it's dangerous" business is just marketing

paggle 7y ago

“It is less good at larger-scale structure, like maintaining a consistent topic or (especially) making a structured argument with sub-parts larger than a few sentences.” Have you surfed Facebook recently? Shockingly few people are good at these either. The quality of actual human writing on social media is so bad that today’s neural networks can be of equal quality, which is very dangerous if used to make certain ideas seem more widely held than they are. After all, our morals/ethics are basically set by what we observe in the world around us (compare your view on torture, public execution, slavery, etc) to what your view would be if a baby with your exact DNA had been born in Ancient Babylon.

dcdulin 7y ago

Its good practice to prematurely assume threat. then in the future they can fall back on a more battle-tested policy

jamesrom 7y ago

Anyone claiming GPT-2 not to be dangerous is very uninformed about humans. If you can generate text, and find a way to measure it's persuasive effectiveness (which is not hard in 2019).. It will be used to push an agenda. And given enough time, it will do so with hypnotic levels of persuasion. The claim it's not dangerous is absolutely missing the mark.

awinter-py 7y ago

fwiw I pumped 'GPT-2 is not as dangerous as' into writeup.ai and got: > GPT-2 is not as dangerous as the older version. The second issue is that the newer version of the MSA- 1 does not support the new EFI bootloader. This means that the old MSA- 1 cannot be used to boot the newer MSA-2. The MSA-2 is a newer version of the MSA-1. Talktotransformer.com thinks: > GPT-2 is not as dangerous as erythromycin and sulfamethoxazole and should be used with caution during pregnancy. While this tool is not yet smart enough to comment on this debate, a future version of it might be. Other good prompts are 'A good pasta sauce begins with'

jonathankoren 7y ago
"hello" generates... hello_world: from funcs import funcs_builder from funcs_spec import funcs_spec from funcs_unit import Unit import ui import os import sys import sys.stdin import time import json from scopes import Context , QueryBuilder , StructContext , ContextHandler , QueryBuilderType , ContextType , StringContext def add_from ( builder , context_id ): """ Create a query builder that expects a builder object. """ # Create the context that contains the custom query builder if Context ( builders_context ) is None : context_type = StructContext ( builder ) . type . get () # Define the builder for this context in context_builder ( context_id ): context = builder . get_context ( context_id ) # Create the query builder for the query builder in context . query_builder ( context , context_id ): return builder return "hello_world" class CustomSchema ( QueryBuilder ): querysource = { 'base' : 'world' , 'world_scopes' : [], 'query_builder' : 'hello_world' , } query_builder = CustomSchema () def add_to

aaron695 7y ago

We knew this at the time, but no one in the media, or even the tech world called them on it. It's great to have a blog with empirical info, but the damage has been done. It was never better than a Markov chain. Yes it was different to a Markov chain, but for any practical use like creating blog spam it improved nothing. Given OpenAI never said how it was dangerous, I'm not sure why people couldn't see the lie a mile off.

Animats 7y ago

Like other chatterbots that don't really know what they're talking about, coherence between sentences is poor. There's no internal model beyond likely words. The next stage is something that takes an outline and cranks out a paper or speech to fit. There are specialized systems like that for sports reporting.

WhitneyLand 7y ago

What's eye opening is the need for articles like these to seriously investigate and characterize the distinction with humans. Then comments here discussing it. Beyond all the AI hype, there are notably more scenarios where the question has to be seriously asked and a detailed answer is informative. It reminds me of the .com bubble in a way. There is too much hype, misunderstanding of the real current state of things, and bad predictions about the future. However only flip side simultaneously something huge is happening, it's real. The real non-academic milestones of progress are just smaller, more numerous and subtle than the headlines. In this case, it's not whether it's dangerous that's interesting as much as the subtle improvements made and how the march continues forward by degree.

buboard 7y ago

My understanding was that the main issue is this would make spam too hard to detect. There is also the question of whether we are overanalyzing the output of the model. The model pretty much spits out garbage, but our brains are hellbent to find patterns in that stream, because thats what brains do. A simple analysis would probably show that it s no more meaningful than random words.

tomweingarten 7y ago

Did anyone else read that whole article waiting for the reveal that it was generated by GPT-2? Ok fine... I skimmed it

designium 7y ago

Just quick question. Which graphic card should I use to run 774M version? I tried running with 2070 RTX in the 355M and it went out of memory.

bitL 7y ago
Haven't tested myself, but the minimum for current SOTA NLP models is 1-2x Titan RTX.

vessenes 7y ago

This is an article that the author is going to regret in five to ten years, sadly. Basic claims: 1. Long form text coherence problems in GPT-2 mean it's not useful for creating propaganda. 2. No propaganda has been noticed in the wild since GPT-2 announced, hence GPT-2 or its near successors are not dangerous. 3. Existing alt-right propaganda (presumed written by humans) is already plenty effective, who needs better written propaganda? Maybe it's enough just to write down the logical premises, but I'll say what I think, (and we can check back here in 10 years and see who's more correct, I hope it's the author) -- anything that changes the cost of information creation and dissemination is fundamentally a highly powerful force. In A16Z terms, software looks like it's going to eat writing, at least certain forms of writing. We know from numerous examples that this means radically faster innovation time, cheaper scale and wealth creation and destruction are likely not far behind. Compounding the problem for threat assessment is the toupeé fallacy -- only obviously poorly generated text is noticeable as "AI generated". I would urge anyone thinking about these things to flat out disregard any statements that AI-generated text is not in the wild or if it is that it is not effective. You literally have no way of knowing if you have read AI generated text -- in fact, the examples curated from existing models suggest curated text can be high quality enough that it is likely being used online in some way today. It's not going to get harder to generate text that reads well. It's not going to get more expensive. It's not going to get slower. Architectures that tune text creation to create clickbait titles with underlying goals are going to get worked on and thought about and tested. Whether it will be more typically a nation state tool, corporate tool or instead the equivalent of a molotov cocktail - cheap digital force extension for renegades or infoterrorists -- this isn't clear yet. But my money would be on all three, sadly. To solutions - Comparatively little science fiction thought work has been done about this that I'm aware of. Vernor Vinge speculated about broadscale disinformation as a public service in Rainbow's End -- The Friends of Privacy -- and at some level projects like Xanadu and more recently advertisement attribution (blockchain or not) tech startups are all working on this from different angles. To my mind, these questions come from a part of the Internet's architecture - text isn't generally signed or strongly attributed, and if it were, we don't really have a solid global identity infrastructure - hence a lot of worrying and hand wringing. I'm trying to invest in stuff that works on this identity layer specifically, but honestly it's an immense problem with few compelling stories about how things could change.

anilakar 7y ago

I know I'm treading on dangerous waters here considering Sam Altman's involvement, but I'm gonna burn some karma regardless: Is there anything open about OpenAI or is it just a name?

juped 7y ago
The mission statement reads, in part: > We’re hoping to grow OpenAI into such an institution. As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies. I have no idea if they live up to this but "GPT-2 is too dangerous to release" crackpottery suggests they at least sometimes don't.
- Miraste 7y ago
  That mission statement has been abandoned. OpenAI restructured as a for-profit corporation several months ago.

thuylun2412 7y ago

godđ

alexmlamb 7y ago

Fake news concern isn't the right place to focus. News type content is low-volume, has a structured sharing model (i.e. people usually share the articles), is long-form, and requires fairly high precision. This is where humans excel and where automated language models are the weakest. On the other hand, comments on sites like Facebook/Reddit/4chan are perfect for language model bots. The content is high-volume, semi-anonymous, usually shared without an explicit network, and can be extremely low precision. So if you had a bot that could get on any discussion network for planning protests for example, and spammed it with destructive and divisive fake comments, it could actually make organizing pretty hard. And while there's some demand on the language model, many comments are just a few sentences long. And the content doesn't need to be very precise, it just needs to be convincing enough to be distracting. I also think that the worst abusers are likely to be current power-players like Google/Facebook/Chinese Gov rather than small actors.

Improvotter 7y ago

Could we talk about how horrible Tumblr's "privacy form" is? It is literally impossible to decline.

Havoc 7y ago

It was a great PR stunt though

ryanmercer 7y ago

I disagree, you can have software spit out these meh articles then pay a room full of people with the writing ability of an average 15 year old to go line by line rewriting the content with their own 'voice' and you can easily churn on massive amounts of content based on whatever you trained it on. For over a decade individuals have used software to take already written content and change it just enough to fool search engines for SEO purposes, it's been an effective tactic despite the articles often being largely unintelligible to a human. If you can make something moderately intelligible to a human, about a given area and have some minimum wage employee 'personalize' it and BAM you can turn a room of a 10, 20, 100 people into a hardcore content machine. I've suggested to Altman before that, given a sufficient body of work, you could churn out fiction in the style of famous deceased authors by having the machine do the bulk of the work then having a small team go in and edit the work to make it fully coherent and an enjoyable read. With someone like me, that uses their name as their username virtually everywhere, you could sufficiently train the machine on my reddit and blog alone to imitate me on social media platforms. It could learn my writing style, my habits of using 'heh' and 'haha' way too much on reddit/twitter/facebook and you suddenly create Bizarro Ryan that you can create new social media accounts for and start tossing in some hate speech in an anti-me campaign. While this wouldn't do much to me, to a celebrity/politician/expert in a field it could absolutely ruin their career, even if later proven to have been faked because popular opinion will still associate that person with that undesirable behavior. While pursuing this technology would be amazing for creating new literary works from people like Verne, Heinlein, Burroughs (your favorite authors here), I can weaponize it RIGHT NOW. This is the problem with entities like Open AI, they're all "AI is great, AI is good, AI is our future savior, yay AI" but are any of them going "well, here's the 16 ways I can think off, off the top of my head, how to wildly exploit this technology for personal/corporate/government gain"? AI doesn't have to be SkyNet or robotic killing drones to be exploited, an individual can benefit considerably (and cause considerable hardship for an entity) with stuff like this. Who at places like Open AI are asking these questions? Where are the employees/advisors/consultants that look at each project and offer real time feedback on ways to abuse the project in its current and near-future states? Maybe they have someone but, methinks they don't.

retsibsi 7y ago

spyder 7y ago

You can see how GPT-2 works on a social media site in the "SubSimulatorGPT2" subreddit, where every post and comment generated by it (but it's probably just the smaller model) sometimes it's funny sometimes it's scary: https://www.reddit.com/r/SubSimulatorGPT2/ https://www.reddit.com/r/SubSimulatorGPT2/