5 ms·

I think this is a pretty compelling claim. The published full GPT-2 samples are significantly better than the initial release, but pretty similar to the 774M co

I think this is a pretty compelling claim. The published full GPT-2 samples are significantly better than the initial release, but pretty similar to the 774M corpus output. They're disconcerting in as much as they're a leap forward for AI writing, and they have remarkably strong tone consistency. But I see two major weaknesses with GPT-2 that make it hard to imagine effective malicious use. First, the output is inconsistent. The scary-good examples are scattered among a lot of duds, and it semi-regularly breaks down into either general incoherence or distinctively robotic loops. You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. Second, it's absolutely crippled by length. Antecedents often go missing after about a paragraph. Nouns (especially proper nouns) that are likely once are likely in numerous roles, so stories will pull in figures like Obama as subject, object, and commentator on the same issue. Even stylistically, the tight guidelines of news ledes slacken within a few paragraphs, which results in a loss of focus and a rising likelihood of loops and gibberish. GPT-2 is absolutely an impressive breakthrough in machine writing, I don't mean to disparage that. But as far as it's potential for deceit or trolling, it's not particularly threatening. Quantity of output is rarely the limiting factor on impact for that, and GPT-2 doesn't offer enough in sophisticated tone to make up for what it loses in basic coherence. (For anyone wondering "why is this sourced to a random tumblr?", nostalgebraist is some flavor of AI professional who's played with GPT-2 to produce some pretty interesting results, and produced some other useful essays like an explanation of Google's Transformer architecture (https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-explained https://nostalgebraist.tumblr.com/post/185326092369/the-tran...).)

jcims 7y ago
>You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. It might be getting closer than you think: https://old.reddit.com/r/SubSimulatorGPT2/ https://old.reddit.com/r/SubSimulatorGPT2/ (Note can be nsfw, meta sub has best-of highlights) Still a lot of complete nonsense but this is just a hobby project (not mine)
- taneq 7y ago
  When you combine the output available now with the fact that a surprising number of humans on the internet would probably fail a written Turing test, it's definitely useful at least for denial-of-service style mischief.
  - c0bb 7y ago
    It seems to be supervisable at this point, specifically for shorter responses. I was thinking if you're trying to AstroTurf or some other such form of maliciously using message boards/forums, rather than giving free reign you could present an operator with the prompting text and some arbitrary number of generated replies to choose from. Still entails some leg work but will likely yield more believable responses more frequently. Sure this is only economical if at all when replying to shorter prompts, but that is likely going to be the majority on most forums and message boards.
    - jcims 7y ago
      Would be good for phishing too.
- worldsayshi 7y ago
  Impressive. It's mostly nonsense but compared to regular r/SubredditSimulator/ it definitely looks a lot more conscious.
  - jcims 7y ago
    The hilarious thing is that the project owner fine-tuned against /r/SubredditSimulator and the extra nonsense shines through. E.g.: https://www.reddit.com/r/SubSimulatorGPT2/comments/d1zh7r/a_new_study_in_the_journal_of_the_american/ https://www.reddit.com/r/SubSimulatorGPT2/comments/d1zh7r/a_... Stroll through the replies to get a flavor of the sanity of them then ctrl+f subredditsimulatorgp
- TheCabin 7y ago
  It's weird, after reading comments in this reddit and coming back to HN it feels like comments here are generated too :)
ma2rten 7y ago
Transformer-XL should be better at learning long-term dependencies. See the last section of this paper: https://arxiv.org/pdf/1901.02860.pdf https://arxiv.org/pdf/1901.02860.pdf
- Bartweiss 7y ago
  Interesting, thanks! I figured that expanding scope was probably a task we knew how to solve/improve, this makes sense. Beyond keeping antecedents, it seems like this could indirectly improve "essay structure" by retaining more knowledge of what's already been said. I'm curious to see how much of GPT-2's length issues turn out to stem from long-range dependencies as opposed to other issues. Right now long output seems to suffer a combination of lost scope, lack of division/structure, diverging reference texts, self-contradiction, and redundancy/looping traps. One of the most interesting patterns I've noticed is that GPT-2 "knows" which concepts relate to one another, but not which stances. Essays for and against Foo are both essays about Foo, so GPT-2 slips between them quite happily. It seems like the sort of problem that should be pretty tractable to improve, but all the obvious approaches would be pretty domain-specific.
  - ma2rten 7y ago
    Language models like GPT-2 have fundamental limitations for text generation. They simply model the most likely sequence of words, they don't have a "real understanding" of what they write. They don't have agency (a purpose), can't do explicit reasoning and don't have symbol grounding. Few people would have thought that we can get this type of quality using just language modeling, so it's unclear how much further we can go before we have to start tackling those fundamental problems.
    - Bartweiss 7y ago
      > Few people would have thought that we can get this type of quality using just language modeling, so it's unclear how much further we can go before we have to start tackling those fundamental problems. Yep, that seems to be the core question. This isn't going to be the road to AGI, but it's already ahead of what most people thought could be done without underlying referents. As you say, GPT-2 has some fundamental limitations. One is 'cognitive': it doesn't have referents or make any kind of logical assessments, only symbolic ones. Even if it develops to the point of writing publication-worthy essays, they'll be syntheses of existing texts, and inventing a novel argument (except by chance) should be essentially impossible. Even cross-format transference is basically out of the question; a news story reading "the mouse ate the tiger" won't be ruled out based on dictionary entries about mice and tigers. The other, though, is 'intentful'. As Nostalgebraist points out, GPT-2 is essentially a text predictor impersonating a text generator. That difference creates an intrinsic limitation, because GPT-2 isn't trying to create output interchangeable with its inputs. It actively moves away from "write a novel" towards "write the most common/formulaic elements of a novel". At the current level of quality, this might actually improve results; GPT is less likely to make jarring errors when emulating LoTR's walking-and-eating sections than when emulating Gandalf's death. But as it improves and is used to generate whole texts, the "most likely" model leaves it writing unsurprising, mid-corpus text indefinitely. Solving the cognitive limitation essentially requires AGI, and even adding crude referents in one domain would be quite tough. So I'll make a wild speculation that fixing the intentful limitation will be a big advance soon: training specifically for generation will produce models that more accurately recreate the structure and 'tempo' of human text, without being put deterred by excess novelty.
tylerl 7y ago
My understanding of the danger was a bit more practical and less abstract. If you use your "best" classifier in a GAN to generate malicious content, then regardless of whether or not it's any good, it will by definition be able to fool your spam classifier. So fighting machine generated garbage has to be done by humans again, which scales much more poorly.
- Bartweiss 7y ago
  That's a really good point, and it never even occurred to me. Thank you! Something I realize I don't know: how much spam detection is done via text analysis? I get the sense that most websites try to block traffic patterns or apply CAPTCHAs, and email spam filters are heavily based on keywords and senders. It seems like this would cause problems, but it's not immediately obvious to me where they'd crop up.

jcims 7y ago

>You couldn't turn it loose to populate a website or argue on forums without any human oversight, so it at most represents a cost reduction in the output of BS, not a qualitative shift. It might be getting closer than you think: https://old.reddit.com/r/SubSimulatorGPT2/ https://old.reddit.com/r/SubSimulatorGPT2/ (Note can be nsfw, meta sub has best-of highlights) Still a lot of complete nonsense but this is just a hobby project (not mine)

taneq 7y ago
When you combine the output available now with the fact that a surprising number of humans on the internet would probably fail a written Turing test, it's definitely useful at least for denial-of-service style mischief.
- c0bb 7y ago
  It seems to be supervisable at this point, specifically for shorter responses. I was thinking if you're trying to AstroTurf or some other such form of maliciously using message boards/forums, rather than giving free reign you could present an operator with the prompting text and some arbitrary number of generated replies to choose from. Still entails some leg work but will likely yield more believable responses more frequently. Sure this is only economical if at all when replying to shorter prompts, but that is likely going to be the majority on most forums and message boards.
  - jcims 7y ago
    Would be good for phishing too.
worldsayshi 7y ago
Impressive. It's mostly nonsense but compared to regular r/SubredditSimulator/ it definitely looks a lot more conscious.
- jcims 7y ago
  The hilarious thing is that the project owner fine-tuned against /r/SubredditSimulator and the extra nonsense shines through. E.g.: https://www.reddit.com/r/SubSimulatorGPT2/comments/d1zh7r/a_new_study_in_the_journal_of_the_american/ https://www.reddit.com/r/SubSimulatorGPT2/comments/d1zh7r/a_... Stroll through the replies to get a flavor of the sanity of them then ctrl+f subredditsimulatorgp
TheCabin 7y ago
It's weird, after reading comments in this reddit and coming back to HN it feels like comments here are generated too :)

ma2rten 7y ago

Transformer-XL should be better at learning long-term dependencies. See the last section of this paper: https://arxiv.org/pdf/1901.02860.pdf https://arxiv.org/pdf/1901.02860.pdf

Bartweiss 7y ago
Interesting, thanks! I figured that expanding scope was probably a task we knew how to solve/improve, this makes sense. Beyond keeping antecedents, it seems like this could indirectly improve "essay structure" by retaining more knowledge of what's already been said. I'm curious to see how much of GPT-2's length issues turn out to stem from long-range dependencies as opposed to other issues. Right now long output seems to suffer a combination of lost scope, lack of division/structure, diverging reference texts, self-contradiction, and redundancy/looping traps. One of the most interesting patterns I've noticed is that GPT-2 "knows" which concepts relate to one another, but not which stances. Essays for and against Foo are both essays about Foo, so GPT-2 slips between them quite happily. It seems like the sort of problem that should be pretty tractable to improve, but all the obvious approaches would be pretty domain-specific.
- ma2rten 7y ago
  Language models like GPT-2 have fundamental limitations for text generation. They simply model the most likely sequence of words, they don't have a "real understanding" of what they write. They don't have agency (a purpose), can't do explicit reasoning and don't have symbol grounding. Few people would have thought that we can get this type of quality using just language modeling, so it's unclear how much further we can go before we have to start tackling those fundamental problems.
  - Bartweiss 7y ago
    > Few people would have thought that we can get this type of quality using just language modeling, so it's unclear how much further we can go before we have to start tackling those fundamental problems. Yep, that seems to be the core question. This isn't going to be the road to AGI, but it's already ahead of what most people thought could be done without underlying referents. As you say, GPT-2 has some fundamental limitations. One is 'cognitive': it doesn't have referents or make any kind of logical assessments, only symbolic ones. Even if it develops to the point of writing publication-worthy essays, they'll be syntheses of existing texts, and inventing a novel argument (except by chance) should be essentially impossible. Even cross-format transference is basically out of the question; a news story reading "the mouse ate the tiger" won't be ruled out based on dictionary entries about mice and tigers. The other, though, is 'intentful'. As Nostalgebraist points out, GPT-2 is essentially a text predictor impersonating a text generator. That difference creates an intrinsic limitation, because GPT-2 isn't trying to create output interchangeable with its inputs. It actively moves away from "write a novel" towards "write the most common/formulaic elements of a novel". At the current level of quality, this might actually improve results; GPT is less likely to make jarring errors when emulating LoTR's walking-and-eating sections than when emulating Gandalf's death. But as it improves and is used to generate whole texts, the "most likely" model leaves it writing unsurprising, mid-corpus text indefinitely. Solving the cognitive limitation essentially requires AGI, and even adding crude referents in one domain would be quite tough. So I'll make a wild speculation that fixing the intentful limitation will be a big advance soon: training specifically for generation will produce models that more accurately recreate the structure and 'tempo' of human text, without being put deterred by excess novelty.

tylerl 7y ago

My understanding of the danger was a bit more practical and less abstract. If you use your "best" classifier in a GAN to generate malicious content, then regardless of whether or not it's any good, it will by definition be able to fool your spam classifier. So fighting machine generated garbage has to be done by humans again, which scales much more poorly.

Bartweiss 7y ago
That's a really good point, and it never even occurred to me. Thank you! Something I realize I don't know: how much spam detection is done via text analysis? I get the sense that most websites try to block traffic patterns or apply CAPTCHAs, and email spam filters are heavily based on keywords and senders. It seems like this would cause problems, but it's not immediately obvious to me where they'd crop up.