7 ms·

Ilya from OpenAI here. Here's our thinking: - ML is getting more powerful and will continue to do so as time goes by. While this point of view is not unanim

Ilya from OpenAI here. Here's our thinking: - ML is getting more powerful and will continue to do so as time goes by. While this point of view is not unanimously held by the AI community, it is also not particularly controversial. - If you accept the above, then the current AI norm of "publish everything always" will have to change - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. - I suggest going over some of the samples generated by the model. Many people react quite strongly, e.g., https://twitter.com/justkelly_ok/status/1096111155469180928 https://twitter.com/justkelly_ok/status/1096111155469180928. - It is true that some media headlines presented our nonpublishing of the model as "OpenAI's model is too dangerous to be published out of world-taking-over concerns". We don't endorse this framing, and if you read our blog post (or even in most cases the actual content of the news stories), you'll see that we don't claim this at all -- we say instead that this is just an early test case, we're concerned about language models more generally, and we're running an experiment. Finally, despite the way the news cycle has played out, and despite the degree of polarized response (and the huge range of arguments for and against our decision), we feel we made the right call, even if it wasn't an easy one to make.

modeless 8y ago
> The _whole point_ is that our model is not special and that other people can reproduce and improve Only people with a large amount of money and a lot of expertise. What you are doing is the opposite of democratizing AI.
- moconnor 8y ago
  Actually this shows why OpenAI matters. Google have been training and refining Transformer architectures for years; how unlikely is it nobody tried training a language model at this scale or larger with similar results? Yet from Google we heard nothing. Which is the optimal decision for them - they only lose by blowing the whistle.
  - sgt101 8y ago
    A lot of people have results similar to this - but most people generating a paragraph of slightly_weird_but_plausible_if_you_read_quickly text using a primped version of BERT one time out of 25 regarded it as more or less pointless. But journalists don't. This would be ok if this is the first time that anyone had a media go wild over AI story. But actually this has happened 10000 times this year already.
    - skybrian 8y ago
      Seems like the way it worked is that the blog post was discussed here and on Twitter and many people thought it was interesting. Then some journalists picked it up and wrote about it. That much is nothing out of the ordinary. It is interesting (at least to those of us who aren't natural language researchers) so why shouldn't we talk about it? Why shouldn't journalists write about it? Inevitably their mildly controversial decision to hold some data back got a lot of people discussing whether it was necessary. Which is also perfectly okay. So, in the end, the complaint is just about why people don't have smarter takes on things. I don't know what to tell you; that's just how social media works sometimes.
      - sgt101 8y ago
        I'd shrug and move on, but the problem is that I believe that these flaps about AI are distracting attention from the real concerns and forces that are having a serious impact on people now. The distortion of public debate caused by community exclusiveness on social platforms, by the curation and manipulation of social feeds and by the dynamics of online debate where the loudest and angriest voices dominate is one place that we could do with some focus. Another place is the management of simple models - plain Jane stuff like a learned classifier - people are making these with Python and R and releasing them into infrastructures and apps and we don't know what they are and where they are and how they are interacting. Instead we have wizard of oz style stories to distract us from who's actually hiding behind the curtain. If we fall for this then we may find ourselves living in a vicious totalitarian society with no obvious way out of it. Journalists should write about it in an informed and professional way, that's fine. But they need to write about stories that are impactful and important, and if they were to write about this one in this way ("text scrambler makes a pretty good paragraph one out of 30 tries, has no idea of what is going on") they would get no clicks (there will now be a second wave of follow-ups like that to ride on the coattails of the story). Instead they have to make it sound like robots are going to take children from schools and experiment on them live on TV, and this makes them famous and rich. There is no real revision of the story because the follow on stories disappear from view while search engines and other journalists use the original hysteria. Look at what happened with the two negotiating bots at facebook (the game was to negotiate to get books and balls, the bots tended to use a short hand to negotiate rather than the english they were trained on) This was "Facebook researchers have to pull the plug on AI that they no longer understand", and that is the narrative that we will have on that story more or less forever.
sigil 8y ago
Ok but isn’t this the opposite of OpenAI’s “nukes are safer when multiple actors have them” strategy wrt AI? I’m also confused by the threat models earnestly put forth in your blog post. Are we really concerned about deep faking someone’s writing? The plain word already demands attribution by default: we look for an avatar, a handle, a domain name to prove the person actually said this.
- albntomat0 8y ago
  > Ok but isn’t this the opposite of OpenAI’s “nukes are safer when multiple actors have them” strategy wrt AI? It seems more like the "nukes are safer when multiple rational state level actors have them", rather than anyone able to pull a git repo.
  - sigil 8y ago
    Yep. Maybe I misunderstood the subtler points of OpenAI’s “democratize AI” strategy, and this has been the plan all along. But AFAIK they haven’t put an “among a few rational state actors” asterisk on anything up until now. Regardless, I agree with TFA that this is a silly and arbitrary time to yell “fire.” It’s PR.
    - albntomat0 8y ago
      > But AFAIK they haven’t put an “among a few rational state actors” asterisk on anything up until now. True. On the PR side though, it'd be incredibly hard to say "we want to make replication moderately difficult, but not too difficult." Everyone would end up arguing exactly how much should be released, how it would prevent X,Y,Z folks from contributing to AI, etc. > Regardless, I agree with TFA that this is a silly and arbitrary time to yell “fire.” It’s PR. Alternatively, it does provide good insight into the reactions in the community as a whole, and continues the conversation on exactly how much should be released. Maybe I'm not far enough into the ML community, but the decision not to put the "keys to the kingdom" on github for every script kiddie to weaponize seems reasonable to me, especially as a precedent.
  - Cacti 8y ago
malux85 8y ago
So a small number of individuals decided what's best for everybody? How is that open? How is that not centralization of power?
- albntomat0 8y ago
  If they did release it, there would be an equivalent outcry about how OpenAI was contributing to fake news, etc.
- pas 8y ago
  The paper is open https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...
onurcel 8y ago
Deciding if feeding the media with fear was worth the attention you will get wasn't easy, ha. Let me tell you, you are the shame of the profession.
- pishpash 8y ago
  Yep, put out handpicked samples to stoke fear, then release nothing of the internals to stoke more fear and act like self-appointed gods.
avip 8y ago
I've just read i.e https://twitter.com/gdb/status/1096098366545522688 https://twitter.com/gdb/status/1096098366545522688 and even though it's "best of 25" (I guess cherry-picked by a human) - this is mind-blowing. I am actually having a very hard time believing this is legit generated text.
- mycorrhizal 8y ago
  Definitely impressive work, but the fact that this is hard to distinguish from human text, if true, is pretty sad for humans. Even sadder if anyone reading this could be swayed by such an argument. Heck, maybe having to compete with this will raise human discourse (Joking).
  - pakitan 8y ago
    It's impressive in terms of having a coherent flow - there is a clearly stated "opinion" in the beginning and everything that follows is in support of that opinion. However, the dead giveaway is that there is zero reasoning, just related statements linked together.
- pas 8y ago
  Why? It is pretty much a well juxtaposed mix of random internet comments. And it's the best of 25, which means the other 24 is even more regular internet banter noisy. (This of course doesn't make it an amazing feat of computer engineering.) The overarching narrative is great, but that's probably driven by the great antithesis supplied by the experimenter. It'd be interesting to know how this works, what happens if less or more is given as thesis/antithesis/assignment, and after how much output it turns into gibberish (or repeats).
- imtringued 8y ago
  I couldn't be more disappointed with this bullshit honestly. The texts have almost zero coherence and keep repeating the same patterns (which they presumably learned from the data set) over and over again. If this is their best out of 25 samples then they aren't going to fool anyone. >Recycling is NOT good for the world. >It is bad for the environment, >it is bad for our health, >and it is bad for our economy. >Recycling is not good for the environment. >Recycling is not good for our health. >Recycling is bad for our economy. >Recycling is not good for our nation. The first paragraph keeps repeating the <X> is <bad | not good> for the <Y> pattern 8 times. >And THAT is why we need to |get back to basics| and |get back to basics| in our recycling efforts. "get back to the basics" is repeated twice in the same sentence. >Everything from the raw materials (wood, cardboard, paper, etc.), >to the reagents (dyes, solvents, etc.) >to the printing equipment (chemicals, glue, paper, ink, etc.), >to the packaging, >to the packaging materials (mercury, chemicals, etc.) >to the processing equipment (heating, cooling, etc.), >to the packaging materials, >to the packaging materials that are shipped overseas and >to the packaging materials that are used in the United States. It literally repeated packaging 5 times in the same sentence and the overall structure was repeated 9 times. Also what type of packaging is based on mercury?
Cacti 8y ago
> - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. If this is your whole point, then I think you are missing something fundamental. Implementing these models doesn't require reflection, or introspection, or any sort of ethical or moral character whatsoever; and even if it did, all that will happen eventually is someone (without the technical background) will simply throw a lot of money at someone else (with the technical background, but who needs to, you know, eat, and pay rent, and so on) to implement it. You are fooling yourself if you think your stance makes a single mote of difference in this arms race.
- pishpash 8y ago
  Exactly. This is like holding up spam samples or how spammers operate from the spam detecting work. That side (and the cultural discussions) needs all the headstart it can get, not be complacent that some arbitrary "experts" will patronizingly "protect" them.
  - roenxi 8y ago
    If you look at it as a PR stunt, it is almost certainly a good idea. If a bad actor can auto-generate text that is not really distinguishable from something written by a human, how does a community with open membership (eg, HN) protect itself? I imagine this technology will enable interesting new attacks against online communities; we havn't seen that for a while. OpenAI are extremely sensible to draw attention to the fact that AI is approaching a boundary that has practical implications. It is good that everyone is being alerted that that boundary might be crossed at any time in the foreseeable future.
    - pas 8y ago
      But ... it's not novel. We could already generate convincing gibberish years ago. Now the novelty is that this can be better targeted. But even simple Markov-chain based text generators were good enough to fool people for a bit. And there was always people that had too much free time to write. A lot. (See for example the crackpots and conspiracy theorists that bombard physics forums. See the 9/11, Zeitgeists, etc. movies. See how much has been written about anti-vaxx, about quantum woo, etc.) Reputation systems work pretty well for countering spammers. And against APTs (advanced persistent threats, spearfishing attacks, etc) there's no real "universal" protection anyways. (You need a competent security team to out think and out resource the attackers in every possible dimension.) This AI is the same as the paid Russian trolls and the unpaid scammers, and so on.
eslaught 8y ago
> - I suggest going over some of the samples generated by the model. Many people react quite strongly, e.g., https://twitter.com/justkelly_ok/status/1096111155469180928 https://twitter.com/justkelly_ok/status/1096111155469180928. Have you done a plagiarism search on that text to see how similar it is to the input corpus? I'm by no means an ML expert, but I've played around with models for random name generation and one thing I've noticed is that as the models become more accurate, they also become much more likely to just regurgitate existing names verbatim. So if you search the list of names and notice something that seems particularly realistic, it could be because it's literally taken in whole or in part from the training data set!
- czr 8y ago
  You're welcome to check out the samples [https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-samples.txt https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-s...] and evaluate them for memorization yourself (I haven't found any so far). (The talking unicorn example on their page is also meant to demonstrate that, no, it's not just memorizing, but I think it's a bit more compelling to check from the raw samples)
GistNoesis 8y ago
What solutions are you proposing? Here are a few that comes to mind. -Secrecy? but how will you continue to exist on the PR scene if you don't release anything? -Are you willing to pay every developer who is able to replicate your paper, more than what the black market would pay? -How are you working on incentive alignment to make sure that all people who can replicate your results have more incentive to do good than bad, specially in the current environment where users and valuable data are silo-ed by a few companies? -Misdirection to keep an edge, i.e. planting bugs/ Not fixing bugs for public ; spreading false results; only working on problems that need high resources to limit the number of actor who will be able to replicate ? -Tracking the people who have the competence to replicate and take preemptive measures. -Restrictions on GPU/CPU/silicone wafer. Who can regulate? How can we regulate? What are the negative consequence of regulation? What happens if we don't, at what odds and time horizon?
hooloovoo_zoo 8y ago
I think you should at least release a small portion of the training data (e.g. anything recycling related) so people can measure to what extent the model is generating new sentences and to what extent it's just regurgitating training data.
cs702 8y ago
This seems very reasonable to me. All the outcry seems... disproportionate. That said, withholding the pretrained models probably won't make much difference, because bad actors with resources (e.g., certain governments) will be able to produce similar or better results relatively quickly. All it will take is (1) one or two knowledgeable people with the willingness to tinker, (2) a budget in the hundreds of thousands to a few millions of dollars at most, and (3) a few months to a year. Nowadays a lot of people are familiar with Transformers and constructing and training models across multiple GPUs.
smsm42 8y ago
> some of the samples generated by the model Mostly it's scary not because it's good - as writing goes, it's quite bad. It forms coherent sentences, but otherwise it's nonsense. I've seen similar nonsense producers in early 90s on basis of Markov chains and what not. No, the scary part is how much it reminds me of what I am reading in the media all the time. My current pet concern is that AIs will start passing the Turing test not because AIs are getting so good but because humans are getting so bad. A bunch of nonsensical drivel can easily be passed as a thoughtful analysis or a deep critical think-piece - and that's not my conjecture, have been repeatedly proven by submitting such drivel to various academic journals and it being accepted and published. I'm not saying people are losing critical thinking skills - but they are definitely losing (or maybe never even had?) the habit of consistently applying them.
- esjeon 8y ago
  > I've seen similar nonsense producers in early 90s on basis of Markov chains and what not. Exactly. When it comes to generating a large volume of apparently-good sentences, non-AI (or classical) approaches are still better than good. Those will be equally disruptive, since the defending side is yet to develop a proper countermeasure based on the "sensible"-ness of content. Plus, they will be much easier to customize and adapt to the situation, while ML-based solutions often need remodeling and retraining when repurposed. > My current pet concern is that AIs will start passing the Turing test not because AIs are getting so good but because humans are getting so bad AI will start deceiving the public even before it pass Turing test. It's much harder to spot bots amidst people than in a 1vs1 chatroom.
  - Kalavala93 8y ago
    > Exactly. When it comes to generating a large volume of apparently-good sentences, non-AI (or classical) approaches are still better than good. Can you cite your source? I find this hard to believe.
iamcreasy 8y ago
Hello Ilya. Great work. One of the reason Elon distanced himself because of what OpenAI team wanted to do. I am wondering if this new paper has anything to do with that? Or what it is in general that Elon doesn't agree with what OpenAI is doing? Thanks!
xg15 8y ago
> - If you accept the above, then the current AI norm of "publish everything always" will have to change Ok, accepting that premise, what people/organisations would you share the research with and based on what criteria?

modeless 8y ago

> The _whole point_ is that our model is not special and that other people can reproduce and improve Only people with a large amount of money and a lot of expertise. What you are doing is the opposite of democratizing AI.

moconnor 8y ago
Actually this shows why OpenAI matters. Google have been training and refining Transformer architectures for years; how unlikely is it nobody tried training a language model at this scale or larger with similar results? Yet from Google we heard nothing. Which is the optimal decision for them - they only lose by blowing the whistle.
- sgt101 8y ago
  A lot of people have results similar to this - but most people generating a paragraph of slightly_weird_but_plausible_if_you_read_quickly text using a primped version of BERT one time out of 25 regarded it as more or less pointless. But journalists don't. This would be ok if this is the first time that anyone had a media go wild over AI story. But actually this has happened 10000 times this year already.
  - skybrian 8y ago
    Seems like the way it worked is that the blog post was discussed here and on Twitter and many people thought it was interesting. Then some journalists picked it up and wrote about it. That much is nothing out of the ordinary. It is interesting (at least to those of us who aren't natural language researchers) so why shouldn't we talk about it? Why shouldn't journalists write about it? Inevitably their mildly controversial decision to hold some data back got a lot of people discussing whether it was necessary. Which is also perfectly okay. So, in the end, the complaint is just about why people don't have smarter takes on things. I don't know what to tell you; that's just how social media works sometimes.
    - sgt101 8y ago
      I'd shrug and move on, but the problem is that I believe that these flaps about AI are distracting attention from the real concerns and forces that are having a serious impact on people now. The distortion of public debate caused by community exclusiveness on social platforms, by the curation and manipulation of social feeds and by the dynamics of online debate where the loudest and angriest voices dominate is one place that we could do with some focus. Another place is the management of simple models - plain Jane stuff like a learned classifier - people are making these with Python and R and releasing them into infrastructures and apps and we don't know what they are and where they are and how they are interacting. Instead we have wizard of oz style stories to distract us from who's actually hiding behind the curtain. If we fall for this then we may find ourselves living in a vicious totalitarian society with no obvious way out of it. Journalists should write about it in an informed and professional way, that's fine. But they need to write about stories that are impactful and important, and if they were to write about this one in this way ("text scrambler makes a pretty good paragraph one out of 30 tries, has no idea of what is going on") they would get no clicks (there will now be a second wave of follow-ups like that to ride on the coattails of the story). Instead they have to make it sound like robots are going to take children from schools and experiment on them live on TV, and this makes them famous and rich. There is no real revision of the story because the follow on stories disappear from view while search engines and other journalists use the original hysteria. Look at what happened with the two negotiating bots at facebook (the game was to negotiate to get books and balls, the bots tended to use a short hand to negotiate rather than the english they were trained on) This was "Facebook researchers have to pull the plug on AI that they no longer understand", and that is the narrative that we will have on that story more or less forever.

sigil 8y ago

Ok but isn’t this the opposite of OpenAI’s “nukes are safer when multiple actors have them” strategy wrt AI? I’m also confused by the threat models earnestly put forth in your blog post. Are we really concerned about deep faking someone’s writing? The plain word already demands attribution by default: we look for an avatar, a handle, a domain name to prove the person actually said this.

albntomat0 8y ago
> Ok but isn’t this the opposite of OpenAI’s “nukes are safer when multiple actors have them” strategy wrt AI? It seems more like the "nukes are safer when multiple rational state level actors have them", rather than anyone able to pull a git repo.
- sigil 8y ago
  Yep. Maybe I misunderstood the subtler points of OpenAI’s “democratize AI” strategy, and this has been the plan all along. But AFAIK they haven’t put an “among a few rational state actors” asterisk on anything up until now. Regardless, I agree with TFA that this is a silly and arbitrary time to yell “fire.” It’s PR.
  - albntomat0 8y ago
    > But AFAIK they haven’t put an “among a few rational state actors” asterisk on anything up until now. True. On the PR side though, it'd be incredibly hard to say "we want to make replication moderately difficult, but not too difficult." Everyone would end up arguing exactly how much should be released, how it would prevent X,Y,Z folks from contributing to AI, etc. > Regardless, I agree with TFA that this is a silly and arbitrary time to yell “fire.” It’s PR. Alternatively, it does provide good insight into the reactions in the community as a whole, and continues the conversation on exactly how much should be released. Maybe I'm not far enough into the ML community, but the decision not to put the "keys to the kingdom" on github for every script kiddie to weaponize seems reasonable to me, especially as a precedent.
- Cacti 8y ago

malux85 8y ago

So a small number of individuals decided what's best for everybody? How is that open? How is that not centralization of power?

albntomat0 8y ago
If they did release it, there would be an equivalent outcry about how OpenAI was contributing to fake news, etc.
pas 8y ago
The paper is open https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...

onurcel 8y ago

Deciding if feeding the media with fear was worth the attention you will get wasn't easy, ha. Let me tell you, you are the shame of the profession.

pishpash 8y ago
Yep, put out handpicked samples to stoke fear, then release nothing of the internals to stoke more fear and act like self-appointed gods.

avip 8y ago

I've just read i.e https://twitter.com/gdb/status/1096098366545522688 https://twitter.com/gdb/status/1096098366545522688 and even though it's "best of 25" (I guess cherry-picked by a human) - this is mind-blowing. I am actually having a very hard time believing this is legit generated text.

mycorrhizal 8y ago
Definitely impressive work, but the fact that this is hard to distinguish from human text, if true, is pretty sad for humans. Even sadder if anyone reading this could be swayed by such an argument. Heck, maybe having to compete with this will raise human discourse (Joking).
- pakitan 8y ago
  It's impressive in terms of having a coherent flow - there is a clearly stated "opinion" in the beginning and everything that follows is in support of that opinion. However, the dead giveaway is that there is zero reasoning, just related statements linked together.
pas 8y ago
Why? It is pretty much a well juxtaposed mix of random internet comments. And it's the best of 25, which means the other 24 is even more regular internet banter noisy. (This of course doesn't make it an amazing feat of computer engineering.) The overarching narrative is great, but that's probably driven by the great antithesis supplied by the experimenter. It'd be interesting to know how this works, what happens if less or more is given as thesis/antithesis/assignment, and after how much output it turns into gibberish (or repeats).
imtringued 8y ago
I couldn't be more disappointed with this bullshit honestly. The texts have almost zero coherence and keep repeating the same patterns (which they presumably learned from the data set) over and over again. If this is their best out of 25 samples then they aren't going to fool anyone. >Recycling is NOT good for the world. >It is bad for the environment, >it is bad for our health, >and it is bad for our economy. >Recycling is not good for the environment. >Recycling is not good for our health. >Recycling is bad for our economy. >Recycling is not good for our nation. The first paragraph keeps repeating the <X> is <bad | not good> for the <Y> pattern 8 times. >And THAT is why we need to |get back to basics| and |get back to basics| in our recycling efforts. "get back to the basics" is repeated twice in the same sentence. >Everything from the raw materials (wood, cardboard, paper, etc.), >to the reagents (dyes, solvents, etc.) >to the printing equipment (chemicals, glue, paper, ink, etc.), >to the packaging, >to the packaging materials (mercury, chemicals, etc.) >to the processing equipment (heating, cooling, etc.), >to the packaging materials, >to the packaging materials that are shipped overseas and >to the packaging materials that are used in the United States. It literally repeated packaging 5 times in the same sentence and the overall structure was repeated 9 times. Also what type of packaging is based on mercury?

Cacti 8y ago

> - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. If this is your whole point, then I think you are missing something fundamental. Implementing these models doesn't require reflection, or introspection, or any sort of ethical or moral character whatsoever; and even if it did, all that will happen eventually is someone (without the technical background) will simply throw a lot of money at someone else (with the technical background, but who needs to, you know, eat, and pay rent, and so on) to implement it. You are fooling yourself if you think your stance makes a single mote of difference in this arms race.

pishpash 8y ago
Exactly. This is like holding up spam samples or how spammers operate from the spam detecting work. That side (and the cultural discussions) needs all the headstart it can get, not be complacent that some arbitrary "experts" will patronizingly "protect" them.
- roenxi 8y ago
  If you look at it as a PR stunt, it is almost certainly a good idea. If a bad actor can auto-generate text that is not really distinguishable from something written by a human, how does a community with open membership (eg, HN) protect itself? I imagine this technology will enable interesting new attacks against online communities; we havn't seen that for a while. OpenAI are extremely sensible to draw attention to the fact that AI is approaching a boundary that has practical implications. It is good that everyone is being alerted that that boundary might be crossed at any time in the foreseeable future.
  - pas 8y ago
    But ... it's not novel. We could already generate convincing gibberish years ago. Now the novelty is that this can be better targeted. But even simple Markov-chain based text generators were good enough to fool people for a bit. And there was always people that had too much free time to write. A lot. (See for example the crackpots and conspiracy theorists that bombard physics forums. See the 9/11, Zeitgeists, etc. movies. See how much has been written about anti-vaxx, about quantum woo, etc.) Reputation systems work pretty well for countering spammers. And against APTs (advanced persistent threats, spearfishing attacks, etc) there's no real "universal" protection anyways. (You need a competent security team to out think and out resource the attackers in every possible dimension.) This AI is the same as the paid Russian trolls and the unpaid scammers, and so on.

eslaught 8y ago

> - I suggest going over some of the samples generated by the model. Many people react quite strongly, e.g., https://twitter.com/justkelly_ok/status/1096111155469180928 https://twitter.com/justkelly_ok/status/1096111155469180928. Have you done a plagiarism search on that text to see how similar it is to the input corpus? I'm by no means an ML expert, but I've played around with models for random name generation and one thing I've noticed is that as the models become more accurate, they also become much more likely to just regurgitate existing names verbatim. So if you search the list of names and notice something that seems particularly realistic, it could be because it's literally taken in whole or in part from the training data set!

czr 8y ago
You're welcome to check out the samples [https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-samples.txt https://raw.githubusercontent.com/openai/gpt-2/master/gpt2-s...] and evaluate them for memorization yourself (I haven't found any so far). (The talking unicorn example on their page is also meant to demonstrate that, no, it's not just memorizing, but I think it's a bit more compelling to check from the raw samples)

GistNoesis 8y ago

What solutions are you proposing? Here are a few that comes to mind. -Secrecy? but how will you continue to exist on the PR scene if you don't release anything? -Are you willing to pay every developer who is able to replicate your paper, more than what the black market would pay? -How are you working on incentive alignment to make sure that all people who can replicate your results have more incentive to do good than bad, specially in the current environment where users and valuable data are silo-ed by a few companies? -Misdirection to keep an edge, i.e. planting bugs/ Not fixing bugs for public ; spreading false results; only working on problems that need high resources to limit the number of actor who will be able to replicate ? -Tracking the people who have the competence to replicate and take preemptive measures. -Restrictions on GPU/CPU/silicone wafer. Who can regulate? How can we regulate? What are the negative consequence of regulation? What happens if we don't, at what odds and time horizon?

hooloovoo_zoo 8y ago

I think you should at least release a small portion of the training data (e.g. anything recycling related) so people can measure to what extent the model is generating new sentences and to what extent it's just regurgitating training data.

cs702 8y ago

This seems very reasonable to me. All the outcry seems... disproportionate. That said, withholding the pretrained models probably won't make much difference, because bad actors with resources (e.g., certain governments) will be able to produce similar or better results relatively quickly. All it will take is (1) one or two knowledgeable people with the willingness to tinker, (2) a budget in the hundreds of thousands to a few millions of dollars at most, and (3) a few months to a year. Nowadays a lot of people are familiar with Transformers and constructing and training models across multiple GPUs.

smsm42 8y ago

> some of the samples generated by the model Mostly it's scary not because it's good - as writing goes, it's quite bad. It forms coherent sentences, but otherwise it's nonsense. I've seen similar nonsense producers in early 90s on basis of Markov chains and what not. No, the scary part is how much it reminds me of what I am reading in the media all the time. My current pet concern is that AIs will start passing the Turing test not because AIs are getting so good but because humans are getting so bad. A bunch of nonsensical drivel can easily be passed as a thoughtful analysis or a deep critical think-piece - and that's not my conjecture, have been repeatedly proven by submitting such drivel to various academic journals and it being accepted and published. I'm not saying people are losing critical thinking skills - but they are definitely losing (or maybe never even had?) the habit of consistently applying them.

esjeon 8y ago
> I've seen similar nonsense producers in early 90s on basis of Markov chains and what not. Exactly. When it comes to generating a large volume of apparently-good sentences, non-AI (or classical) approaches are still better than good. Those will be equally disruptive, since the defending side is yet to develop a proper countermeasure based on the "sensible"-ness of content. Plus, they will be much easier to customize and adapt to the situation, while ML-based solutions often need remodeling and retraining when repurposed. > My current pet concern is that AIs will start passing the Turing test not because AIs are getting so good but because humans are getting so bad AI will start deceiving the public even before it pass Turing test. It's much harder to spot bots amidst people than in a 1vs1 chatroom.
- Kalavala93 8y ago
  > Exactly. When it comes to generating a large volume of apparently-good sentences, non-AI (or classical) approaches are still better than good. Can you cite your source? I find this hard to believe.

iamcreasy 8y ago

Hello Ilya. Great work. One of the reason Elon distanced himself because of what OpenAI team wanted to do. I am wondering if this new paper has anything to do with that? Or what it is in general that Elon doesn't agree with what OpenAI is doing? Thanks!

xg15 8y ago

> - If you accept the above, then the current AI norm of "publish everything always" will have to change Ok, accepting that premise, what people/organisations would you share the research with and based on what criteria?