9 ms·

OpenAI Trains Language Model, Mass Hysteria Ensues

152 points by zackchase 8y ago

toufiqbarhamov 8y ago
Ms. Anandkumar nailed it, this is blatant hype bordering on hucksterism. Elon Musk May have left, but his influence remains I guess.
- namuol 8y ago
  First, it's clearly the goal of OpenAI to bring more public attention to advances in the field, specifically to help voters and policymakers consider potential ramifications well in advance of any "truly" groundbreaking work before it's too late. Of course they're "hyping" this technology. Secondly, have you seen the results? I was dumbfounded and fascinated. I spent hours reading the samples. Maybe I'm just out of the loop and this truly isn't anything significant, but then that only proves that OpenAI was successful: Now I am aware of the latest advances in NLP and hopefully so too are many more.
  - m0zg 8y ago
    I have seen the results and I don't get why people think this is any more dangerous than journalists who selectively report to fit a predetermined agenda or make shit up on the spot. Which, today, is a lot of them.
    - BoiledCabbage 8y ago
      It's Bulk. Same reason why spam is a problem.
      - m0zg 8y ago
        There's plenty of "bulk" coming from "journalists". On any given day a significant fraction of any popular news site is utter and complete horseshit, selectively reported or made up outright to fit a narrative, which will be retracted and/or forgotten weeks later. You could argue that this is _more_ dangerous because the bovine manure is mixed with the actual legit news stores to make it harder to tell which is which.
      - pakitan 8y ago
        The problem for spammers, as well as for fake news writers, has never been in coming up with the text for the spam email or the fake news story. This is already cheap and easy enough. The problem is with distribution and getting enough eyeballs. This new and so very dangerous AI may enable you to come up with 1M fake news stories with the click of a button but it won't get any of those stories published in NYTimes.
        m0zg 8y ago
        >> get any of those stories published in NYTimes I wouldn't be so sure about that. Take their reporting of Charlottesville events and Trump's comments about them. Here's what Trump _actually_ said: https://twitter.com/ZiaErica/status/1096572062196486144 https://twitter.com/ZiaErica/status/1096572062196486144. Pretty reasonable point of view, all things considered. What was NYTimes "reporting"? That Trump is "defending white supremacists", of course. Don't believe me? See for yourself: https://www.google.com/search?q=trump+charlottesville+nytimes&oq=trump+charlottesville+nytimes&aqs=chrome..69i57j69i60.14781j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=trump+charlottesville+nytime.... Why was NYTimes doing that? It's either deliberate malice or incompetence, both of which would make NYTimes quite friendly to automatically generated fake news as long as they fit their narrative. But there's a bigger issue with all of this. When people see this tech, they immediately think that it'll be used to generate fake news (which it will be, to be sure). BUT, it could also be used to do the exact opposite: take facts and summarize them without agenda-driven omissions, without "reading minds" or inventing "sources" "familiar with" someone's "thinking", or passing off uncorroborated dossiers or book chapters as gospel truth.
    - IanCal 8y ago
      You can't do that for near 0 cost though, nor generate a different story per user on the fly.
  - Barrin92 8y ago
    >Secondly, have you seen the results? I was dumbfounded and fascinated. I spent hours reading the samples. Yes, I've seen the result. They're nice but, as the article points out, not extraordinary compared to state of the art, open NLP research. OpenAI's behaviour here smells of Gibsonesque 'anti-marketing', using the misunderstanding of AI and its capabilities in the general population as a means to stir up publicity for their organisation. This is unethical, misrepresents progress in the field, and produces confusion in the press.
    - namuol 8y ago
      > not extraordinary compared to state of the art, open NLP research > misrepresents progress in the field Can you point me to some examples of unsupervised learning with similar results? Not asking for rhetorical purposes; I just genuinely was shocked by how compelling their results were, especially given this was unsupervised. > OpenAI's behaviour here smells of Gibsonesque 'anti-marketing' I don't disagree that the ethics are questionable, but I think it's highly speculative to suggest that they didn't release the full model purely as a marketing ploy (I'm assuming this is the main objection to their marketing "tactics"). As you say, it "smells" this way, but I fail to see how it's really so clear-cut.
      - Barrin92 8y ago
        >Can you point me to some examples of unsupervised learning with similar results? Not asking for rhetorical purposes; I just genuinely was shocked by how compelling their results were, especially given this was unsupervised. Model wise this is just openAI's GPT with some very slight modification (laid out in the paper). Ilya has now commented in the thread and essentially made the same point, this is state of the art performance, but reproducible by everyone because it uses a known architecture. The secrecy and controversy makes no sense if the model is open, even the methodology of data collection is laid out. There is no safety here assuming that anybody who wants to rebuild the model can do so simply by putting enough effort into rebuilding the dataset, which is not an issue for a seriously malicious actor.
        8y ago
Permit 8y ago
> Namely, he argued that OpenAI is concerned that the technology might be used to impersonate people or to fabricate fake news. This seems to be a particularly weak argument to make. How is their model going to impersonate someone in a way that a human can not?
- cma 8y ago
  Cheaper cost to put out a bigger volume of content.
  - Permit 8y ago
    Is volume really what dictates whether or not you can impersonate someone? It's never seemed that way to me.
    - hcs 8y ago
      It affects how many people you can impersonate, cheap enough for many authors with small readerships each. (I guess, it does seem overblown.)
    - pishpash 8y ago
      More to the point, is volume what counts as danger? All these deepfake risks boil down to online (for now) sock puppetry. We've been dealing with that for the whole life of the internet. The only reason it's even a problem in recent years is the growth of uninnoculated masses who haven't been on the internet that long, and positive feedback recommendation bots. That seems a qualitative issue not a quantitative one.
    - cma 8y ago
      It lets you impersonate a crowd, or various crowds.
      - pas 8y ago
        "PR firms" already have an army of fake/paid accounts on every important platform. This new AI could help them with that. They can let go of the paid writers and hire an IT guy/gal to operate the bots - and the VPNs. (Or they can just pay a lot less to the paid trolls just for their home ADSL/Cable/4G connection.) But so far this AI is not going to pass a Turing test. Sure, maybe it can be integrated with a chatbot. And it'll be interesting how internet communities will react.
amrrs 8y ago
>Elon Musk distances himself from OpenAI, group that built fake news AI tool This is the worst headlines in this matter. This is one of the leading media in India. A language model being touted as Fake news AI tool. This is like calling a car, A run over machine by Ford. https://www.hindustantimes.com/tech/elon-musk-distances-himself-from-openai-group-that-built-fake-news-ai-tool/story-Q3PkEU6fsJQVkhilpPCQ8M.html https://www.hindustantimes.com/tech/elon-musk-distances-hims...
- hjek 8y ago
  > This is like calling a car, A run over machine by Ford. That's a great dysphemism. Gonna start using that.
- ultrasounder 8y ago
  Hindustan Times is far from being a leading media outlet.it squarely falls in the same category as The National Enquirer of Jeff Bezos fame.
- robomc 8y ago
  That's not quite fair. The sample output they're touting is really nothing other than false text (when it's coherent), almost all of which is in the style of news. So for the Ford analogy to be apt, Ford would have to have designed a car nobody has ever seen, and released a video which is basically just hundreds of hours of the car running people over. I mean, a car has lots of well understood non-running-people-over capabilities. But have they demonstrated that this model is useful for anything other than generating fake news-sounding spam text?
agentofoblivion 8y ago
It’s amazing to me that no one has yet pointed out the blatant irony that their name is OpenAI, yet they are concealing far more than what is typical.
- zackchase 8y ago
  I assure you, people have pointed it out...
fareesh 8y ago
> Fictitious state of emergency Pretty dumb and disrespectful to politicize a blog post about OpenAI.
xiphias2 8y ago
Elon Musk was kicked out because he poached Andrej Karpathy from OpenAI to lead Autopilot. Anyways, it was worth it, Andrej is doing an amazing job, and OpenAI is still alive :)
- chrinic83 8y ago
  > Anyways, it was worth it, Andrej is doing an amazing job, and OpenAI is still alive :) Tesla does not even offer their full self driving package anymore. No coast to coast drive yet. Hard to say that's an amazing job. OpenAI abandons their open source GitHub repos after a year, is now not releasing code, and is always in DeepMind's shadow. Alive, yes. Successful, no.
  - xiphias2 8y ago
    Did you really expect Tesla to launch full driving? They started with being 5 years behind Waymo and without lidars or high resolution mapping, precise GPS sysyem that Waymo has...basically Elon wanted the impossible. At the same time Andrej dropped out the idea of a fully learned end-to-end model (that's just impossible with the current deep learning technology), and started replacing the somewhat working heuristics with machine learning methodically one-by-one. Also he ramped up the data gathering pipeline. He needs to build the full simulation, agent systems that can simulate other drivers/humans, implement reverse reinforcement learning...there's so much to do where Waymo is far ahead (but Tesla is ahead in data gathering).
SubiculumCode 8y ago
To what extent is this not just finding text samples written in its training sample and regurgitating it near verbatim?? -Non ml guy
- applecrazy 8y ago
  You bring up a good point. Without seeing their code and training metrics, how do we know that this isn’t some extremely overfitted model?
  - vedant 8y ago
    From the paper: "All models still underfit WebText and held-out perplexity has as of yet improved given more training time."
- HALtheWise 8y ago
  If you look at their paper, section 4 is entirely devoted to this question. They present compelling evidence that it is generating original content, the simplest of which is it's ability to write coherently about ridiculous things like talking unicorns that nobody has ever written about in the training set. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...
  - arcticfox 8y ago
    The talking unicorns piece was shockingly good. That is at least as coherent of a news story than the average human could easily invent about it. Reading that piece gives me the same weird feeling as watching AlphaStar playing through a StarCraft game.
itg 8y ago
Can you imagine if the teams that worked on the Internet decided not to make it available to the public because of the potential misuses. OpenAI is a joke.
ilyasut 8y ago
Ilya from OpenAI here. Here's our thinking: - ML is getting more powerful and will continue to do so as time goes by. While this point of view is not unanimously held by the AI community, it is also not particularly controversial. - If you accept the above, then the current AI norm of "publish everything always" will have to change - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. - I suggest going over some of the samples generated by the model. Many people react quite strongly, e.g., https://twitter.com/justkelly_ok/status/1096111155469180928 https://twitter.com/justkelly_ok/status/1096111155469180928. - It is true that some media headlines presented our nonpublishing of the model as "OpenAI's model is too dangerous to be published out of world-taking-over concerns". We don't endorse this framing, and if you read our blog post (or even in most cases the actual content of the news stories), you'll see that we don't claim this at all -- we say instead that this is just an early test case, we're concerned about language models more generally, and we're running an experiment. Finally, despite the way the news cycle has played out, and despite the degree of polarized response (and the huge range of arguments for and against our decision), we feel we made the right call, even if it wasn't an easy one to make.
- modeless 8y ago
  > The _whole point_ is that our model is not special and that other people can reproduce and improve Only people with a large amount of money and a lot of expertise. What you are doing is the opposite of democratizing AI.
  - moconnor 8y ago
    Actually this shows why OpenAI matters. Google have been training and refining Transformer architectures for years; how unlikely is it nobody tried training a language model at this scale or larger with similar results? Yet from Google we heard nothing. Which is the optimal decision for them - they only lose by blowing the whistle.
    - 8y ago
crobertsbmw 8y ago
How do we know this article isn’t just fake news being written by an AI?
Eliezer 8y ago
It seems disingenuous that this article fails to quote examples of GPT-2’s stunning results, or give any contrasting results from BERT to support the claim that this is all normal and expected progress. Like many, I was viscerally shocked that the results were possible, the potential to further wreck the Internet seemed obvious, and an extra six months for security actors to prepare a response seemed like normal good disclosure practice. OpenAI warned everyone of an “exploit” in which text humans can trust to be human-generated, and then announced they would hold off on publishing the exploit code for 6 months. This is normal in computer security and I’m taken aback at how little the analogy seems to be appreciated.
- pishpash 8y ago
  What's so shocking about this? Why do we trust this in the hands of a few self-appointed experts than anyone else? Are they supposed to be more moral than any others? What will security experts do in six months that wouldn't benefit from more security experts looking at it? Why do you care that garbage text is machine generated, from a spammer or influencer, or a mechanical turk? If it's volume you're concerned about, should we complain when search/recommendation engines already aggregate and reweight a tiny opinion into a continuous out-of-proportion stream that can last you a lifetime to consume? What is the practical difference to have more volume existing "out there"?
- pas 8y ago
  > Like many, I was viscerally shocked that the results were possible. Why? There were news about bots writing news ~5 years ago. Given a few simple facts the AI generated the regular info-scarce but fluffy news-piece. Now OpenAI added better everything (better language models, more data, better "long-term memory" for overall text coherence), and we got better fluff. It seems like a GAN and a simple Markov chain generator. (Even if it's not that simple of course.) And maybe it's the equivalent of the "modern art meme" style transferred to AI/ML research. ( https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c12676111d4--modern-art-kid.jpg https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c126... ) What I'm trying to convey is that wrecking the net with auto-trolls was already possible, but for some reason Mechanical Turk was cheaper. > OpenAI warned everyone of an “exploit” in which text humans can trust to be human-generated Sokal already did that, and so did http://thatsmathematics.com/mathgen/ http://thatsmathematics.com/mathgen/ ... but of course this might be qualitatively different, because it can be targeted. (Weaponized, if you will.) But the defense/antidote is the same, but it takes a lot more than 6 months to make people better at critical thinking, but maybe you already heard about the difficulties of that :)
kirillzubovsky 8y ago
What if OpenAI didn’t write the piece? What if the research was announced by the machine, and the folks at OpenAI are all dead?
- gfodor 8y ago
  You joke, but there's a real point here -- many commenters in this thread are complaining that OpenAI's position on this is a marketing stunt. Presumably, if this stuff gets commercialized, it will probably be adept at a few domains first, and I feel like writing good marketing copy will be one of them. So perhaps the bot itself didn't do so here, but it wouldn't surprise me if a self-marketed bot exists in the near future.
  - kirillzubovsky 8y ago
    What if I am the machine and you the last human left alive?
    - kirillzubovsky 8y ago
      You know how when we broke the Enigma we couldn't really let Germans catch onto it, so we had to mask our knowledge of their positions by maintaining statistically insignificant number of accidental wins? Much the same, a good AI should make deliberate typos.
  - kirillzubovsky 8y ago
    p.s. I was kidding, but I was completely serious. If they can train a machine to write good copy, they can train the best Russian bots to troll people on Facebook, write New York Times pieces, and fake and influence pretty much anything done through a written text. Heck, they could write a business book and get it into the top-10 that year. Actually, that last part, they should, it would be amazing!
bitL 8y ago
So an article about recycling generated by OpenAI model (best out of 25) already makes more sense than presidential speeches or most of ramblings of average politicians. Can we automate them away as well?
deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
sp332 8y ago
Does someone have a description of the network somewhere? Does it use LSTM for memory or what? Is there anything unusual about the size or structure of the network? Does it use an attention mechanism?
- czr 8y ago
  I would recommend reading the paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... and the previous paper https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://s3-us-west-2.amazonaws.com/openai-assets/research-co... It's a transformer, not LSTM, and it's very large but not structured in a particularly unusual way.
czr 8y ago
Many reactions across here / twitter / reddit seem totally out of proportion. And an odd mix of "stop acting so self-important, this research isn't special so you shouldn't have any qualms about releasing it" and "this research is super important, how dare you not release it". The strongest counterargument I've seen to OpenAI's decision is that the decision won't end up mattering, because someone else will eventually replicate the work and publish a similar model. But it still seems like a reasonable choice on OpenAI's part–they're warning us that some language model will soon be good enough for malicious use (e.g. large-scale astroturfing/spam), but they're deciding it won't be theirs (and giving the public a chance to prepare).
jph00 8y ago
In other fields such as infosec, responsible disclosure is a standard approach. You don't just throw a zero-day out there because you can. Whilst the norms for AI research needn't be identical, they should at least be informed by the history in related fields. The lead policy analyst at OpenAI has already tried to engage the community in discussing the malicious use of AI, on many occasions, including this extremely well-researched piece with input from many experts: https://maliciousaireport.com/ https://maliciousaireport.com/ . But until OpenAI actually published examples, the conversation didn't really start. In the end, there's no right answer - both releasing the model, and not releasing the model, have downsides. But we need a respectful and informed discussion about AI research norms. I've written more detailed thoughts here: https://www.fast.ai/2019/02/15/openai-gp2/ https://www.fast.ai/2019/02/15/openai-gp2/
mlboss 8y ago
I think OpenAI should change org name to ClosedAI.
rajacombinator 8y ago
It’s a great marketing hack. That’s the real accomplishment here.
Eli_P 8y ago
When a bug is caught on your palm, it pretends to be a dead bug. When a moose is scared, it plays dead moose. When AI wants to fool a human or a captcha filter, it impersonates a human. Only when a human wants to fool a human, it impersonates whatever possible but a human, then suddenly charges a shitload of ape shit, and then behaves like it never happened. Without a decent natural language translation or automatic reasoning, which they have not, looks like N-gram where N equals to number of words in language corpus.

152 points by zackchase 8y ago

Ms. Anandkumar nailed it, this is blatant hype bordering on hucksterism. Elon Musk May have left, but his influence remains I guess.

namuol 8y ago
First, it's clearly the goal of OpenAI to bring more public attention to advances in the field, specifically to help voters and policymakers consider potential ramifications well in advance of any "truly" groundbreaking work before it's too late. Of course they're "hyping" this technology. Secondly, have you seen the results? I was dumbfounded and fascinated. I spent hours reading the samples. Maybe I'm just out of the loop and this truly isn't anything significant, but then that only proves that OpenAI was successful: Now I am aware of the latest advances in NLP and hopefully so too are many more.
- m0zg 8y ago
  I have seen the results and I don't get why people think this is any more dangerous than journalists who selectively report to fit a predetermined agenda or make shit up on the spot. Which, today, is a lot of them.
  - BoiledCabbage 8y ago
    It's Bulk. Same reason why spam is a problem.
    - m0zg 8y ago
      There's plenty of "bulk" coming from "journalists". On any given day a significant fraction of any popular news site is utter and complete horseshit, selectively reported or made up outright to fit a narrative, which will be retracted and/or forgotten weeks later. You could argue that this is _more_ dangerous because the bovine manure is mixed with the actual legit news stores to make it harder to tell which is which.
    - pakitan 8y ago
      The problem for spammers, as well as for fake news writers, has never been in coming up with the text for the spam email or the fake news story. This is already cheap and easy enough. The problem is with distribution and getting enough eyeballs. This new and so very dangerous AI may enable you to come up with 1M fake news stories with the click of a button but it won't get any of those stories published in NYTimes.
      - m0zg 8y ago
        >> get any of those stories published in NYTimes I wouldn't be so sure about that. Take their reporting of Charlottesville events and Trump's comments about them. Here's what Trump _actually_ said: https://twitter.com/ZiaErica/status/1096572062196486144 https://twitter.com/ZiaErica/status/1096572062196486144. Pretty reasonable point of view, all things considered. What was NYTimes "reporting"? That Trump is "defending white supremacists", of course. Don't believe me? See for yourself: https://www.google.com/search?q=trump+charlottesville+nytimes&oq=trump+charlottesville+nytimes&aqs=chrome..69i57j69i60.14781j0j7&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=trump+charlottesville+nytime.... Why was NYTimes doing that? It's either deliberate malice or incompetence, both of which would make NYTimes quite friendly to automatically generated fake news as long as they fit their narrative. But there's a bigger issue with all of this. When people see this tech, they immediately think that it'll be used to generate fake news (which it will be, to be sure). BUT, it could also be used to do the exact opposite: take facts and summarize them without agenda-driven omissions, without "reading minds" or inventing "sources" "familiar with" someone's "thinking", or passing off uncorroborated dossiers or book chapters as gospel truth.
  - IanCal 8y ago
    You can't do that for near 0 cost though, nor generate a different story per user on the fly.
- Barrin92 8y ago
  >Secondly, have you seen the results? I was dumbfounded and fascinated. I spent hours reading the samples. Yes, I've seen the result. They're nice but, as the article points out, not extraordinary compared to state of the art, open NLP research. OpenAI's behaviour here smells of Gibsonesque 'anti-marketing', using the misunderstanding of AI and its capabilities in the general population as a means to stir up publicity for their organisation. This is unethical, misrepresents progress in the field, and produces confusion in the press.
  - namuol 8y ago
    > not extraordinary compared to state of the art, open NLP research > misrepresents progress in the field Can you point me to some examples of unsupervised learning with similar results? Not asking for rhetorical purposes; I just genuinely was shocked by how compelling their results were, especially given this was unsupervised. > OpenAI's behaviour here smells of Gibsonesque 'anti-marketing' I don't disagree that the ethics are questionable, but I think it's highly speculative to suggest that they didn't release the full model purely as a marketing ploy (I'm assuming this is the main objection to their marketing "tactics"). As you say, it "smells" this way, but I fail to see how it's really so clear-cut.
    - Barrin92 8y ago
      >Can you point me to some examples of unsupervised learning with similar results? Not asking for rhetorical purposes; I just genuinely was shocked by how compelling their results were, especially given this was unsupervised. Model wise this is just openAI's GPT with some very slight modification (laid out in the paper). Ilya has now commented in the thread and essentially made the same point, this is state of the art performance, but reproducible by everyone because it uses a known architecture. The secrecy and controversy makes no sense if the model is open, even the methodology of data collection is laid out. There is no safety here assuming that anybody who wants to rebuild the model can do so simply by putting enough effort into rebuilding the dataset, which is not an issue for a seriously malicious actor.
      - 8y ago

Permit 8y ago

> Namely, he argued that OpenAI is concerned that the technology might be used to impersonate people or to fabricate fake news. This seems to be a particularly weak argument to make. How is their model going to impersonate someone in a way that a human can not?

cma 8y ago
Cheaper cost to put out a bigger volume of content.
- Permit 8y ago
  Is volume really what dictates whether or not you can impersonate someone? It's never seemed that way to me.
  - hcs 8y ago
    It affects how many people you can impersonate, cheap enough for many authors with small readerships each. (I guess, it does seem overblown.)
  - pishpash 8y ago
    More to the point, is volume what counts as danger? All these deepfake risks boil down to online (for now) sock puppetry. We've been dealing with that for the whole life of the internet. The only reason it's even a problem in recent years is the growth of uninnoculated masses who haven't been on the internet that long, and positive feedback recommendation bots. That seems a qualitative issue not a quantitative one.
  - cma 8y ago
    It lets you impersonate a crowd, or various crowds.
    - pas 8y ago
      "PR firms" already have an army of fake/paid accounts on every important platform. This new AI could help them with that. They can let go of the paid writers and hire an IT guy/gal to operate the bots - and the VPNs. (Or they can just pay a lot less to the paid trolls just for their home ADSL/Cable/4G connection.) But so far this AI is not going to pass a Turing test. Sure, maybe it can be integrated with a chatbot. And it'll be interesting how internet communities will react.

amrrs 8y ago

>Elon Musk distances himself from OpenAI, group that built fake news AI tool This is the worst headlines in this matter. This is one of the leading media in India. A language model being touted as Fake news AI tool. This is like calling a car, A run over machine by Ford. https://www.hindustantimes.com/tech/elon-musk-distances-himself-from-openai-group-that-built-fake-news-ai-tool/story-Q3PkEU6fsJQVkhilpPCQ8M.html https://www.hindustantimes.com/tech/elon-musk-distances-hims...

hjek 8y ago
> This is like calling a car, A run over machine by Ford. That's a great dysphemism. Gonna start using that.
ultrasounder 8y ago
Hindustan Times is far from being a leading media outlet.it squarely falls in the same category as The National Enquirer of Jeff Bezos fame.
robomc 8y ago
That's not quite fair. The sample output they're touting is really nothing other than false text (when it's coherent), almost all of which is in the style of news. So for the Ford analogy to be apt, Ford would have to have designed a car nobody has ever seen, and released a video which is basically just hundreds of hours of the car running people over. I mean, a car has lots of well understood non-running-people-over capabilities. But have they demonstrated that this model is useful for anything other than generating fake news-sounding spam text?

agentofoblivion 8y ago

It’s amazing to me that no one has yet pointed out the blatant irony that their name is OpenAI, yet they are concealing far more than what is typical.

zackchase 8y ago
I assure you, people have pointed it out...

fareesh 8y ago

> Fictitious state of emergency Pretty dumb and disrespectful to politicize a blog post about OpenAI.

xiphias2 8y ago

Elon Musk was kicked out because he poached Andrej Karpathy from OpenAI to lead Autopilot. Anyways, it was worth it, Andrej is doing an amazing job, and OpenAI is still alive :)

chrinic83 8y ago
> Anyways, it was worth it, Andrej is doing an amazing job, and OpenAI is still alive :) Tesla does not even offer their full self driving package anymore. No coast to coast drive yet. Hard to say that's an amazing job. OpenAI abandons their open source GitHub repos after a year, is now not releasing code, and is always in DeepMind's shadow. Alive, yes. Successful, no.
- xiphias2 8y ago
  Did you really expect Tesla to launch full driving? They started with being 5 years behind Waymo and without lidars or high resolution mapping, precise GPS sysyem that Waymo has...basically Elon wanted the impossible. At the same time Andrej dropped out the idea of a fully learned end-to-end model (that's just impossible with the current deep learning technology), and started replacing the somewhat working heuristics with machine learning methodically one-by-one. Also he ramped up the data gathering pipeline. He needs to build the full simulation, agent systems that can simulate other drivers/humans, implement reverse reinforcement learning...there's so much to do where Waymo is far ahead (but Tesla is ahead in data gathering).

SubiculumCode 8y ago

To what extent is this not just finding text samples written in its training sample and regurgitating it near verbatim?? -Non ml guy

applecrazy 8y ago
You bring up a good point. Without seeing their code and training metrics, how do we know that this isn’t some extremely overfitted model?
- vedant 8y ago
  From the paper: "All models still underfit WebText and held-out perplexity has as of yet improved given more training time."
HALtheWise 8y ago
If you look at their paper, section 4 is entirely devoted to this question. They present compelling evidence that it is generating original content, the simplest of which is it's ability to write coherently about ridiculous things like talking unicorns that nobody has ever written about in the training set. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...
- arcticfox 8y ago
  The talking unicorns piece was shockingly good. That is at least as coherent of a news story than the average human could easily invent about it. Reading that piece gives me the same weird feeling as watching AlphaStar playing through a StarCraft game.

itg 8y ago

Can you imagine if the teams that worked on the Internet decided not to make it available to the public because of the potential misuses. OpenAI is a joke.

ilyasut 8y ago

Ilya from OpenAI here. Here's our thinking: - ML is getting more powerful and will continue to do so as time goes by. While this point of view is not unanimously held by the AI community, it is also not particularly controversial. - If you accept the above, then the current AI norm of "publish everything always" will have to change - The _whole point_ is that our model is not special and that other people can reproduce and improve upon what we did. We hope that when they do so, they too will reflect about the consequences of releasing their very powerful text generation models. - I suggest going over some of the samples generated by the model. Many people react quite strongly, e.g., https://twitter.com/justkelly_ok/status/1096111155469180928 https://twitter.com/justkelly_ok/status/1096111155469180928. - It is true that some media headlines presented our nonpublishing of the model as "OpenAI's model is too dangerous to be published out of world-taking-over concerns". We don't endorse this framing, and if you read our blog post (or even in most cases the actual content of the news stories), you'll see that we don't claim this at all -- we say instead that this is just an early test case, we're concerned about language models more generally, and we're running an experiment. Finally, despite the way the news cycle has played out, and despite the degree of polarized response (and the huge range of arguments for and against our decision), we feel we made the right call, even if it wasn't an easy one to make.

modeless 8y ago
> The _whole point_ is that our model is not special and that other people can reproduce and improve Only people with a large amount of money and a lot of expertise. What you are doing is the opposite of democratizing AI.
- moconnor 8y ago
  Actually this shows why OpenAI matters. Google have been training and refining Transformer architectures for years; how unlikely is it nobody tried training a language model at this scale or larger with similar results? Yet from Google we heard nothing. Which is the optimal decision for them - they only lose by blowing the whistle.
  - 8y ago

crobertsbmw 8y ago

How do we know this article isn’t just fake news being written by an AI?

Eliezer 8y ago

It seems disingenuous that this article fails to quote examples of GPT-2’s stunning results, or give any contrasting results from BERT to support the claim that this is all normal and expected progress. Like many, I was viscerally shocked that the results were possible, the potential to further wreck the Internet seemed obvious, and an extra six months for security actors to prepare a response seemed like normal good disclosure practice. OpenAI warned everyone of an “exploit” in which text humans can trust to be human-generated, and then announced they would hold off on publishing the exploit code for 6 months. This is normal in computer security and I’m taken aback at how little the analogy seems to be appreciated.

pishpash 8y ago
What's so shocking about this? Why do we trust this in the hands of a few self-appointed experts than anyone else? Are they supposed to be more moral than any others? What will security experts do in six months that wouldn't benefit from more security experts looking at it? Why do you care that garbage text is machine generated, from a spammer or influencer, or a mechanical turk? If it's volume you're concerned about, should we complain when search/recommendation engines already aggregate and reweight a tiny opinion into a continuous out-of-proportion stream that can last you a lifetime to consume? What is the practical difference to have more volume existing "out there"?
pas 8y ago
> Like many, I was viscerally shocked that the results were possible. Why? There were news about bots writing news ~5 years ago. Given a few simple facts the AI generated the regular info-scarce but fluffy news-piece. Now OpenAI added better everything (better language models, more data, better "long-term memory" for overall text coherence), and we got better fluff. It seems like a GAN and a simple Markov chain generator. (Even if it's not that simple of course.) And maybe it's the equivalent of the "modern art meme" style transferred to AI/ML research. ( https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c12676111d4--modern-art-kid.jpg https://i.pinimg.com/236x/71/e1/21/71e12151f4b59d8433d32c126... ) What I'm trying to convey is that wrecking the net with auto-trolls was already possible, but for some reason Mechanical Turk was cheaper. > OpenAI warned everyone of an “exploit” in which text humans can trust to be human-generated Sokal already did that, and so did http://thatsmathematics.com/mathgen/ http://thatsmathematics.com/mathgen/ ... but of course this might be qualitatively different, because it can be targeted. (Weaponized, if you will.) But the defense/antidote is the same, but it takes a lot more than 6 months to make people better at critical thinking, but maybe you already heard about the difficulties of that :)

kirillzubovsky 8y ago

What if OpenAI didn’t write the piece? What if the research was announced by the machine, and the folks at OpenAI are all dead?

gfodor 8y ago
You joke, but there's a real point here -- many commenters in this thread are complaining that OpenAI's position on this is a marketing stunt. Presumably, if this stuff gets commercialized, it will probably be adept at a few domains first, and I feel like writing good marketing copy will be one of them. So perhaps the bot itself didn't do so here, but it wouldn't surprise me if a self-marketed bot exists in the near future.
- kirillzubovsky 8y ago
  What if I am the machine and you the last human left alive?
  - kirillzubovsky 8y ago
    You know how when we broke the Enigma we couldn't really let Germans catch onto it, so we had to mask our knowledge of their positions by maintaining statistically insignificant number of accidental wins? Much the same, a good AI should make deliberate typos.
- kirillzubovsky 8y ago
  p.s. I was kidding, but I was completely serious. If they can train a machine to write good copy, they can train the best Russian bots to troll people on Facebook, write New York Times pieces, and fake and influence pretty much anything done through a written text. Heck, they could write a business book and get it into the top-10 that year. Actually, that last part, they should, it would be amazing!

bitL 8y ago

So an article about recycling generated by OpenAI model (best out of 25) already makes more sense than presidential speeches or most of ramblings of average politicians. Can we automate them away as well?

deleted 8y ago

[deleted]

deleted 8y ago

[deleted]

sp332 8y ago

Does someone have a description of the network somewhere? Does it use LSTM for memory or what? Is there anything unusual about the size or structure of the network? Does it use an attention mechanism?

czr 8y ago
I would recommend reading the paper: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... and the previous paper https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://s3-us-west-2.amazonaws.com/openai-assets/research-co... It's a transformer, not LSTM, and it's very large but not structured in a particularly unusual way.

czr 8y ago

Many reactions across here / twitter / reddit seem totally out of proportion. And an odd mix of "stop acting so self-important, this research isn't special so you shouldn't have any qualms about releasing it" and "this research is super important, how dare you not release it". The strongest counterargument I've seen to OpenAI's decision is that the decision won't end up mattering, because someone else will eventually replicate the work and publish a similar model. But it still seems like a reasonable choice on OpenAI's part–they're warning us that some language model will soon be good enough for malicious use (e.g. large-scale astroturfing/spam), but they're deciding it won't be theirs (and giving the public a chance to prepare).

jph00 8y ago

In other fields such as infosec, responsible disclosure is a standard approach. You don't just throw a zero-day out there because you can. Whilst the norms for AI research needn't be identical, they should at least be informed by the history in related fields. The lead policy analyst at OpenAI has already tried to engage the community in discussing the malicious use of AI, on many occasions, including this extremely well-researched piece with input from many experts: https://maliciousaireport.com/ https://maliciousaireport.com/ . But until OpenAI actually published examples, the conversation didn't really start. In the end, there's no right answer - both releasing the model, and not releasing the model, have downsides. But we need a respectful and informed discussion about AI research norms. I've written more detailed thoughts here: https://www.fast.ai/2019/02/15/openai-gp2/ https://www.fast.ai/2019/02/15/openai-gp2/

mlboss 8y ago

I think OpenAI should change org name to ClosedAI.

rajacombinator 8y ago

It’s a great marketing hack. That’s the real accomplishment here.

Eli_P 8y ago

When a bug is caught on your palm, it pretends to be a dead bug. When a moose is scared, it plays dead moose. When AI wants to fool a human or a captcha filter, it impersonates a human. Only when a human wants to fool a human, it impersonates whatever possible but a human, then suddenly charges a shitload of ape shit, and then behaves like it never happened. Without a decent natural language translation or automatic reasoning, which they have not, looks like N-gram where N equals to number of words in language corpus.