11 ms·

Dear OpenAI: Please Open Source Your Language Model

289 points by hughzhang 8y ago

akyu 8y ago
My conspiracy theory on this: This entire fiasco is actually an experiment to gauge public reaction to this kind of release strategy. Eventually the point will come where the choice of releasing a model may have real ethical considerations, so they are test driving the possibilities with a relatively harmless model. Given the, in my opinion, huge overreaction to all of this, I fear this may only encourage AI research groups to be more secretive with their work.
- tomphoolery 8y ago
  OpenAI is just trying to make a buck. Just like most other programmers, they don't give a damn about how damaging their code is going to be.
  - minimaxir 8y ago
    OpenAI is literally a non-profit.
    - nightski 8y ago
      Not saying I agree with the parent but while OpenAI is non-profit it's owned by billionaires whose companies directly benefit from their research.
      - sterlind 8y ago
        Perhaps they kept it closed source because they sold it to one of those companies? Non-profits are allowed to sell things, after all.
    - vokep 8y ago
      Do you think that means they don't get paid? Non-profit itself isn't a super meaningful term as far as if profits are being made or not. To be a non-profit, many hoops have to be jumped through, certain forms of profit have to be zero (I don't know the specifics, but a friend of mine was considering starting a non-profit at one point) but the employees are still paid.
    - peterwwillis 8y ago
      You can literally make millions of dollars working for non-profits. The highest paid CEOs make several million dollars per year and the biggest non-profits have annual budgets in the billions. The annual revenue from all non-profits in the US is in the trillions.
      - jstarfish 8y ago
        The NFL is a nonprofit organization too...
  - pmichaud 8y ago
    This is just false. Cynicism isn't a substitute for knowledge.
- phiresky 8y ago
  That's pretty much what they are publicly saying, right? > This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas.
  - akyu 8y ago
    Ah I missed that.
- currymj 8y ago
  they acknowledged even in the original blog post that any well-funded group of NLP researchers would be able to replicate their work within a few weeks/months (including whatever corporation or state or terrorist group you worry about), and that in terms of methodology it is a natural, incremental improvement over existing techniques. so it's sort of obvious that withholding release can't prevent any harm. the good faith reason to withhold release is as you say -- start a conversation now about research norms so researchers have some decision making framework in case they come up with something really surprisingly dangerous. the bad faith reason is it gives them great PR. this was surely part of the motivation, but I bet OpenAI didn't quite expect the level of derangement in the articles that got published, and may regret it a little bit.
  - Cybiote 8y ago
    Can we even say for certain it's an improvement over say something like TransformerXL? As far as I could see, the changes over GPT were a couple extra and tweak to layer normalizations, a small change to initialization and a change to text pre-processing. Other than for pre-processing, I didn't catch anything on theoretical motivations for these choices nor anything on ablation studies. The only thing that can be said for certain is it used lots of data and a very large number of parameters, trained on powerful hardware and achieved unmatched results in natural language generation.
- vokep 8y ago
  From their blog post: >These samples have substantial policy implications: large language models are becoming increasingly easy to steer towards scalable, customized, coherent text generation, which in turn could be used in a number of beneficial as well as malicious ways. We’ll discuss these implications below in more detail, and outline a publication experiment we are taking in light of such considerations. >a publication experiment we are taking in light of such considerations. So I think you're correct, they aren't really too scared about this one, but they realize they might be soon. Seeing how this turns out (how quickly its replicated, how quickly someone makes a wrapper for it that you can download and run yourself in minutes) will inform for more serious situations in the future.
- bitL 8y ago
  When you have whole teams doing "AI safety" at FANG, what do you think would be a logical output of their work? Wouldn't it correlate with what we see from OpenAI?
minimaxir 8y ago
It's worth noting that there's a surprising difference between the full model (1542M hyperparameters) and the released smaller model in the repo (117M): the full model can account for narrative structure much better (https://twitter.com/JanelleCShane/status/1097652984316481537 https://twitter.com/JanelleCShane/status/1097652984316481537), while the smaller model tends to go on trainwrecks, especially if you give it a bespoke grammatical structure (https://twitter.com/JanelleCShane/status/1097656934180696064 https://twitter.com/JanelleCShane/status/1097656934180696064) The arguments OpenAI have given about people using the model to create fake news are IMO not good; bad actors will still create fake news anyways (or with the paper/repo, they might even create more bespoke models trained on a specific type of dataset for even better fake news). With the open model, we would know how to better fight such attacks.
- make3 8y ago
  plus they have opened source the code, and the training cost of the full model is evaluated at 45k$, which is not much for a state actor or a serious org of any kind
  - Jach 8y ago
    $45k is "new car" money, plenty of individuals on this site could afford it if they wanted. Don't even have to be that serious about it.
  - buboard 8y ago
    what would they do with it? It is a text decorator, it can't make sense of itself. maybe they can make 1000 fake facebook accounts that post a lot. still, i doubt facebook uses text to detect bots, its easier to use ip/network patterns.
    - make3 8y ago
      it's not a text decorator, it's a text generator
      - 8y ago
pmichaud 8y ago
I really strongly disagree with this. I don't have much time to write, but: 1. Photo manipulation has been extremely destructive in a variety of ways. People "know" that photoshop is a thing, yet fake pictures abound at all levels of publications, and unrealistic standards propagate at full speed nonetheless. The effects are wide spread and insidious. A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize would be a devastating blow. People "knowing" that it was possible would do precious little to stop the onslaught, and it all but destroy online discourse in any open channels. I'm certain there would be other consequences that I can't fathom right now because I don't grok the potential scale or nth order effects. 2. There is no fire alarm for AGI[1]. Is this thing the key to really dangerous AI? Will the next thing be? No one knows. It's better to be conservative. [1] https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/
- hughzhang 8y ago
  Thanks for your comments! On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. I'm sure I've been tricked by a whole host of random facts which are probably actually BS based on random stuff I've read online. On the other hand, if "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin, I think this is basically impossible. Given widespread knowledge of photoshop, changing someone's mind about something significant with a doctored photo seems difficult (it is hard enough to change someone's mind with the truth!). Doctored photographs aren't completely harmless in that they can reinforce existing held beliefs (confirmation bias) and random fake facts aren't totally harmless either, I just think that the danger here is far below what has been implied in other places. In a nutshell, the effects are indeed wide spread, but perhaps not so insidious. On 2) there being no fire alarm, I'm actually very thankful to OpenAI for raising this discussion. While I disagree with their decision not to open source, this discussion was certainly worth having.
  - wallflower 8y ago
    > On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. No, I think it is even more insidious what they were referring to. I think they meant that Photoshop has enabled and continues to enable unrealistic physical standards of beauty and BMI (mostly for women) that are transmitted in advertisements and news articles. In other words, Photoshop has enabled an alternative visual reality that is not realistic.
sytelus 8y ago
This is very nice summary of lot of social media discussion that has been happening. Very thoughtful arguments from people who actually work on this stuff.
fossuser 8y ago
I thought this blog post response was pretty good: OpenAI’s GPT-2: the model, the hype, and the controversy https://towardsdatascience.com/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8 https://towardsdatascience.com/openais-gpt-2-the-model-the-h... I think this is a better response than the article submitted here. There are ethical concerns and risk with these releases and it's probably a good idea to start considering them. What they did release seems reasonable in context. I find the argument that making the technology as open as possible leading to people taking a skeptical look at things not very convincing. Recent election interference, disinformation campaigns, and the general inability for the public to disambiguate fact from fiction seems like decent evidence that just because people know something can be faked doesn't lead to critical thinking - confirmation bias is strong.
- buboard 8y ago
  as an FOSSuser you should know that many parts of opensource software used to be outright illegal (e.g. pgp). opensource democratized all of them
wodenokoto 8y ago
Not releasing the model seems to me to be entirely a PR stunt, and has made me lose a lot of respect for OpenAI.
kogir 8y ago
Everything important is open sourced already. The article cites replication as important and then paradoxically argues for releasing the model in lieu of replicating the training and verifying the result. This is the AI equivalent of a pharmaceutical company releasing all their research and patents and then taking heat for not providing reagents and doing the organic synthesis for everyone too.
- minimaxir 8y ago
  The difference is a pharmaceutical company is for-profit, while OpenAI's (a non-profit) mission (https://openai.com/about/ https://openai.com/about/) is "to build safe AGI, and ensure AGI's benefits are as widely and evenly distributed as possible." (emphasis mine)
  - kogir 8y ago
    I don’t see how corporate structure changes anything. It’s a huge gift of tons of work, which is far and away the hard part. They’ve enabled literally anyone who wants to to train their own model on whatever source material they want.
new_guy 8y ago
This is the best publicity they could ever have and it didn't cost them a penny. There's probably 10 year olds in their bedrooms light years ahead of these guys, certainly from we've seen it's definitely nothing to write home about, but it's all about the marketing and hype, convincing the mass media they've invented the next Terminator.
- leesec 8y ago
  This is crazy. It is a substantial step forward for NLP and basically every researcher/important person in the field agrees. And hype for what? They aren't selling anything.
  - buboard 8y ago
    they are selling hype. i also see it as a jab to fb/google "look boys, we re gonna keep this nuclear bomb from filling your databases with spam"
- newen 8y ago
  > There's probably [amateurs] light years ahead of these guys This is not an 80s hacker movie and amateur hackers can get into the DoD computers using a public phone or whatever. This is actual research funded with millions of dollars done by people with years of experience in the field. I find this whole meme in the programming world of some random programmer being able to work miracles and be super advanced kind of dumb.
  - carapace 8y ago
    Is it a meme in the programming world? I've only encountered it in Hollywood and TV. It's like how every on-screen computer beeps and boops on every interaction: it's a dramatic device. No one does that IRL because it would drive you bananas in like ten seconds. Cf. Glyph's "The Television Writer's Guide to Cryptography" https://glyph.twistedmatrix.com/2009/01/television-writer-guide-to-cryptography.html https://glyph.twistedmatrix.com/2009/01/television-writer-gu...
cartercole 8y ago
release the code! you cant stop the signal
- minimaxir 8y ago
  The code is out (https://github.com/openai/gpt-2 https://github.com/openai/gpt-2), just not the larger model itself.
  - gwern 8y ago
    Saying 'the code is out' is a little misleading. It's some supporting code for working with a trained model, and a model definition of what is already a standard and widely-implemented architecture. It doesn't include the dataset, the code used to construct the dataset, or perhaps most importantly of all, the code to train a model on a fleet of TPUv3s.
- kogir 8y ago
  They already released the code and that wasn’t enough for people.
  - i_cant_speel 8y ago
    It's like releasing a car without giving access to the specific fuel needed to run it.
    - minimaxir 8y ago
      The smaller model is out so you can test drive it, but you won't be able to hit 100mph.
gambler 8y ago
Fortunately, we have outstanding and trustworthy corporate citizens like Google, Facebook, Uber and others that will continue AI research safely, ethically and responsibly without the risk of such technologies falling into the unsteady hands of the public.
boltzmannbrain 8y ago
Req'd reading: "OpenAI Trains Language Model, Mass Hysteria Ensues" by CMU Professor Zachary Lipton, http://approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/ http://approximatelycorrect.com/2019/02/17/openai-trains-lan... A salient quote from the article: > However, what makes OpenAI’s decision puzzling is that it seems to presume that OpenAI is somehow special—that their technology is somehow different than what everyone else in the entire NLP community is doing—otherwise, what is achieved by withholding it? However, from reading the paper, it appears that this work is straight down the middle of the mainstream NLP research. To be clear, it is good work and could likely be published, but it is precisely the sort of science-as-usual step forward that you would expect to see in a month or two, from any of tens of equally strong NLP labs.
- eanzenberg 8y ago
  Suffice to say, most of what comes out of OpenAI is vanilla type work that I haven't seen go beyond academic research labs. They do spend a lot of time on PR and making stuff look pretty, I guess.
  - tedivm 8y ago
    To be fair they've also released some useful tools, such as the AI Gym.
    - ipsum2 8y ago
      To be fair, all of their useful tools have been deprecated or in 'maintenance mode'. See https://github.com/openai/gym https://github.com/openai/gym
- rojobuffalo 8y ago
  They list several tests with record breaking results here: https://blog.openai.com/better-language-models/#zeroshot https://blog.openai.com/better-language-models/#zeroshot Are those previous records their own or were those the best scores in the whole field of NLP? It sure looks like a pretty big step forward for one model to break that many records.
buboard 8y ago
The author doesnt mention the FUD that is being spread through tech media. "AI company is so afraid of its creation, it won't release its source code". Good job helping people understand their AI future. This is a PR stunt. Come on, a little extra spam is not shocking news, and whatever can be used to spread "fake news" can also be used to spread "correct news". Whatever the PR stunt they tried to pull, it is creating a bad narrative for AI.
ForrestN 8y ago
I’m agnostic about the short term effects of OpenAI’s decision, but I wonder if and hope that the eventual equilibrium will look more like things did before the internet. In, say, the 1960’s, if someone stood on a street corner with a sign saying “The End Is Nigh,” most people would react with a shrug. They’d know that if the world were ending they’d probably hear about it from a trusted source like a newspaper or broadcaster. I’m not worried about being tricked by fakes because I know that the NYT will be doing infinitely more than I ever could to verify hugely important facts, and won’t overreact to random bits of media. If you’re not someone I know personally, and you’re not a trusted source of public information, you may as well be holding a sign on a street corner, no matter how realistic the image or text on your sign. I think there may be vast unintended consequences that have to do with scale, but intuitively it doesn’t feel like the “who to trust” problem is more than a matter of cultural behaviors around trust adjusting to the internet age.
- jpttsn 8y ago
  Yeah, if this tech was the WMD that OpenAI says it is, the NYT would know about it.
nisten 8y ago
I don't understand why the choice of opensourcing seems to be so binary for some. It feels as if it's either: A-Leave everything open and hope that the global interoperability and cathartic flow of information creates a magic utopia for humanity. B-Everyone incorporates into the soviet-union/chinese-firewall for protection. There are manageable, more analogue alternatives too, like reducing the risk and probability of a disaster by trusting a human's judgement to limit access to potentially malicious code; which is exactly what they're doing in this case.
rippeltippel 8y ago
<quote>Precisely because everyone knows about Photoshop.</quote> Everyone?! All the non-digital-aware people on this planet, which I assume is the majority, may have never heard about Photoshop, or know what can be done with it. Out of the remaining digital-aware human beings who may well know about Photoshop, how many think about it when seeing a photo. I bet it's a very tiny percentage.
yongjik 8y ago
I don't buy the argument that open-sourcing would allow citizens to better fight this. Let's be realistic. If the full model is published, for every one person who studies the model to understand how to fight false propaganda, there will be 5,000 who use it to shitposting for lulz. Also, as everyone's pointing out, the cat's out of the bag. Whatever OpenAI does, the same ability will be in the hands of state actors, and large corporations, and soon not-too-large corporations and dedicated individuals, probably in a few years. So basically whatever OpenAI does today doesn't really matter. We might as well think about totally different ways of building online communities when everyone has shitpointing sockpuppet engines powered by AWS/Azure DeepLearningToolKit Trial Version.
- dannyw 8y ago
  Or a neural net for detecting fake text. How insane would it be if the birth of AGI comes from GANs of fake news?
deleted 8y ago
[deleted]
robrenaud 8y ago
What's extremely bad for research is that they didn't even release their train and test sets. It's not hard to grab 40GB of text from the web that comes from links with at least +3 votes on reddit. But even if you do that, you won't get the same train/test set, and the same split. So impossible to know if a model is outperforming their model.
- sanxiyn 8y ago
  They tested against standard language modeling benchmarks like PTB and WikiText, so it's entirely possible to compare.
peterwwillis 8y ago
A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again.
bitL 8y ago
Sooner or later we will have tools for e.g. Reddit where one specifies a desired outcome of a discussion and lets bots paired up with some optimization algorithm figure out how to get popularity majority based on comments so far in individual threads and drive discussion towards stated goal. Later the same will be used in elections or other public "theaters" that won't matter any longer as humanity will be "cracked", i.e. modeled sufficiently enough to predict/influence average humans with just tiny remains of feelings that something might be wrong which only complete societal outcasts would pursue.
mark_l_watson 8y ago
I like the work OpenAI has done in the past, so please take my complaints as gentle complaints: - a very small percentage of generated samples ‘read well’ - I Don’t see any evidence of ‘understanding’, at least in the sense that BERT can identify original nouns that pronouns refer to (anaphora resolution): amazing to get state of the art results - the ‘we aren’t going to share code and results’ thing reminds me of the movie The Wizard of Oz: trust them that they developed cool technology - in 6 months, high school students might be getting similar results, given how fast our field is moving forward: this makes the danger concern less important in my opinion This whole thing seems like a throwback to how science was conducted 10 years ago - ancient times in our fast moving tech world.
- msla 8y ago
  > the ‘we aren’t going to share code and results’ thing reminds me of the movie The Wizard of Oz: trust them that they developed cool technology And this is where it stands: They claim to have something. They won't release it. If they had nothing but a somewhat creative streak to make up the results, the world would look exactly the same as it would if they had something. Them lying is the least hypothesis here.
  - unimpressive 8y ago
    Given that experts in the field seem to think this is an incremental step forward, OpenAI has already demonstrated a track record of doing things probably more impressive than this, and the massive loss of credibility that would come with being caught in a lie here, on priors we should expect they're not lying.
baron_harkonnen 8y ago
Why is there so much focus on just the language generation aspect of this model? The language generation examples are to show how well the model generalized the language it's trained on, but the whole point of releasing the pretrained model is that this model also: >achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training Every comment I see here and else where is strictly about automatically generating comments, but the real loss of having this model kept private is that it would allow anyone to create very, very good NLP projects right away. The big problem in ML/"AI" now and going forward is that there is an extreme asymmetry emerging in tech. The start of the last major tech boom was because of "disruption". Any clever hacker with a laptop and developer skills could create a product that would topple a giant. Likewise anyone could learn to do anything that needed to be done on their own computer in their own time. Now that's changing, to even play with these new tools you need special access and corporate support. Releasing the pretrained model to the public would allow a range of groups that cannot afford to build a model such as this to experiment on their own and achieve state of the art NLP results, not just in language generation but in classification, translation, summarization etc. Some kid in the middle of know where could download this and start messing with ideas about building a chat bot. The threat is not that some rogue actor will create evil spam bots, but that some small scrappy startup would leverage the same, currently exclusive tools, of large technology companies. The reason why democratizing AI is important is not because we can better fight off some vague threat, but so that, at least for the near term, individuals and small groups can still reasonably compete with major players in the ML space. And of course the fact that someone who claims openness and that they exist to "ensure AGI's benefits are as widely and evenly distributed as possible." Can arbitrarily choose not to do go the other direction just shows that tech is moving in a direction where "disruption" is going to become increasingly less possible, and the future of the industry is going to be controlled by a handful of major players.
doublekill 8y ago
Another thing that has not been discussed yet: OpenAI does not want to be responsible for the output of this model. Can you imagine the headlines? "OpenAI released a text generating AI and it is racist as hell". People should have learned their lesson after the Tay fiasco. I am ambiguous on the issue of release vs. non-release, but the mockery and derision they faced makes me ashamed to contribute to this field. No good faith is assumed, but projection of PR-blitzes and academic penis envy. Perhaps AI researchers are simply not the best for dealing with ethical and societal issues. Look at how long it took to get decent research into fairness, and its current low focus in industry. If you were in predictive modeling 10 years ago, it is likely you contributed to promoting and institutionalizing bias and racism. Do you want these same people deciding on responsible disclosure standards? Does the head of Facebook AI or those that OK'd project Dragonfly or Maven have any real authority on the responsible ethical use of new technology? I am not too sure about the impact of a human-level text generating tool. It may throw us back to the old days of email spam (before Bayesian spam filters). It is always easier to troll and derail than it is to employ such techniques for good. Scaling up disinformation campaigns is a real threat to our democracies (or maybe this decade-old technique is already in use at scale by militaries, and this work is merely showing what the AI community's love for military funding looks like). I am sure that the impact of the NIPS abbreviation is an order of magnitude lower than that of this technology, yet companies like NVIDIA used Neurips in their marketing PR before it was officially introduced (made them look like the good guys for a profit). How is that for malaligning ML research for PR purposes? Would the current vitriol displayed in online discussons be appreciated when there was a name change proposal for the betterment of society? Disclaimer: this comment in favor of OpenAI was written by a real human. Could you tell for sure now you know the current state of the art? What would these comment sections look like if one person controls 20% of the accounts here?
- jstarfish 8y ago
  > It is always easier to troll and derail than it is to employ such techniques for good. Curious-- what possible good comes with the ability to generate grammatically correct text devoid of actual meaning? Sure, you could generate an arbitrary essay, but it's less an essay about anything and more just an arrangement of words that happen to statistically relate to each other. Markov chains already do this and while the output looks technically correct, you're not going to learn or interpret anything from it. Same goes with things like autocomplete. You could generate an entire passage just accepting its suggestions. It would pass a grammar test but doesn't mean anything. Chatbots are an obvious application, but how is that "good," or any different than the Harlow experiment performed on humans? Fooling people into developing social relationships with an algorithm (however short or trivial) is cruel and unethical beyond belief. A photoshopped image might be pleasant or stimulating to look at, and does have known problems in terms of normalizing a fictitious reality. But fake text? What non-malicious use can there possibly be?
angel_j 8y ago
They did release the model [0]. A smaller-scale version of it. If you want the big one, all you have to do is scrape 45M web pages, and increase the parameters to scale. What they didn't release is the training corpus and the trained filters. By not releasing these, and focusing on ethics, they are trying to avert attention from the mistake of training a massive cyclops on unfiltered internet text. They probably read a lot of embarrassing text they didn't share, imagine that. 0. https://github.com/openai/gpt-2/ https://github.com/openai/gpt-2/
scottlocklin 8y ago
Yes +1; mostly so we can laugh at how much your PR team exaggerated your results.
darklycan51 8y ago
The fact that this thread is in the site is scary, seems like there are parties interested in a public outcry to make some potentially very harmful tools be released to the public... the question is, why?
PaulHoule 8y ago
Network architectures and trainers are improving quickly so it won't be long before others can create a similar model. Their ace is having a huge clean dataset of grammatically correct text. This is not about the fine points of grammar, but that the content of the average web page is practically word salad.
aaron695 8y ago
It's an obvious lie that there's an issue to release it. Can anyone explain what it might do bad, WITHOUT hand waving? What? Blog spam to attack Googles algorithm? We can already attack it using markov chains, they simply use IP's and incoming links and other tricks to fight back. You can read the examples, they are meaningless to humans obviously, the intelligence to create a article that's intelligent would be sentient AI. Unlike the human photo creating AI which actually is an issue. A non handwaving example why it's bad, people can quickly Tinyeye a photo to see if it's fake or not. With this AI they cannot. Fresh photos take real world people and money, people have learned they add authenticity because of this, while people relearn AI means this is no longer true they are open to attack. A jumble of words.... we've all seen before. (And the researchers are using humans to pick the best jumble of words so it still takes a skilled human = $)
paulsutter 8y ago
Are they planning to change their name to ClosedAI?
bllguo 8y ago
To me it feels like everything about OpenAI's media strategy is calculated and specifically designed to maximize good PR with the public. It leaves a sour taste in my mouth.
- wrsh07 8y ago
  It's very weird to me that a team of researchers who are credibly doing good work and are making an effort to clearly explain their research garners such suspicion. Are you similarly skeptical of Deepmind, Distill, or Two Minute Papers? Honestly, the norm of providing two versions of your research (arxiv and blog post) makes ML easier to follow and is something I would like more academics to attempt
n1231231231234 8y ago
1) What really bothered me personally about GPT2 is that they made it look sciency by putting out a paper that looks like other scientific papers -- but then they undermine a key aspect of science: reproducability/verifiability. I struggle to believe 'science' that cannot be verified/replicated. 2) In addition to this, they stand on the shoulder of giants and profit from a long tradition of researchers and even companies making their data and tools available. but "open"AI chose to go down a different path. 3) which makes me wonder what they are trying to add to the discussion? the discussion about the dangers of AI is fully ongoing. by not releasing background info also means that openAI is not contributing to how dangerous AI can be approached. openAI might or might not have a model that is closer to some worrisome threshold. but we don't know for sure. so imv, what openAI primarily brought to the discussion are some vague fears against technological progress -- which doesn't help anyone.
- sanxiyn 8y ago
  Re 1: GPT2 is no different from most stuffs by DeepMind. DeepMind, in general, does not release code, data, or model. DeepMind does not seem to get reproducibility complaints, supposedly "key aspect of science".
namuol 8y ago
Here's the most thoughtful piece I've been able to find on the controversial topic so far: https://www.fast.ai/2019/02/15/openai-gp2/ https://www.fast.ai/2019/02/15/openai-gp2/
jayantj 8y ago
I strongly disagree with OP's logic when he says - Photoshop hasn't negatively affected the world "precisely because everyone knows about Photoshop." - the crux of his argument. IMO the main reason Photoshop hasn't affected the world negatively is because it is HARD to make convincing fakes - precisely what OpenAI are ensuring by not releasing the model. Notice that this reasoning doesn't take into account whether or not GPT-2 can actually negatively impact the world. It says as long as people are aware that the text they are reading could be fake/AI-generated, we'll be fine. I think people are already aware of that. I don't see how releasing the pre-trained model will help with that.
Gatsky 8y ago
The argument in the article is unconvincing, and shows a distinct failure to think on a societal level or on time lines beyond a few weeks. The calculus seems simple to me. The potential harms from not open sourcing the model completely - minimal. Potential harms from open sourcing it completely - greater than minimal. Hence their decision. OpenAI should be lauded for making this decision. If you think about the sort of people who would choose to work at OpenAI, it is clear this was a very tough choice for them.
dchichkov 8y ago
@OpenAI - please, research reputations systems and authorship identification, not fake news generators.
psandersen 8y ago
I'm honestly pretty conflicted here and the best thing that is coming out of this is some healthy debate. IMHO the strongest argument against publishing it is because its built from Reddit it includes a lot of toxicity... VinayPrabhu originally made that argument here https://medium.com/@VinayPrabhu/reddit-a-one-word-reason-why-i-support-openais-gpt-2-decision-56b82912443c https://medium.com/@VinayPrabhu/reddit-a-one-word-reason-why... Can't blame them for not opening up that can of PR worms. It could be like a version of Microsoft Tay that can form coherent paragraphs. The bigger picture is this could be used for some very effective social media manipulation. A bit of clever infrastructure, maybe another generation of improvement and some understanding of human psychology (overton window etc) and managing 100,000 believable human accounts on something like reddit that subtly move debate becomes very doable. Then again its going to happen anyway so better focus on the defensive techniques now...

289 points by hughzhang 8y ago

akyu 8y ago

My conspiracy theory on this: This entire fiasco is actually an experiment to gauge public reaction to this kind of release strategy. Eventually the point will come where the choice of releasing a model may have real ethical considerations, so they are test driving the possibilities with a relatively harmless model. Given the, in my opinion, huge overreaction to all of this, I fear this may only encourage AI research groups to be more secretive with their work.

tomphoolery 8y ago
OpenAI is just trying to make a buck. Just like most other programmers, they don't give a damn about how damaging their code is going to be.
- minimaxir 8y ago
  OpenAI is literally a non-profit.
  - nightski 8y ago
    Not saying I agree with the parent but while OpenAI is non-profit it's owned by billionaires whose companies directly benefit from their research.
    - sterlind 8y ago
      Perhaps they kept it closed source because they sold it to one of those companies? Non-profits are allowed to sell things, after all.
  - vokep 8y ago
    Do you think that means they don't get paid? Non-profit itself isn't a super meaningful term as far as if profits are being made or not. To be a non-profit, many hoops have to be jumped through, certain forms of profit have to be zero (I don't know the specifics, but a friend of mine was considering starting a non-profit at one point) but the employees are still paid.
  - peterwwillis 8y ago
    You can literally make millions of dollars working for non-profits. The highest paid CEOs make several million dollars per year and the biggest non-profits have annual budgets in the billions. The annual revenue from all non-profits in the US is in the trillions.
    - jstarfish 8y ago
      The NFL is a nonprofit organization too...
- pmichaud 8y ago
  This is just false. Cynicism isn't a substitute for knowledge.
phiresky 8y ago
That's pretty much what they are publicly saying, right? > This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas.
- akyu 8y ago
  Ah I missed that.
currymj 8y ago
they acknowledged even in the original blog post that any well-funded group of NLP researchers would be able to replicate their work within a few weeks/months (including whatever corporation or state or terrorist group you worry about), and that in terms of methodology it is a natural, incremental improvement over existing techniques. so it's sort of obvious that withholding release can't prevent any harm. the good faith reason to withhold release is as you say -- start a conversation now about research norms so researchers have some decision making framework in case they come up with something really surprisingly dangerous. the bad faith reason is it gives them great PR. this was surely part of the motivation, but I bet OpenAI didn't quite expect the level of derangement in the articles that got published, and may regret it a little bit.
- Cybiote 8y ago
  Can we even say for certain it's an improvement over say something like TransformerXL? As far as I could see, the changes over GPT were a couple extra and tweak to layer normalizations, a small change to initialization and a change to text pre-processing. Other than for pre-processing, I didn't catch anything on theoretical motivations for these choices nor anything on ablation studies. The only thing that can be said for certain is it used lots of data and a very large number of parameters, trained on powerful hardware and achieved unmatched results in natural language generation.
vokep 8y ago
From their blog post: >These samples have substantial policy implications: large language models are becoming increasingly easy to steer towards scalable, customized, coherent text generation, which in turn could be used in a number of beneficial as well as malicious ways. We’ll discuss these implications below in more detail, and outline a publication experiment we are taking in light of such considerations. >a publication experiment we are taking in light of such considerations. So I think you're correct, they aren't really too scared about this one, but they realize they might be soon. Seeing how this turns out (how quickly its replicated, how quickly someone makes a wrapper for it that you can download and run yourself in minutes) will inform for more serious situations in the future.
bitL 8y ago
When you have whole teams doing "AI safety" at FANG, what do you think would be a logical output of their work? Wouldn't it correlate with what we see from OpenAI?

minimaxir 8y ago

It's worth noting that there's a surprising difference between the full model (1542M hyperparameters) and the released smaller model in the repo (117M): the full model can account for narrative structure much better (https://twitter.com/JanelleCShane/status/1097652984316481537 https://twitter.com/JanelleCShane/status/1097652984316481537), while the smaller model tends to go on trainwrecks, especially if you give it a bespoke grammatical structure (https://twitter.com/JanelleCShane/status/1097656934180696064 https://twitter.com/JanelleCShane/status/1097656934180696064) The arguments OpenAI have given about people using the model to create fake news are IMO not good; bad actors will still create fake news anyways (or with the paper/repo, they might even create more bespoke models trained on a specific type of dataset for even better fake news). With the open model, we would know how to better fight such attacks.

make3 8y ago
plus they have opened source the code, and the training cost of the full model is evaluated at 45k$, which is not much for a state actor or a serious org of any kind
- Jach 8y ago
  $45k is "new car" money, plenty of individuals on this site could afford it if they wanted. Don't even have to be that serious about it.
- buboard 8y ago
  what would they do with it? It is a text decorator, it can't make sense of itself. maybe they can make 1000 fake facebook accounts that post a lot. still, i doubt facebook uses text to detect bots, its easier to use ip/network patterns.
  - make3 8y ago
    it's not a text decorator, it's a text generator
    - 8y ago

pmichaud 8y ago

I really strongly disagree with this. I don't have much time to write, but: 1. Photo manipulation has been extremely destructive in a variety of ways. People "know" that photoshop is a thing, yet fake pictures abound at all levels of publications, and unrealistic standards propagate at full speed nonetheless. The effects are wide spread and insidious. A way to automate the production of median-literacy shitposts on the internet tuned to whatever you want to propagandize would be a devastating blow. People "knowing" that it was possible would do precious little to stop the onslaught, and it all but destroy online discourse in any open channels. I'm certain there would be other consequences that I can't fathom right now because I don't grok the potential scale or nth order effects. 2. There is no fire alarm for AGI[1]. Is this thing the key to really dangerous AI? Will the next thing be? No one knows. It's better to be conservative. [1] https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/

hughzhang 8y ago
Thanks for your comments! On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. I'm sure I've been tricked by a whole host of random facts which are probably actually BS based on random stuff I've read online. On the other hand, if "dangerous" means that a malicious agent can systematically manipulate the public via propaganda a la 20th century Stalin, I think this is basically impossible. Given widespread knowledge of photoshop, changing someone's mind about something significant with a doctored photo seems difficult (it is hard enough to change someone's mind with the truth!). Doctored photographs aren't completely harmless in that they can reinforce existing held beliefs (confirmation bias) and random fake facts aren't totally harmless either, I just think that the danger here is far below what has been implied in other places. In a nutshell, the effects are indeed wide spread, but perhaps not so insidious. On 2) there being no fire alarm, I'm actually very thankful to OpenAI for raising this discussion. While I disagree with their decision not to open source, this discussion was certainly worth having.
- wallflower 8y ago
  > On 1) photo manipulation. If "dangerous" means that random, relatively harmless fake facts easily spread, I'm inclined to agree. No, I think it is even more insidious what they were referring to. I think they meant that Photoshop has enabled and continues to enable unrealistic physical standards of beauty and BMI (mostly for women) that are transmitted in advertisements and news articles. In other words, Photoshop has enabled an alternative visual reality that is not realistic.

sytelus 8y ago

This is very nice summary of lot of social media discussion that has been happening. Very thoughtful arguments from people who actually work on this stuff.

fossuser 8y ago

I thought this blog post response was pretty good: OpenAI’s GPT-2: the model, the hype, and the controversy https://towardsdatascience.com/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8 https://towardsdatascience.com/openais-gpt-2-the-model-the-h... I think this is a better response than the article submitted here. There are ethical concerns and risk with these releases and it's probably a good idea to start considering them. What they did release seems reasonable in context. I find the argument that making the technology as open as possible leading to people taking a skeptical look at things not very convincing. Recent election interference, disinformation campaigns, and the general inability for the public to disambiguate fact from fiction seems like decent evidence that just because people know something can be faked doesn't lead to critical thinking - confirmation bias is strong.

buboard 8y ago
as an FOSSuser you should know that many parts of opensource software used to be outright illegal (e.g. pgp). opensource democratized all of them

wodenokoto 8y ago

Not releasing the model seems to me to be entirely a PR stunt, and has made me lose a lot of respect for OpenAI.

kogir 8y ago

Everything important is open sourced already. The article cites replication as important and then paradoxically argues for releasing the model in lieu of replicating the training and verifying the result. This is the AI equivalent of a pharmaceutical company releasing all their research and patents and then taking heat for not providing reagents and doing the organic synthesis for everyone too.

minimaxir 8y ago
The difference is a pharmaceutical company is for-profit, while OpenAI's (a non-profit) mission (https://openai.com/about/ https://openai.com/about/) is "to build safe AGI, and ensure AGI's benefits are as widely and evenly distributed as possible." (emphasis mine)
- kogir 8y ago
  I don’t see how corporate structure changes anything. It’s a huge gift of tons of work, which is far and away the hard part. They’ve enabled literally anyone who wants to to train their own model on whatever source material they want.

new_guy 8y ago

This is the best publicity they could ever have and it didn't cost them a penny. There's probably 10 year olds in their bedrooms light years ahead of these guys, certainly from we've seen it's definitely nothing to write home about, but it's all about the marketing and hype, convincing the mass media they've invented the next Terminator.

leesec 8y ago
This is crazy. It is a substantial step forward for NLP and basically every researcher/important person in the field agrees. And hype for what? They aren't selling anything.
- buboard 8y ago
  they are selling hype. i also see it as a jab to fb/google "look boys, we re gonna keep this nuclear bomb from filling your databases with spam"
newen 8y ago
> There's probably [amateurs] light years ahead of these guys This is not an 80s hacker movie and amateur hackers can get into the DoD computers using a public phone or whatever. This is actual research funded with millions of dollars done by people with years of experience in the field. I find this whole meme in the programming world of some random programmer being able to work miracles and be super advanced kind of dumb.
- carapace 8y ago
  Is it a meme in the programming world? I've only encountered it in Hollywood and TV. It's like how every on-screen computer beeps and boops on every interaction: it's a dramatic device. No one does that IRL because it would drive you bananas in like ten seconds. Cf. Glyph's "The Television Writer's Guide to Cryptography" https://glyph.twistedmatrix.com/2009/01/television-writer-guide-to-cryptography.html https://glyph.twistedmatrix.com/2009/01/television-writer-gu...

cartercole 8y ago

release the code! you cant stop the signal

minimaxir 8y ago
The code is out (https://github.com/openai/gpt-2 https://github.com/openai/gpt-2), just not the larger model itself.
- gwern 8y ago
  Saying 'the code is out' is a little misleading. It's some supporting code for working with a trained model, and a model definition of what is already a standard and widely-implemented architecture. It doesn't include the dataset, the code used to construct the dataset, or perhaps most importantly of all, the code to train a model on a fleet of TPUv3s.
kogir 8y ago
They already released the code and that wasn’t enough for people.
- i_cant_speel 8y ago
  It's like releasing a car without giving access to the specific fuel needed to run it.
  - minimaxir 8y ago
    The smaller model is out so you can test drive it, but you won't be able to hit 100mph.

gambler 8y ago

Fortunately, we have outstanding and trustworthy corporate citizens like Google, Facebook, Uber and others that will continue AI research safely, ethically and responsibly without the risk of such technologies falling into the unsteady hands of the public.

boltzmannbrain 8y ago

Req'd reading: "OpenAI Trains Language Model, Mass Hysteria Ensues" by CMU Professor Zachary Lipton, http://approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/ http://approximatelycorrect.com/2019/02/17/openai-trains-lan... A salient quote from the article: > However, what makes OpenAI’s decision puzzling is that it seems to presume that OpenAI is somehow special—that their technology is somehow different than what everyone else in the entire NLP community is doing—otherwise, what is achieved by withholding it? However, from reading the paper, it appears that this work is straight down the middle of the mainstream NLP research. To be clear, it is good work and could likely be published, but it is precisely the sort of science-as-usual step forward that you would expect to see in a month or two, from any of tens of equally strong NLP labs.

eanzenberg 8y ago
Suffice to say, most of what comes out of OpenAI is vanilla type work that I haven't seen go beyond academic research labs. They do spend a lot of time on PR and making stuff look pretty, I guess.
- tedivm 8y ago
  To be fair they've also released some useful tools, such as the AI Gym.
  - ipsum2 8y ago
    To be fair, all of their useful tools have been deprecated or in 'maintenance mode'. See https://github.com/openai/gym https://github.com/openai/gym
rojobuffalo 8y ago
They list several tests with record breaking results here: https://blog.openai.com/better-language-models/#zeroshot https://blog.openai.com/better-language-models/#zeroshot Are those previous records their own or were those the best scores in the whole field of NLP? It sure looks like a pretty big step forward for one model to break that many records.

buboard 8y ago

The author doesnt mention the FUD that is being spread through tech media. "AI company is so afraid of its creation, it won't release its source code". Good job helping people understand their AI future. This is a PR stunt. Come on, a little extra spam is not shocking news, and whatever can be used to spread "fake news" can also be used to spread "correct news". Whatever the PR stunt they tried to pull, it is creating a bad narrative for AI.

ForrestN 8y ago

I’m agnostic about the short term effects of OpenAI’s decision, but I wonder if and hope that the eventual equilibrium will look more like things did before the internet. In, say, the 1960’s, if someone stood on a street corner with a sign saying “The End Is Nigh,” most people would react with a shrug. They’d know that if the world were ending they’d probably hear about it from a trusted source like a newspaper or broadcaster. I’m not worried about being tricked by fakes because I know that the NYT will be doing infinitely more than I ever could to verify hugely important facts, and won’t overreact to random bits of media. If you’re not someone I know personally, and you’re not a trusted source of public information, you may as well be holding a sign on a street corner, no matter how realistic the image or text on your sign. I think there may be vast unintended consequences that have to do with scale, but intuitively it doesn’t feel like the “who to trust” problem is more than a matter of cultural behaviors around trust adjusting to the internet age.

jpttsn 8y ago
Yeah, if this tech was the WMD that OpenAI says it is, the NYT would know about it.

nisten 8y ago

I don't understand why the choice of opensourcing seems to be so binary for some. It feels as if it's either: A-Leave everything open and hope that the global interoperability and cathartic flow of information creates a magic utopia for humanity. B-Everyone incorporates into the soviet-union/chinese-firewall for protection. There are manageable, more analogue alternatives too, like reducing the risk and probability of a disaster by trusting a human's judgement to limit access to potentially malicious code; which is exactly what they're doing in this case.

rippeltippel 8y ago

<quote>Precisely because everyone knows about Photoshop.</quote> Everyone?! All the non-digital-aware people on this planet, which I assume is the majority, may have never heard about Photoshop, or know what can be done with it. Out of the remaining digital-aware human beings who may well know about Photoshop, how many think about it when seeing a photo. I bet it's a very tiny percentage.

yongjik 8y ago

I don't buy the argument that open-sourcing would allow citizens to better fight this. Let's be realistic. If the full model is published, for every one person who studies the model to understand how to fight false propaganda, there will be 5,000 who use it to shitposting for lulz. Also, as everyone's pointing out, the cat's out of the bag. Whatever OpenAI does, the same ability will be in the hands of state actors, and large corporations, and soon not-too-large corporations and dedicated individuals, probably in a few years. So basically whatever OpenAI does today doesn't really matter. We might as well think about totally different ways of building online communities when everyone has shitpointing sockpuppet engines powered by AWS/Azure DeepLearningToolKit Trial Version.

dannyw 8y ago
Or a neural net for detecting fake text. How insane would it be if the birth of AGI comes from GANs of fake news?

deleted 8y ago

[deleted]

robrenaud 8y ago

What's extremely bad for research is that they didn't even release their train and test sets. It's not hard to grab 40GB of text from the web that comes from links with at least +3 votes on reddit. But even if you do that, you won't get the same train/test set, and the same split. So impossible to know if a model is outperforming their model.

sanxiyn 8y ago
They tested against standard language modeling benchmarks like PTB and WikiText, so it's entirely possible to compare.

peterwwillis 8y ago

A little learning is a dangerous thing; drink deep, or taste not the Pierian spring: there shallow draughts intoxicate the brain, and drinking largely sobers us again.

bitL 8y ago

Sooner or later we will have tools for e.g. Reddit where one specifies a desired outcome of a discussion and lets bots paired up with some optimization algorithm figure out how to get popularity majority based on comments so far in individual threads and drive discussion towards stated goal. Later the same will be used in elections or other public "theaters" that won't matter any longer as humanity will be "cracked", i.e. modeled sufficiently enough to predict/influence average humans with just tiny remains of feelings that something might be wrong which only complete societal outcasts would pursue.

mark_l_watson 8y ago

I like the work OpenAI has done in the past, so please take my complaints as gentle complaints: - a very small percentage of generated samples ‘read well’ - I Don’t see any evidence of ‘understanding’, at least in the sense that BERT can identify original nouns that pronouns refer to (anaphora resolution): amazing to get state of the art results - the ‘we aren’t going to share code and results’ thing reminds me of the movie The Wizard of Oz: trust them that they developed cool technology - in 6 months, high school students might be getting similar results, given how fast our field is moving forward: this makes the danger concern less important in my opinion This whole thing seems like a throwback to how science was conducted 10 years ago - ancient times in our fast moving tech world.

msla 8y ago
> the ‘we aren’t going to share code and results’ thing reminds me of the movie The Wizard of Oz: trust them that they developed cool technology And this is where it stands: They claim to have something. They won't release it. If they had nothing but a somewhat creative streak to make up the results, the world would look exactly the same as it would if they had something. Them lying is the least hypothesis here.
- unimpressive 8y ago
  Given that experts in the field seem to think this is an incremental step forward, OpenAI has already demonstrated a track record of doing things probably more impressive than this, and the massive loss of credibility that would come with being caught in a lie here, on priors we should expect they're not lying.

baron_harkonnen 8y ago

Why is there so much focus on just the language generation aspect of this model? The language generation examples are to show how well the model generalized the language it's trained on, but the whole point of releasing the pretrained model is that this model also: >achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training Every comment I see here and else where is strictly about automatically generating comments, but the real loss of having this model kept private is that it would allow anyone to create very, very good NLP projects right away. The big problem in ML/"AI" now and going forward is that there is an extreme asymmetry emerging in tech. The start of the last major tech boom was because of "disruption". Any clever hacker with a laptop and developer skills could create a product that would topple a giant. Likewise anyone could learn to do anything that needed to be done on their own computer in their own time. Now that's changing, to even play with these new tools you need special access and corporate support. Releasing the pretrained model to the public would allow a range of groups that cannot afford to build a model such as this to experiment on their own and achieve state of the art NLP results, not just in language generation but in classification, translation, summarization etc. Some kid in the middle of know where could download this and start messing with ideas about building a chat bot. The threat is not that some rogue actor will create evil spam bots, but that some small scrappy startup would leverage the same, currently exclusive tools, of large technology companies. The reason why democratizing AI is important is not because we can better fight off some vague threat, but so that, at least for the near term, individuals and small groups can still reasonably compete with major players in the ML space. And of course the fact that someone who claims openness and that they exist to "ensure AGI's benefits are as widely and evenly distributed as possible." Can arbitrarily choose not to do go the other direction just shows that tech is moving in a direction where "disruption" is going to become increasingly less possible, and the future of the industry is going to be controlled by a handful of major players.

doublekill 8y ago

Another thing that has not been discussed yet: OpenAI does not want to be responsible for the output of this model. Can you imagine the headlines? "OpenAI released a text generating AI and it is racist as hell". People should have learned their lesson after the Tay fiasco. I am ambiguous on the issue of release vs. non-release, but the mockery and derision they faced makes me ashamed to contribute to this field. No good faith is assumed, but projection of PR-blitzes and academic penis envy. Perhaps AI researchers are simply not the best for dealing with ethical and societal issues. Look at how long it took to get decent research into fairness, and its current low focus in industry. If you were in predictive modeling 10 years ago, it is likely you contributed to promoting and institutionalizing bias and racism. Do you want these same people deciding on responsible disclosure standards? Does the head of Facebook AI or those that OK'd project Dragonfly or Maven have any real authority on the responsible ethical use of new technology? I am not too sure about the impact of a human-level text generating tool. It may throw us back to the old days of email spam (before Bayesian spam filters). It is always easier to troll and derail than it is to employ such techniques for good. Scaling up disinformation campaigns is a real threat to our democracies (or maybe this decade-old technique is already in use at scale by militaries, and this work is merely showing what the AI community's love for military funding looks like). I am sure that the impact of the NIPS abbreviation is an order of magnitude lower than that of this technology, yet companies like NVIDIA used Neurips in their marketing PR before it was officially introduced (made them look like the good guys for a profit). How is that for malaligning ML research for PR purposes? Would the current vitriol displayed in online discussons be appreciated when there was a name change proposal for the betterment of society? Disclaimer: this comment in favor of OpenAI was written by a real human. Could you tell for sure now you know the current state of the art? What would these comment sections look like if one person controls 20% of the accounts here?

jstarfish 8y ago
> It is always easier to troll and derail than it is to employ such techniques for good. Curious-- what possible good comes with the ability to generate grammatically correct text devoid of actual meaning? Sure, you could generate an arbitrary essay, but it's less an essay about anything and more just an arrangement of words that happen to statistically relate to each other. Markov chains already do this and while the output looks technically correct, you're not going to learn or interpret anything from it. Same goes with things like autocomplete. You could generate an entire passage just accepting its suggestions. It would pass a grammar test but doesn't mean anything. Chatbots are an obvious application, but how is that "good," or any different than the Harlow experiment performed on humans? Fooling people into developing social relationships with an algorithm (however short or trivial) is cruel and unethical beyond belief. A photoshopped image might be pleasant or stimulating to look at, and does have known problems in terms of normalizing a fictitious reality. But fake text? What non-malicious use can there possibly be?

angel_j 8y ago

They did release the model [0]. A smaller-scale version of it. If you want the big one, all you have to do is scrape 45M web pages, and increase the parameters to scale. What they didn't release is the training corpus and the trained filters. By not releasing these, and focusing on ethics, they are trying to avert attention from the mistake of training a massive cyclops on unfiltered internet text. They probably read a lot of embarrassing text they didn't share, imagine that. 0. https://github.com/openai/gpt-2/ https://github.com/openai/gpt-2/

scottlocklin 8y ago

Yes +1; mostly so we can laugh at how much your PR team exaggerated your results.

darklycan51 8y ago

The fact that this thread is in the site is scary, seems like there are parties interested in a public outcry to make some potentially very harmful tools be released to the public... the question is, why?

PaulHoule 8y ago

Network architectures and trainers are improving quickly so it won't be long before others can create a similar model. Their ace is having a huge clean dataset of grammatically correct text. This is not about the fine points of grammar, but that the content of the average web page is practically word salad.

aaron695 8y ago

It's an obvious lie that there's an issue to release it. Can anyone explain what it might do bad, WITHOUT hand waving? What? Blog spam to attack Googles algorithm? We can already attack it using markov chains, they simply use IP's and incoming links and other tricks to fight back. You can read the examples, they are meaningless to humans obviously, the intelligence to create a article that's intelligent would be sentient AI. Unlike the human photo creating AI which actually is an issue. A non handwaving example why it's bad, people can quickly Tinyeye a photo to see if it's fake or not. With this AI they cannot. Fresh photos take real world people and money, people have learned they add authenticity because of this, while people relearn AI means this is no longer true they are open to attack. A jumble of words.... we've all seen before. (And the researchers are using humans to pick the best jumble of words so it still takes a skilled human = $)

paulsutter 8y ago

Are they planning to change their name to ClosedAI?

bllguo 8y ago

To me it feels like everything about OpenAI's media strategy is calculated and specifically designed to maximize good PR with the public. It leaves a sour taste in my mouth.

wrsh07 8y ago
It's very weird to me that a team of researchers who are credibly doing good work and are making an effort to clearly explain their research garners such suspicion. Are you similarly skeptical of Deepmind, Distill, or Two Minute Papers? Honestly, the norm of providing two versions of your research (arxiv and blog post) makes ML easier to follow and is something I would like more academics to attempt

n1231231231234 8y ago

1) What really bothered me personally about GPT2 is that they made it look sciency by putting out a paper that looks like other scientific papers -- but then they undermine a key aspect of science: reproducability/verifiability. I struggle to believe 'science' that cannot be verified/replicated. 2) In addition to this, they stand on the shoulder of giants and profit from a long tradition of researchers and even companies making their data and tools available. but "open"AI chose to go down a different path. 3) which makes me wonder what they are trying to add to the discussion? the discussion about the dangers of AI is fully ongoing. by not releasing background info also means that openAI is not contributing to how dangerous AI can be approached. openAI might or might not have a model that is closer to some worrisome threshold. but we don't know for sure. so imv, what openAI primarily brought to the discussion are some vague fears against technological progress -- which doesn't help anyone.

sanxiyn 8y ago
Re 1: GPT2 is no different from most stuffs by DeepMind. DeepMind, in general, does not release code, data, or model. DeepMind does not seem to get reproducibility complaints, supposedly "key aspect of science".

namuol 8y ago

Here's the most thoughtful piece I've been able to find on the controversial topic so far: https://www.fast.ai/2019/02/15/openai-gp2/ https://www.fast.ai/2019/02/15/openai-gp2/

jayantj 8y ago

I strongly disagree with OP's logic when he says - Photoshop hasn't negatively affected the world "precisely because everyone knows about Photoshop." - the crux of his argument. IMO the main reason Photoshop hasn't affected the world negatively is because it is HARD to make convincing fakes - precisely what OpenAI are ensuring by not releasing the model. Notice that this reasoning doesn't take into account whether or not GPT-2 can actually negatively impact the world. It says as long as people are aware that the text they are reading could be fake/AI-generated, we'll be fine. I think people are already aware of that. I don't see how releasing the pre-trained model will help with that.

Gatsky 8y ago

The argument in the article is unconvincing, and shows a distinct failure to think on a societal level or on time lines beyond a few weeks. The calculus seems simple to me. The potential harms from not open sourcing the model completely - minimal. Potential harms from open sourcing it completely - greater than minimal. Hence their decision. OpenAI should be lauded for making this decision. If you think about the sort of people who would choose to work at OpenAI, it is clear this was a very tough choice for them.

dchichkov 8y ago

@OpenAI - please, research reputations systems and authorship identification, not fake news generators.

psandersen 8y ago

I'm honestly pretty conflicted here and the best thing that is coming out of this is some healthy debate. IMHO the strongest argument against publishing it is because its built from Reddit it includes a lot of toxicity... VinayPrabhu originally made that argument here https://medium.com/@VinayPrabhu/reddit-a-one-word-reason-why-i-support-openais-gpt-2-decision-56b82912443c https://medium.com/@VinayPrabhu/reddit-a-one-word-reason-why... Can't blame them for not opening up that can of PR worms. It could be like a version of Microsoft Tay that can form coherent paragraphs. The bigger picture is this could be used for some very effective social media manipulation. A bit of clever infrastructure, maybe another generation of improvement and some understanding of human psychology (overton window etc) and managing 100,000 believable human accounts on something like reddit that subtly move debate becomes very doable. Then again its going to happen anyway so better focus on the defensive techniques now...