6 ms·

It sounds like all the drama OpenAI made about not releasing the model was all just marketing. $50,000 is nothing for a nation-state or even just a motivated th

It sounds like all the drama OpenAI made about not releasing the model was all just marketing. $50,000 is nothing for a nation-state or even just a motivated third party. I had always assumed OpenAI had spent well into the 6 or even 7 figures to train the full model. MSFT has sort of invested $1 billion into OpenAI so I guess it worked!

skrebbel 7y ago
I have a beef with this. OpenAI is run by people who believe and spread horror stories about how AI can totally change society, for better or worse. A lot of what AI can and cannot do depends on cost and computational power as much as anything else. If I understand correctly, there's a whole bunch of "you could but it's prohibitively expensive" stuff going around. I'm convinced that the people behind OpenAI understand this nuance, even if most non geeks struggle with the difference between possible and feasible. This means that OpenAI mixed the two up on purpose. If the people warning us of the robot apocalypse can't be trusted to communicate openly and honestly about odds, then what do we do? They clearly have no qualms about misinforming the public to serve a hidden agenda. It might be pretty harmless in this case, but it's indicative of a cultural pattern. Basically I just hope (and kind of believe) that we'll never achieve AGI because that would make all this talk unimportant.
- repolfx 7y ago
  OpenAI is run by people who believe and spread horror stories about how AI can totally change society, for better or worse. I think this is reflective of a substantial disconnect and polarisation in society, because OpenAI's position here was driven by a very fundamental intuition about human nature which is not universally shared. Namely, their rationale was something like this: the world is dominated by people whose opinions are fundamentally shaped by what others are saying, and moreover, by how frequently other people seem to be saying it. They don't really think about things or rely on their own experience: they're just mimics. In such a world being able to auto-generate fake messages of support for some political position or another at scale would give you immense power, because the population would automatically swing behind you based merely on the perception that everyone was swinging behind you. So that's their fear. But is it realistic? Well, this is where we get into the polarisation. "You aren't smart enough to have an opinion" is the sort of viewpoint that leads to an elitist vs populist conflict. There have been reams of analysis about this and it's not really AI specific - e.g. did people vote for Brexit because of Twitter, or because of things they saw in the press, or did they vote based on their own experiences, or what their close friends/family thought, or what mix of "all of the above" is the truest mix? When they refused to release GPT-2 the OpenAI researchers took an extreme on that spectrum, asserting that essentially they had built a mind control device. But of course which didn't work on themselves, only on lesser minds. Not surprisingly this was very controversial, albeit I think the Valley/Hacker News set would be surprised at just how controversial it would have been if anyone outside the AI world had really noticed. You can't tell most of the world their opinions aren't really their own and not expect pushback. Personally I don't share those intuitions at all. The controversy over DeepFakes is similar. Photoshop has existed for decades and hasn't led to a dystopia, but suddenly DeepFakes is going to create fundamental social change? I don't think so. AI researchers, especially in academia, need big narratives to keep up the funding and suggest they're on a mission to save the world vs e.g. making slightly better playlist recommendations. GPT-2 seems like a classic case of this being taken to absurdity.
  - Veedrac 7y ago
    > When they refused to release GPT-2 the OpenAI researchers took an extreme on that spectrum, asserting that essentially they had built a mind control device. Having read everything I could find from OpenAI on this topic, I have no recollection at all of such claims by OpenAI. Sources would be appropriate.
    - repolfx 7y ago
      They said they wouldn't release the biggest model because it could be abused to make bots that sway politics. Consider the chain of reasoning required to believe that.
      - Veedrac 7y ago
        Again, I suggest you give direct sources for your arguments.
        repolfx 7y ago
        Are you sure you read their announcement blog? You seem to believe that they didn't say what they spent an entire blog post saying. Here it is again. https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/ They said "Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code." And in case it wasn't clear what they mean by "abusive language", they clarified: "Today, malicious actors—some of which are political in nature—have already begun to target the shared online commons, using things like “robotic tools, fake accounts and dedicated teams to troll individuals with hateful commentary or smears that make them afraid to speak, or difficult to be heard or believed”" "These findings, combined with earlier results on synthetic imagery, audio, and video, imply that technologies are reducing the cost of generating fake content and waging disinformation campaigns. The public at large will need to become more skeptical of text they find online." Or summarised: • We think people believe things they read online today. • We think there are political "disinformation campaigns" being waged, by the sort of people who would use AI to wage them more effectively. • We think GPT-2 would let them create "deceptive, biased or abusive language" automatically, and this is too dangerous to release because people aren't skeptical enough of what they read online. • We think the entire population needs to change its behaviour for GPT-2 to be safe to release. I think that directly supports my arguments. Their whole worldview assumes the population is naive, easily manipulated by text and more importantly the volume of it ... they also draw a bogus analogy to DeepFakes. It's bogus because nothing stops anyone writing words today that OpenAI staff might find "biased" or "deceptive", literally anyone can do that. But substituting a celebrities face onto porn would require very advanced video editing skills most people don't have: DeepFakes is a truly new capability, whereas GPT-2 is merely a capability of scale. To believe GPT-2 changes anything you must believe the majority of people make up their minds on political issues based purely on volumes of anonymous online comments. Hence my argument.
- Veedrac 7y ago
  > I'm convinced that the people behind OpenAI understand this nuance, even if most non geeks struggle with the difference between possible and feasible. This means that OpenAI mixed the two up on purpose. You should probably actually read what OpenAI have to say before making claims about what they have to say. “We are aware that some researchers have the technical capacity to reproduce and open source our results. We believe our release strategy limits the initial set of organizations who may choose to do this, and gives the AI community more time to have a discussion about the implications of such systems.” https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
  - Bartweiss 7y ago
    It's definitely worth reading OpenAI's statement, but I don't think speculation is out of bounds here either. The statement is reasonable on its own, but doesn't acknowledge that the decision is a sharp break from OpenAI's entire founding ideology. Musk said in 2015 that the point of OpenAI was "to empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower." Sikka argued that "openness" was the fundamental reason he supported the project. OpenAI was heavily criticized by other AI risk researchers for its public approach, and argued strongly that it was doing the right thing. It specifically invoked the threat of AI tools being abused by small groups of people in secret as more pressing than the threats from self-directed or public-use AI. But the closest the statement comes to addressing that is describing the restrictions as "an experiment", and mentioning that in 2018 OpenAI added a charter with a caveat saying it might publish less stuff. That caveat and the decision to invoke it here are essentially unexplained; since the charter almost certainly postdates some GPT-2 work it seems to be a circular argument. Frankly, GPT-2 seems like exactly the sort of work OpenAI was worried about in the first place. Ten thousand people can troll and spam ideas without any AI support, but with a GPT-2 equivalent it becomes possible to a government, company, or intelligence group to massively scale its impact on public debate. If OpenAI thinks state actors can't replicate their work, the fears invoked at its founding are called into question. If it instead thinks public access to such tools is dangerous and they need to be kept in select hands, the project's entire rationale is invalidated. If they simply hope to spark discussion and give people time to prepare (mentally or technologically) for the idea of compelling auto-generated text, that really ought to be said more clearly.
- sytelus 7y ago
  Initially I was amused by OpenAI's claim of "too dangerous to release" but during the past few months I have come to agree with the decision they made. Auto-generation of convincing text is actually very disruptive. Our news cycles and influencing stories are driven by things like Twitter hashtags, how many tweets are on a given story. People look for validation of their thoughts in places like 8chan, reddit, hn etc. There is an undeniable influence of all these discussions that occur online. Now imagine, one is able to create hundreds of accounts and do autonomous discussions to create an impression of consensus towards desired goal. Imagine large number of tweets that makes wise cracks are actually autogenerated. Imagine you as a person who is on borderline is taking cues from number of people pro-something to make decision. Imagine NYT running article on smearing some activist because there were thousands of Twitter smart-bots convincingly calling for his resignation with humorous mimes. Better the quality of AI, better the chance journalists and others would think it's real. I think GPT-2 would go down as one of the first real large scale danger of AI.
buboard 7y ago
OpenAI is the google of the new ML era. Except that they went from "dont be evil" to typical greedy corporate in less than a year.
- est31 7y ago
  My personal (conspiracy) theory is that the "don't be evil" trick was done to attract ML talent. Many people, if they are convinced they are working for a good cause, are okay with being paid less than if they worked at evil inc or a defense contractor or something. It's a big reason why academia has so many people at the top of their fields accept little pay (compared to their pay in the industry).
  - buboard 7y ago
    These people are now held hostage by the "100x" promises. I 'm hopeful however that all this "Old Tech Money" - FAANGs&OpenAI won't be able to keep them for long, because AI may be used to destroy their existing revenue streams (advertising - optimization - engagement)
p1esk 7y ago
OpenAI most likely spent 6 or even 7 figures to produce GPT-2 model. The guys who wrote this article mentioned they spent $500k to replicate OpenAI’s results. Each training run costs $50k, and they needed many runs to find the optimal config. Note that the GPT-2 paper had already described many crucial details (eg dataset). Without those details it would be a lot harder to do this.
anchpop 7y ago
Before they release it, they need to prove to themselves it's safe (I'm speaking morally). I'm not sure why so many on HN seem to think it's the other way around, that you have to prove it's dangerous before withholding it. $500k for reproduction is peanuts for a nation state but quite a bit for many other groups who might be interested. Even if it is entirely safe, it's still good to withhold it because it helps create a culture where people think about the safety of their projects before releasing them
- deleted 7y ago
  [deleted]
- jgalt212 7y ago
  > Before they release it, they need to prove to themselves it's safe (I'm speaking morally) I don't think it was a moral question. I think the lawyers got involved and were uncomfortable with potential litigation exposure.
- Bartweiss 7y ago
  There are two very different questions here: is it good to withhold, and is that sensible for OpenAI to withhold? I think a lot of people on HN view restricting GPT as incoherent by OpenAI, where they might not for another group. OpenAI's raison d'etre was democratizing access to AI tools, and preventing them from being abused by concentrated powers like governments. If replicating GPT-2 is trivial for state actors but prohibitive for hobbyists and other private citizens, it's creating the same issue Musk described OpenAI as setting out to oppose. Even the general idea of treating AI tools as hazardous-by-default goes a long way to validating the project's original critics.
- buboard 7y ago
  ITs really difficult to impossible to prove it s eithrt safe or not. We re still in the early stage of ML research so this kind of fud is a brake to research. Similar to how genetic research has been held back by regulators
gradys 7y ago
My understanding of their rationale for the disclosure policy is that while this model might not necessarily be so incredibly dangerous, 1) There was a non-zero chance that their releasing it would do more harm than good by making things like fake news, spam, and sock puppeting cheaper and more scalable, and 2) It's very likely that eventually they or others will develop more unambiguously dangerous models, and it's valuable to begin experimenting with responsible disclosure policies now so we're better prepared. Perhaps some here would disagree with the premise of (2), that it's possible for a model to be dangerous. I can understand that. Working in the field, and having been exposed to many of the same arguments that are likely informing OpenAI's concern, I think it's valid. Valid enough at least that I'm prepared to believe that OpenAI is acting in good faith, and that the disclosure policy is not "just marketing".
skybrian 7y ago
Nation-state actors aren't the only people who can do damage. You also have to worry about the script kiddies. There are people who will do bad things for the lulz, but only if it's easy.

skrebbel 7y ago

I have a beef with this. OpenAI is run by people who believe and spread horror stories about how AI can totally change society, for better or worse. A lot of what AI can and cannot do depends on cost and computational power as much as anything else. If I understand correctly, there's a whole bunch of "you could but it's prohibitively expensive" stuff going around. I'm convinced that the people behind OpenAI understand this nuance, even if most non geeks struggle with the difference between possible and feasible. This means that OpenAI mixed the two up on purpose. If the people warning us of the robot apocalypse can't be trusted to communicate openly and honestly about odds, then what do we do? They clearly have no qualms about misinforming the public to serve a hidden agenda. It might be pretty harmless in this case, but it's indicative of a cultural pattern. Basically I just hope (and kind of believe) that we'll never achieve AGI because that would make all this talk unimportant.

repolfx 7y ago
OpenAI is run by people who believe and spread horror stories about how AI can totally change society, for better or worse. I think this is reflective of a substantial disconnect and polarisation in society, because OpenAI's position here was driven by a very fundamental intuition about human nature which is not universally shared. Namely, their rationale was something like this: the world is dominated by people whose opinions are fundamentally shaped by what others are saying, and moreover, by how frequently other people seem to be saying it. They don't really think about things or rely on their own experience: they're just mimics. In such a world being able to auto-generate fake messages of support for some political position or another at scale would give you immense power, because the population would automatically swing behind you based merely on the perception that everyone was swinging behind you. So that's their fear. But is it realistic? Well, this is where we get into the polarisation. "You aren't smart enough to have an opinion" is the sort of viewpoint that leads to an elitist vs populist conflict. There have been reams of analysis about this and it's not really AI specific - e.g. did people vote for Brexit because of Twitter, or because of things they saw in the press, or did they vote based on their own experiences, or what their close friends/family thought, or what mix of "all of the above" is the truest mix? When they refused to release GPT-2 the OpenAI researchers took an extreme on that spectrum, asserting that essentially they had built a mind control device. But of course which didn't work on themselves, only on lesser minds. Not surprisingly this was very controversial, albeit I think the Valley/Hacker News set would be surprised at just how controversial it would have been if anyone outside the AI world had really noticed. You can't tell most of the world their opinions aren't really their own and not expect pushback. Personally I don't share those intuitions at all. The controversy over DeepFakes is similar. Photoshop has existed for decades and hasn't led to a dystopia, but suddenly DeepFakes is going to create fundamental social change? I don't think so. AI researchers, especially in academia, need big narratives to keep up the funding and suggest they're on a mission to save the world vs e.g. making slightly better playlist recommendations. GPT-2 seems like a classic case of this being taken to absurdity.
- Veedrac 7y ago
  > When they refused to release GPT-2 the OpenAI researchers took an extreme on that spectrum, asserting that essentially they had built a mind control device. Having read everything I could find from OpenAI on this topic, I have no recollection at all of such claims by OpenAI. Sources would be appropriate.
  - repolfx 7y ago
    They said they wouldn't release the biggest model because it could be abused to make bots that sway politics. Consider the chain of reasoning required to believe that.
    - Veedrac 7y ago
      Again, I suggest you give direct sources for your arguments.
      - repolfx 7y ago
        Are you sure you read their announcement blog? You seem to believe that they didn't say what they spent an entire blog post saying. Here it is again. https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/ They said "Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code." And in case it wasn't clear what they mean by "abusive language", they clarified: "Today, malicious actors—some of which are political in nature—have already begun to target the shared online commons, using things like “robotic tools, fake accounts and dedicated teams to troll individuals with hateful commentary or smears that make them afraid to speak, or difficult to be heard or believed”" "These findings, combined with earlier results on synthetic imagery, audio, and video, imply that technologies are reducing the cost of generating fake content and waging disinformation campaigns. The public at large will need to become more skeptical of text they find online." Or summarised: • We think people believe things they read online today. • We think there are political "disinformation campaigns" being waged, by the sort of people who would use AI to wage them more effectively. • We think GPT-2 would let them create "deceptive, biased or abusive language" automatically, and this is too dangerous to release because people aren't skeptical enough of what they read online. • We think the entire population needs to change its behaviour for GPT-2 to be safe to release. I think that directly supports my arguments. Their whole worldview assumes the population is naive, easily manipulated by text and more importantly the volume of it ... they also draw a bogus analogy to DeepFakes. It's bogus because nothing stops anyone writing words today that OpenAI staff might find "biased" or "deceptive", literally anyone can do that. But substituting a celebrities face onto porn would require very advanced video editing skills most people don't have: DeepFakes is a truly new capability, whereas GPT-2 is merely a capability of scale. To believe GPT-2 changes anything you must believe the majority of people make up their minds on political issues based purely on volumes of anonymous online comments. Hence my argument.
Veedrac 7y ago
> I'm convinced that the people behind OpenAI understand this nuance, even if most non geeks struggle with the difference between possible and feasible. This means that OpenAI mixed the two up on purpose. You should probably actually read what OpenAI have to say before making claims about what they have to say. “We are aware that some researchers have the technical capacity to reproduce and open source our results. We believe our release strategy limits the initial set of organizations who may choose to do this, and gives the AI community more time to have a discussion about the implications of such systems.” https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/
- Bartweiss 7y ago
  It's definitely worth reading OpenAI's statement, but I don't think speculation is out of bounds here either. The statement is reasonable on its own, but doesn't acknowledge that the decision is a sharp break from OpenAI's entire founding ideology. Musk said in 2015 that the point of OpenAI was "to empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower." Sikka argued that "openness" was the fundamental reason he supported the project. OpenAI was heavily criticized by other AI risk researchers for its public approach, and argued strongly that it was doing the right thing. It specifically invoked the threat of AI tools being abused by small groups of people in secret as more pressing than the threats from self-directed or public-use AI. But the closest the statement comes to addressing that is describing the restrictions as "an experiment", and mentioning that in 2018 OpenAI added a charter with a caveat saying it might publish less stuff. That caveat and the decision to invoke it here are essentially unexplained; since the charter almost certainly postdates some GPT-2 work it seems to be a circular argument. Frankly, GPT-2 seems like exactly the sort of work OpenAI was worried about in the first place. Ten thousand people can troll and spam ideas without any AI support, but with a GPT-2 equivalent it becomes possible to a government, company, or intelligence group to massively scale its impact on public debate. If OpenAI thinks state actors can't replicate their work, the fears invoked at its founding are called into question. If it instead thinks public access to such tools is dangerous and they need to be kept in select hands, the project's entire rationale is invalidated. If they simply hope to spark discussion and give people time to prepare (mentally or technologically) for the idea of compelling auto-generated text, that really ought to be said more clearly.
sytelus 7y ago
Initially I was amused by OpenAI's claim of "too dangerous to release" but during the past few months I have come to agree with the decision they made. Auto-generation of convincing text is actually very disruptive. Our news cycles and influencing stories are driven by things like Twitter hashtags, how many tweets are on a given story. People look for validation of their thoughts in places like 8chan, reddit, hn etc. There is an undeniable influence of all these discussions that occur online. Now imagine, one is able to create hundreds of accounts and do autonomous discussions to create an impression of consensus towards desired goal. Imagine large number of tweets that makes wise cracks are actually autogenerated. Imagine you as a person who is on borderline is taking cues from number of people pro-something to make decision. Imagine NYT running article on smearing some activist because there were thousands of Twitter smart-bots convincingly calling for his resignation with humorous mimes. Better the quality of AI, better the chance journalists and others would think it's real. I think GPT-2 would go down as one of the first real large scale danger of AI.

buboard 7y ago

OpenAI is the google of the new ML era. Except that they went from "dont be evil" to typical greedy corporate in less than a year.

est31 7y ago
My personal (conspiracy) theory is that the "don't be evil" trick was done to attract ML talent. Many people, if they are convinced they are working for a good cause, are okay with being paid less than if they worked at evil inc or a defense contractor or something. It's a big reason why academia has so many people at the top of their fields accept little pay (compared to their pay in the industry).
- buboard 7y ago
  These people are now held hostage by the "100x" promises. I 'm hopeful however that all this "Old Tech Money" - FAANGs&OpenAI won't be able to keep them for long, because AI may be used to destroy their existing revenue streams (advertising - optimization - engagement)

p1esk 7y ago

OpenAI most likely spent 6 or even 7 figures to produce GPT-2 model. The guys who wrote this article mentioned they spent $500k to replicate OpenAI’s results. Each training run costs $50k, and they needed many runs to find the optimal config. Note that the GPT-2 paper had already described many crucial details (eg dataset). Without those details it would be a lot harder to do this.

anchpop 7y ago

Before they release it, they need to prove to themselves it's safe (I'm speaking morally). I'm not sure why so many on HN seem to think it's the other way around, that you have to prove it's dangerous before withholding it. $500k for reproduction is peanuts for a nation state but quite a bit for many other groups who might be interested. Even if it is entirely safe, it's still good to withhold it because it helps create a culture where people think about the safety of their projects before releasing them

deleted 7y ago
[deleted]
jgalt212 7y ago
> Before they release it, they need to prove to themselves it's safe (I'm speaking morally) I don't think it was a moral question. I think the lawyers got involved and were uncomfortable with potential litigation exposure.
Bartweiss 7y ago
There are two very different questions here: is it good to withhold, and is that sensible for OpenAI to withhold? I think a lot of people on HN view restricting GPT as incoherent by OpenAI, where they might not for another group. OpenAI's raison d'etre was democratizing access to AI tools, and preventing them from being abused by concentrated powers like governments. If replicating GPT-2 is trivial for state actors but prohibitive for hobbyists and other private citizens, it's creating the same issue Musk described OpenAI as setting out to oppose. Even the general idea of treating AI tools as hazardous-by-default goes a long way to validating the project's original critics.
buboard 7y ago
ITs really difficult to impossible to prove it s eithrt safe or not. We re still in the early stage of ML research so this kind of fud is a brake to research. Similar to how genetic research has been held back by regulators

gradys 7y ago

My understanding of their rationale for the disclosure policy is that while this model might not necessarily be so incredibly dangerous, 1) There was a non-zero chance that their releasing it would do more harm than good by making things like fake news, spam, and sock puppeting cheaper and more scalable, and 2) It's very likely that eventually they or others will develop more unambiguously dangerous models, and it's valuable to begin experimenting with responsible disclosure policies now so we're better prepared. Perhaps some here would disagree with the premise of (2), that it's possible for a model to be dangerous. I can understand that. Working in the field, and having been exposed to many of the same arguments that are likely informing OpenAI's concern, I think it's valid. Valid enough at least that I'm prepared to believe that OpenAI is acting in good faith, and that the disclosure policy is not "just marketing".

skybrian 7y ago

Nation-state actors aren't the only people who can do damage. You also have to worry about the script kiddies. There are people who will do bad things for the lulz, but only if it's easy.