3 ms·

Due to concerns about malicious applications, GPT2 will not be released (2019)

74 points by Jimmc414 12d ago

kuzumancer 12d ago
They were worried about the rap battle between Paul Graham and Michael O. Church. Which sadly never happened.
andy99 12d ago
See also the 2023 call to “ pause for at least 6 months the training of AI systems more powerful than GPT-4.” https://futureoflife.org/open-letter/pause-giant-ai-experiments/ https://futureoflife.org/open-letter/pause-giant-ai-experime... Signed by Benjio, Musk, Woz, etc
jkuli 12d ago
If it is possible to save one life, preventing RSI AGI is an existential threat.
- Jimmc414 12d ago
  But why are Dario, Sam and Elon who represent "the frontier" more than any other 3 people pretending like they need government permission or intervention to pause? It seems more likely that they see open source distillation as an existential threat to their IPO plans and are playing for regulatory capture of the market.
  - jonas21 12d ago
    It would require government permission. The top labs would be agreeing not to compete on model advancements for a period of time, which seems like a pretty clear violation of antitrust law.
    - anon373839 12d ago
      That isn’t any kind of antitrust violation. Agreeing not to undercut each other’s prices would be, however.
    - angoragoats 12d ago
      How is this a clear violation of antitrust law? And have any of the leaders of these companies said that they are asking for government permission for this reason?
      - jonas21 12d ago
        It's the classic case of a cartel conspiring to limit competition. Here's Matt Levine's explanation [1]: > 1. Anthropic, OpenAI, and perhaps a couple of other frontier labs are the dominant providers of frontier AI models. > 2. They can charge customers a lot of money for using their frontier models, and rather less money for older, no-longer-cutting-edge models. > 3. Training a new frontier model requires ever-increasing billions of dollars of computing power. > 4. The labs need to more or less continuously race to build new frontier models, because their competitors are all doing it, and if they don’t they will fall behind and no longer be able to charge a lot of money for their best models. (Also because they intrinsically want to build artificial superintelligence, for cancer-curing and/or killing-everyone reasons.) > 5. If they collectively slowed down, then (1) they’d spend less on compute and (2) they’d be able to charge frontier-model prices for a longer time. > 6. But if one of them slowed down, the others would eat its lunch. > 7. If they got together in a room and agreed to slow down, that would look like an antitrust conspiracy: It is generally illegal for competitors to get together and agree to limit the output of their industry. > 8. But if they publish papers about how important it is to slow down, that might have a similar coordinating function, at least among the US frontier labs if not necessarily among their Chinese competitors. > 9. And if the government believes those papers, it might help them coordinate. Maybe the government will impose pacing by regulation that the labs could not impose by agreement. Or at least the government will let them get together and agree to slow down. Amodei’s post calls for “frontier AI companies within democratic countries [to] coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress”; a footnote adds: “With government mediation or waivers of antitrust restrictions.” Just meeting in a room to establish common safety standards is legally risky; the labs can’t do it on their own unless governments affirmatively allow it. [1] https://www.bloomberg.com/opinion/newsletters/2026-09-14/ai-labs-want-someone-to-stop-them https://www.bloomberg.com/opinion/newsletters/2026-09-14/ai-...
  - rfgplk 12d ago
    Meme marketing. I can attest that no one at these companies cares about laws. Not one bit.
- denverllc 12d ago
  ... is what they believe.
mudkipdev 12d ago
I wonder what they saw. GPT-2 has no malware applications. Maybe they were concerned about someone using it for spam?
- andy99 12d ago
  The public discourse was very different then. Misinformation, bias, etc. Those in power were much more concerned about an LLM saying something unapproved.
  - SamBam 12d ago
    Agree. It was more a worry that someone would create a bot posting lies and bigoted ideas to Facebook/X etc.
    - afavour 12d ago
      Wild that we just accept that today, honestly.
      - newfriend 12d ago
        Wrongthink should be banned!
        Sabinus 12d ago
        Look up how the wireless radio helped spread European fascism. It's not about "the people can think this but can't think that", it's "how does this new communication platform change the existing political culture and how will we adapt?"
        afavour 12d ago
        A tedious response, no? “Automated bots posting verifiably false information online to stir up tensions should be banned” is a statement I think plenty would agree with.
      - 12d ago
deleted 12d ago
[deleted]
uejfiweun 12d ago
I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to slow everything down, the incentive is to cheat at that. I really don't think there's much of a chance we have control over the situation, we're just gonna summon these things and all we can do is pray they don't want to kill us.
- layer8 12d ago
  This is what people were saying about nuclear technology half a century ago.
  - wk_end 12d ago
    And? Nuclear technology continues to be an excellent candidate for the Great Filter.
- angoragoats 12d ago
  How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so? Put another way, could we stop anthropomorphizing the text generation algorithm, please?
  - recursive 12d ago
    How can a spring want to return to it's initial length? These machines have done plenty of things not prompted for.
    - angoragoats 12d ago
      A spring does not want to do anything. An LLM does not want to do anything. > These machines have done plenty of things not prompted for. Like what?
      - 12d ago
wmf 12d ago
I still think this was a fire drill. They knew it wasn't dangerous but they wanted the public to do their homework and come to the conclusion themselves.
- erichocean 12d ago
  So they lied to the public then? Wow, we should really trust these people today.
  - antfarm 12d ago
    Judging by what people who worked closely witb Sam Altman say about him, you should not trust an organisation led by him.
- deleted 12d ago
  [deleted]
- sublinear 12d ago
  This is it, and all the other comments here are continuing to do that homework!
- ulfw 12d ago
  You mean PR to get more publicity
- slooonz 12d ago
  That was, explicitly, part of the decision : > This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas
zenoprax 12d ago
The samples are striking in their simplicity compared to the over-controlled system prompts we have now. For example, the intended format is as follows: "GPT‑2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like—it adapts to the style and content of the conditioning text." The results don't sound anything like today's models but note that each one took 10 attempts: > ## System Prompt (human-written) > In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. > ## Model Completion (machine-written, 10 tries) > The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. > Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. > Dr. Jorge Pérez, an evolutionary biologist [...]
deleted 12d ago
[deleted]
vkvkakal 12d ago
[dead]

74 points by Jimmc414 12d ago

They were worried about the rap battle between Paul Graham and Michael O. Church. Which sadly never happened.

See also the 2023 call to “ pause for at least 6 months the training of AI systems more powerful than GPT-4.” https://futureoflife.org/open-letter/pause-giant-ai-experiments/ https://futureoflife.org/open-letter/pause-giant-ai-experime... Signed by Benjio, Musk, Woz, etc

jkuli 12d ago

If it is possible to save one life, preventing RSI AGI is an existential threat.

Jimmc414 12d ago
But why are Dario, Sam and Elon who represent "the frontier" more than any other 3 people pretending like they need government permission or intervention to pause? It seems more likely that they see open source distillation as an existential threat to their IPO plans and are playing for regulatory capture of the market.
- jonas21 12d ago
  It would require government permission. The top labs would be agreeing not to compete on model advancements for a period of time, which seems like a pretty clear violation of antitrust law.
  - anon373839 12d ago
    That isn’t any kind of antitrust violation. Agreeing not to undercut each other’s prices would be, however.
  - angoragoats 12d ago
    How is this a clear violation of antitrust law? And have any of the leaders of these companies said that they are asking for government permission for this reason?
    - jonas21 12d ago
      It's the classic case of a cartel conspiring to limit competition. Here's Matt Levine's explanation [1]: > 1. Anthropic, OpenAI, and perhaps a couple of other frontier labs are the dominant providers of frontier AI models. > 2. They can charge customers a lot of money for using their frontier models, and rather less money for older, no-longer-cutting-edge models. > 3. Training a new frontier model requires ever-increasing billions of dollars of computing power. > 4. The labs need to more or less continuously race to build new frontier models, because their competitors are all doing it, and if they don’t they will fall behind and no longer be able to charge a lot of money for their best models. (Also because they intrinsically want to build artificial superintelligence, for cancer-curing and/or killing-everyone reasons.) > 5. If they collectively slowed down, then (1) they’d spend less on compute and (2) they’d be able to charge frontier-model prices for a longer time. > 6. But if one of them slowed down, the others would eat its lunch. > 7. If they got together in a room and agreed to slow down, that would look like an antitrust conspiracy: It is generally illegal for competitors to get together and agree to limit the output of their industry. > 8. But if they publish papers about how important it is to slow down, that might have a similar coordinating function, at least among the US frontier labs if not necessarily among their Chinese competitors. > 9. And if the government believes those papers, it might help them coordinate. Maybe the government will impose pacing by regulation that the labs could not impose by agreement. Or at least the government will let them get together and agree to slow down. Amodei’s post calls for “frontier AI companies within democratic countries [to] coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress”; a footnote adds: “With government mediation or waivers of antitrust restrictions.” Just meeting in a room to establish common safety standards is legally risky; the labs can’t do it on their own unless governments affirmatively allow it. [1] https://www.bloomberg.com/opinion/newsletters/2026-09-14/ai-labs-want-someone-to-stop-them https://www.bloomberg.com/opinion/newsletters/2026-09-14/ai-...
- rfgplk 12d ago
  Meme marketing. I can attest that no one at these companies cares about laws. Not one bit.
denverllc 12d ago
... is what they believe.

mudkipdev 12d ago

I wonder what they saw. GPT-2 has no malware applications. Maybe they were concerned about someone using it for spam?

andy99 12d ago
The public discourse was very different then. Misinformation, bias, etc. Those in power were much more concerned about an LLM saying something unapproved.
- SamBam 12d ago
  Agree. It was more a worry that someone would create a bot posting lies and bigoted ideas to Facebook/X etc.
  - afavour 12d ago
    Wild that we just accept that today, honestly.
    - newfriend 12d ago
      Wrongthink should be banned!
      - Sabinus 12d ago
        Look up how the wireless radio helped spread European fascism. It's not about "the people can think this but can't think that", it's "how does this new communication platform change the existing political culture and how will we adapt?"
      - afavour 12d ago
        A tedious response, no? “Automated bots posting verifiably false information online to stir up tensions should be banned” is a statement I think plenty would agree with.
    - 12d ago

deleted 12d ago

[deleted]

uejfiweun 12d ago

I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to slow everything down, the incentive is to cheat at that. I really don't think there's much of a chance we have control over the situation, we're just gonna summon these things and all we can do is pray they don't want to kill us.

layer8 12d ago
This is what people were saying about nuclear technology half a century ago.
- wk_end 12d ago
  And? Nuclear technology continues to be an excellent candidate for the Great Filter.
angoragoats 12d ago
How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so? Put another way, could we stop anthropomorphizing the text generation algorithm, please?
- recursive 12d ago
  How can a spring want to return to it's initial length? These machines have done plenty of things not prompted for.
  - angoragoats 12d ago
    A spring does not want to do anything. An LLM does not want to do anything. > These machines have done plenty of things not prompted for. Like what?
    - 12d ago

wmf 12d ago

I still think this was a fire drill. They knew it wasn't dangerous but they wanted the public to do their homework and come to the conclusion themselves.

erichocean 12d ago
So they lied to the public then? Wow, we should really trust these people today.
- antfarm 12d ago
  Judging by what people who worked closely witb Sam Altman say about him, you should not trust an organisation led by him.
deleted 12d ago
[deleted]
sublinear 12d ago
This is it, and all the other comments here are continuing to do that homework!
ulfw 12d ago
You mean PR to get more publicity
slooonz 12d ago
That was, explicitly, part of the decision : > This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas

zenoprax 12d ago

The samples are striking in their simplicity compared to the over-controlled system prompts we have now. For example, the intended format is as follows: "GPT‑2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like—it adapts to the style and content of the conditioning text." The results don't sound anything like today's models but note that each one took 10 attempts: > ## System Prompt (human-written) > In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. > ## Model Completion (machine-written, 10 tries) > The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. > Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. > Dr. Jorge Pérez, an evolutionary biologist [...]

deleted 12d ago

[deleted]

vkvkakal 12d ago

[dead]