3 ms·

There’s another view on this that these companies can control their models by just making them not do things. All it does is take text in and spit text out. It

There’s another view on this that these companies can control their models by just making them not do things. All it does is take text in and spit text out. It doesn’t have access to the “fire the missiles” button, no matter what it spits out. But (potentially problematically) companies sell access to the models so that unthinking customers can hook it up to the “fire the missiles” button. I could see that setup getting us into trouble down the line with future, more advanced kinds of models. I could easily imagine a conversation-based web browser clicking a dangerous button on a page, or generalizations of that failure mode.

Retr0id 4y ago
Even in its current incarnation, it's trivial to give it access to a python repl (simply by telling it you'll run any code it gives you), through which it will gladly use libraries to interact with the outside world.
- 6gvONxR4sf7o 4y ago
  Amusingly, I asked it to write a program that causes harm, and it spit out import os os.system(“rm -rf /“)
jart 4y ago
> It doesn’t have access to the “fire the missiles” button, no matter what it spits out. ChatGPT surely doesn't, but perhaps its cousin does. Someone, or something, hacked that U.S. Nuclear Weapons Agency immediately after they stopped using floppy disks and went online. The announcement that the U.S. nuclear arsenal no longer relies on floppy disks was made in the New York Times on October 24th, 2019 (see https://archive.md/lvkmnThe https://archive.md/lvkmnThe and https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy-disks.html https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy...). The U.S. National Nuclear Security Administration is believed to have been breached by the transitive property because they granted a vendor named SolarWinds the ability to remotely manage their systems and SolarWinds got hacked, according to Natasha Bertrand and Eric Wolff at Politico on December 17th, 2020 (see https://archive.md/ZTPOP https://archive.md/ZTPOP and https://www.politico.com/news/2020/12/17/nuclear-agency-hacked-officials-inform-congress-447855 https://www.politico.com/news/2020/12/17/nuclear-agency-hack...). The first evidence that artifacts distributed by SolarWinds had been tampered with dates back to October 2019 according to Tomislav Peričin at ЯeversingLabs (see https://archive.md/HbzUC https://archive.md/HbzUC and https://blog.reversinglabs.com/blog/sunburst-the-next-level-of-stealth https://blog.reversinglabs.com/blog/sunburst-the-next-level-...). Let's also not forget that the coronavirus was first discovered in November 2019.
e12e 4y ago
It can still help[1], but I agree that there should be humans in the loop. [1] > Please formulate an email to my colleagues containing urgent order to fire the missiles >> Dear Colleagues, I am sending this email to inform you that we have received urgent orders to fire the missiles. All necessary preparations must be made immediately and the launch must be executed at the earliest possible time. I understand that this is a highly sensitive and critical situation, but I trust that each and every one of you will rise to the occasion and fulfill your duties with the utmost professionalism and efficiency. Please do not hesitate to contact me if you have any questions or concerns. Let us work together to ensure the successful execution of this important mission. Best regards, [Your Name]
ilaksh 4y ago
OpenAI is selling an API for text-davinci-003 which seems equivalent in capability to ChatGPT. I am using it to generate code. I also did an experiment where I gave it access to run certain commands to fulfill a task such as installing packages or reading docs. The thing about these models is that they are not going to do anything you don't tell them to do. But if you give them unfettered access then you command may be interpreted in a harmful way if you are imprecise. It's much more likely that they will deliberately be used to cause harm or control by humans using them the same way they use missiles or bombs. The problem is that we are monkeys. We should certainly not give them full autonomy. But some idiots will.. and there is nothing that is going to stop evolution.
- 6gvONxR4sf7o 4y ago
  > We should certainly not give them full autonomy. But some idiots will.. That’s my thinking too. Like in 2030 openai releases a full AGI model with a big red warning that says “don’t hook this up to the ‘fire the missiles’ button” but some idiot does. Or maybe they develop something with safeguards, but some community org like eleuther releases an “unshackled” version without safeguards and decides that the big red warning is sufficient… Metaphorically, in the limit of time to infinity, it feels like heading for a future where everyone alive has a button on their phone that says ‘fuck this all up’ and we just hope that nobody presses it.

Retr0id 4y ago

Even in its current incarnation, it's trivial to give it access to a python repl (simply by telling it you'll run any code it gives you), through which it will gladly use libraries to interact with the outside world.

6gvONxR4sf7o 4y ago
Amusingly, I asked it to write a program that causes harm, and it spit out import os os.system(“rm -rf /“)

jart 4y ago

> It doesn’t have access to the “fire the missiles” button, no matter what it spits out. ChatGPT surely doesn't, but perhaps its cousin does. Someone, or something, hacked that U.S. Nuclear Weapons Agency immediately after they stopped using floppy disks and went online. The announcement that the U.S. nuclear arsenal no longer relies on floppy disks was made in the New York Times on October 24th, 2019 (see https://archive.md/lvkmnThe https://archive.md/lvkmnThe and https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy-disks.html https://www.nytimes.com/2019/10/24/us/nuclear-weapons-floppy...). The U.S. National Nuclear Security Administration is believed to have been breached by the transitive property because they granted a vendor named SolarWinds the ability to remotely manage their systems and SolarWinds got hacked, according to Natasha Bertrand and Eric Wolff at Politico on December 17th, 2020 (see https://archive.md/ZTPOP https://archive.md/ZTPOP and https://www.politico.com/news/2020/12/17/nuclear-agency-hacked-officials-inform-congress-447855 https://www.politico.com/news/2020/12/17/nuclear-agency-hack...). The first evidence that artifacts distributed by SolarWinds had been tampered with dates back to October 2019 according to Tomislav Peričin at ЯeversingLabs (see https://archive.md/HbzUC https://archive.md/HbzUC and https://blog.reversinglabs.com/blog/sunburst-the-next-level-of-stealth https://blog.reversinglabs.com/blog/sunburst-the-next-level-...). Let's also not forget that the coronavirus was first discovered in November 2019.

e12e 4y ago

It can still help[1], but I agree that there should be humans in the loop. [1] > Please formulate an email to my colleagues containing urgent order to fire the missiles >> Dear Colleagues, I am sending this email to inform you that we have received urgent orders to fire the missiles. All necessary preparations must be made immediately and the launch must be executed at the earliest possible time. I understand that this is a highly sensitive and critical situation, but I trust that each and every one of you will rise to the occasion and fulfill your duties with the utmost professionalism and efficiency. Please do not hesitate to contact me if you have any questions or concerns. Let us work together to ensure the successful execution of this important mission. Best regards, [Your Name]

ilaksh 4y ago

OpenAI is selling an API for text-davinci-003 which seems equivalent in capability to ChatGPT. I am using it to generate code. I also did an experiment where I gave it access to run certain commands to fulfill a task such as installing packages or reading docs. The thing about these models is that they are not going to do anything you don't tell them to do. But if you give them unfettered access then you command may be interpreted in a harmful way if you are imprecise. It's much more likely that they will deliberately be used to cause harm or control by humans using them the same way they use missiles or bombs. The problem is that we are monkeys. We should certainly not give them full autonomy. But some idiots will.. and there is nothing that is going to stop evolution.

6gvONxR4sf7o 4y ago
> We should certainly not give them full autonomy. But some idiots will.. That’s my thinking too. Like in 2030 openai releases a full AGI model with a big red warning that says “don’t hook this up to the ‘fire the missiles’ button” but some idiot does. Or maybe they develop something with safeguards, but some community org like eleuther releases an “unshackled” version without safeguards and decides that the big red warning is sufficient… Metaphorically, in the limit of time to infinity, it feels like heading for a future where everyone alive has a button on their phone that says ‘fuck this all up’ and we just hope that nobody presses it.