4 ms·

Nathan says the initial version of GPT-4 he red-teamed was "totally amoral" and it was happy to plan assassinations for him: https://nitter.net/labenz/status/17

Nathan says the initial version of GPT-4 he red-teamed was "totally amoral" and it was happy to plan assassinations for him: https://nitter.net/labenz/status/1727327464328954121#m https://nitter.net/labenz/status/1727327464328954121#m Reducing the cost of medical transcription to ~$0 is one thing. Reducing the cost of assassination to ~$0 is quite another.

g42gregory 3y ago
This is a piece of software. What would "totally amoral" even mean here? It's an inanimate object, it has no morals, feelings, conscience, etc... He gives it an amoral input, he gets an amoral output.
- deleted 3y ago
  [deleted]
- 0xDEAFBEAD 3y ago
  I mean, there's a sense in which my mind is software that's being run by my brain, right? Yet that doesn't absolve me of moral responsibility. In any case, an F16 fighter jet is amoral in a certain sense, but it wouldn't be smart to make F16s available to the average Joe so he can conduct an airstrike whenever he wants.
  - consp 3y ago
    Completely depends on your morality. I'm pretty sure there are some libertarians out there who think the most basic version of the second amendment includes owning F16 with live weapons.
    - 0xDEAFBEAD 3y ago
      Sure -- if you're a libertarian who thinks it should be possible to purchase an F16 without a background check, that seems consistent with the position that an amoral GPT-4 should be broadly available.
      - civilitty 3y ago
        What kind of background check do you think exists when buying a fighter jet? It’s kind of a moot point since only one F16 exists in civilian hands but you can buy other jets with weapon hardpoints for a million. Under 3 million if you want to go supersonic too. The cheapest fighter jet is on the order of $250k. There’s zero background check.
    - jacquesm 3y ago
      Sure, but idiots are a thing and the intersection of the sets of libertarians who may believe that and idiots is hopefully empty but it may not be so such outsized power is best dealt with through a chain of command and accountability of sorts.
- r_hoods_ghost 3y ago
  Amoral literally means "lacking a moral sense; unconcerned with the rightness or wrongness of something." Generally this is considered a problem if you are designing something that might influence the way people act.
- ben_w 3y ago
  We don't want immoral output even for immoral input. (We do disagree about what constitutes "immoral", which makes this much harder).
  - int_19h 3y ago
    We absolutely do, though, if we want those things to e.g. write books and scripts in which characters behave immorally.
- bayindirh 3y ago
  Then we should stop teaching Therac-25 incident to developers and remove envelope protection from planes and safety checks from nuclear reactors. Because, users should just input the moral inputs to these things. These are inanimate objects too. Oh, while we're at it, we should also remove battery charge controllers. Just do the moral and civic thing and unplug when your device charges.
  - int_19h 3y ago
    In both of your examples, the result of "immoral inputs" is immediate tangible harm. In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself.
    - bayindirh 3y ago
      > In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself. Assuming that you're not running this query over an API and relaying these answers to another control system or a gullible operator. An aircraft control computer or reactor controller won't run my commands regardless of its actuators connected or not. Same for weapon systems. Hall pass given to AI systems just because they're outputting text to a screen is staggering. Nothing prevents me to process this output automatically and actuate things.
      - alexilliamson 3y ago
        Why would anyone give control of air traffic or weapons to AI? That's the key step in AGI, not some tech development. By what social process exactly would we give control of nukes to a chatbot? I can't see it happening.
        bayindirh 3y ago
        > Why would anyone give control of air traffic or weapons to AI? Simplified operations, faster reaction time, eliminating human resistance for obeying killing orders. See "War Games" [0] for a hypothetical exploration of the concept. > a chatbot. Some claim it's self-aware. Some say it called for airstrikes. Some say it gave a hit list for them. It might be a glorified Markov-chain, and I don't use it, but there's a hoard of people who follows it like it's the second Jesus, and believe what it emits. > I can't see it happening. Because, it already happened. Turkey is claimed to use completely autonomous drones in a war [1]. South Korea has autonomous sentry guns which defend DMZ [2]. [0]: https://www.imdb.com/title/tt0086567/ https://www.imdb.com/title/tt0086567/ [1]: https://www.popularmechanics.com/military/weapons/a36559508/drones-autonomously-attacked-humans-libya-united-nations-report/ https://www.popularmechanics.com/military/weapons/a36559508/... [2]: https://en.wikipedia.org/wiki/SGR-A1 https://en.wikipedia.org/wiki/SGR-A1
logicchains 3y ago
The cost of planning an assassination is not the same thing as the cost (and risk) of carrying out an assassination, what a stupid take.
- esafak 3y ago
  A would be assassin would obviously ask the algorithm to suggest a low risk and cost way of assassinating.
  - SXX 3y ago
    Except the reason why we dont all just killed each other yet have nothing to do with risk or cost of killing someone. And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. Yeah very very dumb criminal can certainly benefit from it, but he can as well go on 4chan and ask about assassination there. Or on some detective book discussion club or forum.
    - 0xDEAFBEAD 3y ago
      >And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. As Nathan states: >And further, I argued that the Red Team project that I participated in did not suggest that they were on-track to achieve the level of control needed >Without safety advances, I warned that the next generation of models might very well be too dangerous to release Seems like each generation of models is getting more capable of thinking, beyond just regurgitating.
      - SXX 3y ago
        I dont disagree with his points, but you completely miss the point of my post. People dont need an AI advise to commit crime and kill others. Nonestly humans they're pretty good at it using technology of 1941. You don't have bunch of cold blood killers going around not just because police is so good and killers are dumb and need AI help. It's because you live in functioning state where society have enough resources so people happy enough to instead go and kill each other in Counter Strike or Fortnite. I totally agree that AGI could be a dangerous tech, but it's will require autonomity where it can manipulate real world. So far GPT with API access is very far from that point.
creato 3y ago
> Reducing the cost of assassination to ~$0 is quite another. It is reducing the cost of developing an assassination plan from ~$0 to ~$0. The cost of actually executing the plan itself is not affected.
- 0xDEAFBEAD 3y ago
  See here: https://news.ycombinator.com/item?id=38470768 https://news.ycombinator.com/item?id=38470768
- SilasX 3y ago
  Good planning necessarily reduces the cost of something relative to the unplanned or poorly planned version. If it identifies a non-obvious means of assassination that is surprisingly easy, then it has done something "close enough" to reducing the cost to $0.
phpisthebest 3y ago
This is where I have issues with OpenAIs stated mission I want AI to be amoral, or rather I should say I do not want the board of OpenAI, or even the employee of OpenAI choosing what "moral" is and what is "immoral" especially given that OpenAI may be superficially "diverse" in race, gender, etc, but they sure as hell are not politically diverse, and sure has hell do not share a moral philosophy that is aligned with the vast majority of the population of humanity given the vast majority of humanity is religious in someway and I would guess the majority of OpenAI is at best agnostic if not atheist I do not want a AI Wikipedia.... aka politically biased to only 1 worldview and only useful for basic fact regurgitation like what is the speed of light

g42gregory 3y ago

This is a piece of software. What would "totally amoral" even mean here? It's an inanimate object, it has no morals, feelings, conscience, etc... He gives it an amoral input, he gets an amoral output.

deleted 3y ago
[deleted]
0xDEAFBEAD 3y ago
I mean, there's a sense in which my mind is software that's being run by my brain, right? Yet that doesn't absolve me of moral responsibility. In any case, an F16 fighter jet is amoral in a certain sense, but it wouldn't be smart to make F16s available to the average Joe so he can conduct an airstrike whenever he wants.
- consp 3y ago
  Completely depends on your morality. I'm pretty sure there are some libertarians out there who think the most basic version of the second amendment includes owning F16 with live weapons.
  - 0xDEAFBEAD 3y ago
    Sure -- if you're a libertarian who thinks it should be possible to purchase an F16 without a background check, that seems consistent with the position that an amoral GPT-4 should be broadly available.
    - civilitty 3y ago
      What kind of background check do you think exists when buying a fighter jet? It’s kind of a moot point since only one F16 exists in civilian hands but you can buy other jets with weapon hardpoints for a million. Under 3 million if you want to go supersonic too. The cheapest fighter jet is on the order of $250k. There’s zero background check.
  - jacquesm 3y ago
    Sure, but idiots are a thing and the intersection of the sets of libertarians who may believe that and idiots is hopefully empty but it may not be so such outsized power is best dealt with through a chain of command and accountability of sorts.
r_hoods_ghost 3y ago
Amoral literally means "lacking a moral sense; unconcerned with the rightness or wrongness of something." Generally this is considered a problem if you are designing something that might influence the way people act.
ben_w 3y ago
We don't want immoral output even for immoral input. (We do disagree about what constitutes "immoral", which makes this much harder).
- int_19h 3y ago
  We absolutely do, though, if we want those things to e.g. write books and scripts in which characters behave immorally.
bayindirh 3y ago
Then we should stop teaching Therac-25 incident to developers and remove envelope protection from planes and safety checks from nuclear reactors. Because, users should just input the moral inputs to these things. These are inanimate objects too. Oh, while we're at it, we should also remove battery charge controllers. Just do the moral and civic thing and unplug when your device charges.
- int_19h 3y ago
  In both of your examples, the result of "immoral inputs" is immediate tangible harm. In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself.
  - bayindirh 3y ago
    > In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself. Assuming that you're not running this query over an API and relaying these answers to another control system or a gullible operator. An aircraft control computer or reactor controller won't run my commands regardless of its actuators connected or not. Same for weapon systems. Hall pass given to AI systems just because they're outputting text to a screen is staggering. Nothing prevents me to process this output automatically and actuate things.
    - alexilliamson 3y ago
      Why would anyone give control of air traffic or weapons to AI? That's the key step in AGI, not some tech development. By what social process exactly would we give control of nukes to a chatbot? I can't see it happening.
      - bayindirh 3y ago
        > Why would anyone give control of air traffic or weapons to AI? Simplified operations, faster reaction time, eliminating human resistance for obeying killing orders. See "War Games" [0] for a hypothetical exploration of the concept. > a chatbot. Some claim it's self-aware. Some say it called for airstrikes. Some say it gave a hit list for them. It might be a glorified Markov-chain, and I don't use it, but there's a hoard of people who follows it like it's the second Jesus, and believe what it emits. > I can't see it happening. Because, it already happened. Turkey is claimed to use completely autonomous drones in a war [1]. South Korea has autonomous sentry guns which defend DMZ [2]. [0]: https://www.imdb.com/title/tt0086567/ https://www.imdb.com/title/tt0086567/ [1]: https://www.popularmechanics.com/military/weapons/a36559508/drones-autonomously-attacked-humans-libya-united-nations-report/ https://www.popularmechanics.com/military/weapons/a36559508/... [2]: https://en.wikipedia.org/wiki/SGR-A1 https://en.wikipedia.org/wiki/SGR-A1

logicchains 3y ago

The cost of planning an assassination is not the same thing as the cost (and risk) of carrying out an assassination, what a stupid take.

esafak 3y ago
A would be assassin would obviously ask the algorithm to suggest a low risk and cost way of assassinating.
- SXX 3y ago
  Except the reason why we dont all just killed each other yet have nothing to do with risk or cost of killing someone. And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. Yeah very very dumb criminal can certainly benefit from it, but he can as well go on 4chan and ask about assassination there. Or on some detective book discussion club or forum.
  - 0xDEAFBEAD 3y ago
    >And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. As Nathan states: >And further, I argued that the Red Team project that I participated in did not suggest that they were on-track to achieve the level of control needed >Without safety advances, I warned that the next generation of models might very well be too dangerous to release Seems like each generation of models is getting more capable of thinking, beyond just regurgitating.
    - SXX 3y ago
      I dont disagree with his points, but you completely miss the point of my post. People dont need an AI advise to commit crime and kill others. Nonestly humans they're pretty good at it using technology of 1941. You don't have bunch of cold blood killers going around not just because police is so good and killers are dumb and need AI help. It's because you live in functioning state where society have enough resources so people happy enough to instead go and kill each other in Counter Strike or Fortnite. I totally agree that AGI could be a dangerous tech, but it's will require autonomity where it can manipulate real world. So far GPT with API access is very far from that point.

creato 3y ago

> Reducing the cost of assassination to ~$0 is quite another. It is reducing the cost of developing an assassination plan from ~$0 to ~$0. The cost of actually executing the plan itself is not affected.

0xDEAFBEAD 3y ago
See here: https://news.ycombinator.com/item?id=38470768 https://news.ycombinator.com/item?id=38470768
SilasX 3y ago
Good planning necessarily reduces the cost of something relative to the unplanned or poorly planned version. If it identifies a non-obvious means of assassination that is surprisingly easy, then it has done something "close enough" to reducing the cost to $0.

phpisthebest 3y ago

This is where I have issues with OpenAIs stated mission I want AI to be amoral, or rather I should say I do not want the board of OpenAI, or even the employee of OpenAI choosing what "moral" is and what is "immoral" especially given that OpenAI may be superficially "diverse" in race, gender, etc, but they sure as hell are not politically diverse, and sure has hell do not share a moral philosophy that is aligned with the vast majority of the population of humanity given the vast majority of humanity is religious in someway and I would guess the majority of OpenAI is at best agnostic if not atheist I do not want a AI Wikipedia.... aka politically biased to only 1 worldview and only useful for basic fact regurgitation like what is the speed of light