5 ms·

So what is Nathan Labenz saying? That GPT-4 is dangerous somehow? It will get many people out of jobs? MS Office got all the typists out of jobs. OCR and Medica

So what is Nathan Labenz saying? That GPT-4 is dangerous somehow? It will get many people out of jobs? MS Office got all the typists out of jobs. OCR and Medical Software got all the medical transcriptionists out of jobs. And they created a lot more jobs in the process. GPT-4 is a very powerful tool. It has not a whiff of AGI in it. The whole AGI "scare" seems to be extremely political.

0xDEAFBEAD 3y ago
Nathan says the initial version of GPT-4 he red-teamed was "totally amoral" and it was happy to plan assassinations for him: https://nitter.net/labenz/status/1727327464328954121#m https://nitter.net/labenz/status/1727327464328954121#m Reducing the cost of medical transcription to ~$0 is one thing. Reducing the cost of assassination to ~$0 is quite another.
- g42gregory 3y ago
  This is a piece of software. What would "totally amoral" even mean here? It's an inanimate object, it has no morals, feelings, conscience, etc... He gives it an amoral input, he gets an amoral output.
  - deleted 3y ago
    [deleted]
  - 0xDEAFBEAD 3y ago
    I mean, there's a sense in which my mind is software that's being run by my brain, right? Yet that doesn't absolve me of moral responsibility. In any case, an F16 fighter jet is amoral in a certain sense, but it wouldn't be smart to make F16s available to the average Joe so he can conduct an airstrike whenever he wants.
    - consp 3y ago
      Completely depends on your morality. I'm pretty sure there are some libertarians out there who think the most basic version of the second amendment includes owning F16 with live weapons.
      - 0xDEAFBEAD 3y ago
        Sure -- if you're a libertarian who thinks it should be possible to purchase an F16 without a background check, that seems consistent with the position that an amoral GPT-4 should be broadly available.
        civilitty 3y ago
        What kind of background check do you think exists when buying a fighter jet? It’s kind of a moot point since only one F16 exists in civilian hands but you can buy other jets with weapon hardpoints for a million. Under 3 million if you want to go supersonic too. The cheapest fighter jet is on the order of $250k. There’s zero background check.
      - jacquesm 3y ago
        Sure, but idiots are a thing and the intersection of the sets of libertarians who may believe that and idiots is hopefully empty but it may not be so such outsized power is best dealt with through a chain of command and accountability of sorts.
  - r_hoods_ghost 3y ago
    Amoral literally means "lacking a moral sense; unconcerned with the rightness or wrongness of something." Generally this is considered a problem if you are designing something that might influence the way people act.
  - ben_w 3y ago
    We don't want immoral output even for immoral input. (We do disagree about what constitutes "immoral", which makes this much harder).
    - int_19h 3y ago
      We absolutely do, though, if we want those things to e.g. write books and scripts in which characters behave immorally.
  - bayindirh 3y ago
    Then we should stop teaching Therac-25 incident to developers and remove envelope protection from planes and safety checks from nuclear reactors. Because, users should just input the moral inputs to these things. These are inanimate objects too. Oh, while we're at it, we should also remove battery charge controllers. Just do the moral and civic thing and unplug when your device charges.
    - int_19h 3y ago
      In both of your examples, the result of "immoral inputs" is immediate tangible harm. In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself.
      - bayindirh 3y ago
        > In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself. Assuming that you're not running this query over an API and relaying these answers to another control system or a gullible operator. An aircraft control computer or reactor controller won't run my commands regardless of its actuators connected or not. Same for weapon systems. Hall pass given to AI systems just because they're outputting text to a screen is staggering. Nothing prevents me to process this output automatically and actuate things.
        alexilliamson 3y ago
        Why would anyone give control of air traffic or weapons to AI? That's the key step in AGI, not some tech development. By what social process exactly would we give control of nukes to a chatbot? I can't see it happening.
        bayindirh 3y ago
        > Why would anyone give control of air traffic or weapons to AI? Simplified operations, faster reaction time, eliminating human resistance for obeying killing orders. See "War Games" [0] for a hypothetical exploration of the concept. > a chatbot. Some claim it's self-aware. Some say it called for airstrikes. Some say it gave a hit list for them. It might be a glorified Markov-chain, and I don't use it, but there's a hoard of people who follows it like it's the second Jesus, and believe what it emits. > I can't see it happening. Because, it already happened. Turkey is claimed to use completely autonomous drones in a war [1]. South Korea has autonomous sentry guns which defend DMZ [2]. [0]: https://www.imdb.com/title/tt0086567/ https://www.imdb.com/title/tt0086567/ [1]: https://www.popularmechanics.com/military/weapons/a36559508/drones-autonomously-attacked-humans-libya-united-nations-report/ https://www.popularmechanics.com/military/weapons/a36559508/... [2]: https://en.wikipedia.org/wiki/SGR-A1 https://en.wikipedia.org/wiki/SGR-A1
- logicchains 3y ago
  The cost of planning an assassination is not the same thing as the cost (and risk) of carrying out an assassination, what a stupid take.
  - esafak 3y ago
    A would be assassin would obviously ask the algorithm to suggest a low risk and cost way of assassinating.
    - SXX 3y ago
      Except the reason why we dont all just killed each other yet have nothing to do with risk or cost of killing someone. And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. Yeah very very dumb criminal can certainly benefit from it, but he can as well go on 4chan and ask about assassination there. Or on some detective book discussion club or forum.
      - 0xDEAFBEAD 3y ago
        >And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. As Nathan states: >And further, I argued that the Red Team project that I participated in did not suggest that they were on-track to achieve the level of control needed >Without safety advances, I warned that the next generation of models might very well be too dangerous to release Seems like each generation of models is getting more capable of thinking, beyond just regurgitating.
        SXX 3y ago
        I dont disagree with his points, but you completely miss the point of my post. People dont need an AI advise to commit crime and kill others. Nonestly humans they're pretty good at it using technology of 1941. You don't have bunch of cold blood killers going around not just because police is so good and killers are dumb and need AI help. It's because you live in functioning state where society have enough resources so people happy enough to instead go and kill each other in Counter Strike or Fortnite. I totally agree that AGI could be a dangerous tech, but it's will require autonomity where it can manipulate real world. So far GPT with API access is very far from that point.
- creato 3y ago
  > Reducing the cost of assassination to ~$0 is quite another. It is reducing the cost of developing an assassination plan from ~$0 to ~$0. The cost of actually executing the plan itself is not affected.
  - 0xDEAFBEAD 3y ago
    See here: https://news.ycombinator.com/item?id=38470768 https://news.ycombinator.com/item?id=38470768
  - SilasX 3y ago
    Good planning necessarily reduces the cost of something relative to the unplanned or poorly planned version. If it identifies a non-obvious means of assassination that is surprisingly easy, then it has done something "close enough" to reducing the cost to $0.
- phpisthebest 3y ago
  This is where I have issues with OpenAIs stated mission I want AI to be amoral, or rather I should say I do not want the board of OpenAI, or even the employee of OpenAI choosing what "moral" is and what is "immoral" especially given that OpenAI may be superficially "diverse" in race, gender, etc, but they sure as hell are not politically diverse, and sure has hell do not share a moral philosophy that is aligned with the vast majority of the population of humanity given the vast majority of humanity is religious in someway and I would guess the majority of OpenAI is at best agnostic if not atheist I do not want a AI Wikipedia.... aka politically biased to only 1 worldview and only useful for basic fact regurgitation like what is the speed of light
djhn 3y ago
Medical transcriptionists out of jobs? As far as I'm aware, medical transcription is still very much the domain of human experts, since getting doctors to cater their audio notes to the whims of software turned out to be impossible (at least in my corner of the EU).
- consp 3y ago
  My mom had to do this for her job and apparently some of the docs are so mumbly they have to infer a lot of the words from context and type of procedure but there is a lot of crossover everywhere so it depends a lot on which doc is mumbeling what. And yes you need special training for it (no medical degree though)
tomjakubowski 3y ago
FWIW I had a doctor's appointment just this year with a transcriptionist present. (USA)
vikramkr 3y ago
Did they really create a ton more jobs? The past few rounds of industrialization and automation have coinved with plagues/the black death that massively reduced the population, mass agitation, increasing inequality, and recently a major opioid epidemic in regions devastated by the economic changes of globalization and industrialization. I think these tools are very good and we should develop, I also think it's delusional to think it'll just balance out magically and dangerous to expect our already failed systems to protect people left behind. Doesnt exactly look like they worked any of the previous times!
ben_w 3y ago
Would you rather: 1) be surprised by and unprepared for AGI and every step on the path to that goal, or 2) have the developers of every AI examine their work for its potential impact, both when used as intended and when misused, with regard to all the known ways even non-G AI can already go wrong: bugs, making stuff up, reward hacking, domain shift, etc.; or economically speaking how many people will be made unemployed by just a fully [G]eneral self-driving AI? What happens if this is deployed over one year? Do existing LLMs get used by SEO to systematically undermine the assumptions behind Page rank and thus web search?; and culturally: how much economic harm do artists really suffer from Diffusion models? Are harms caused by AI porn unavoidable thanks to human psychology, or artefacts of out milieu that will disappear as people become accustomed to it? There's also a definition problem for AGI, with a lot of people using a standard I'd reserve for ASI. Also some people think an AGI would have to be conscious, I don't know why. The best outcome is Fully Automated Luxury Communism, but assuming the best outcome is the median outcome is how actual Communism turned into gulags and secret police.
Towaway69 3y ago
It seems that silicon valley is developing not a conscious sentient piece of software rather a conscience, a moral compass is beginning to emerge. After giving us Facebook, insta, Twitter, and ego-search, influencing many people negatively, suddenly there are moral values being discussed amongst those that decide our collective tech futures. AI will have even more influence on humankind and some are questioning the morality of money (hint: money had no morals).

0xDEAFBEAD 3y ago

Nathan says the initial version of GPT-4 he red-teamed was "totally amoral" and it was happy to plan assassinations for him: https://nitter.net/labenz/status/1727327464328954121#m https://nitter.net/labenz/status/1727327464328954121#m Reducing the cost of medical transcription to ~$0 is one thing. Reducing the cost of assassination to ~$0 is quite another.

g42gregory 3y ago
This is a piece of software. What would "totally amoral" even mean here? It's an inanimate object, it has no morals, feelings, conscience, etc... He gives it an amoral input, he gets an amoral output.
- deleted 3y ago
  [deleted]
- 0xDEAFBEAD 3y ago
  I mean, there's a sense in which my mind is software that's being run by my brain, right? Yet that doesn't absolve me of moral responsibility. In any case, an F16 fighter jet is amoral in a certain sense, but it wouldn't be smart to make F16s available to the average Joe so he can conduct an airstrike whenever he wants.
  - consp 3y ago
    Completely depends on your morality. I'm pretty sure there are some libertarians out there who think the most basic version of the second amendment includes owning F16 with live weapons.
    - 0xDEAFBEAD 3y ago
      Sure -- if you're a libertarian who thinks it should be possible to purchase an F16 without a background check, that seems consistent with the position that an amoral GPT-4 should be broadly available.
      - civilitty 3y ago
        What kind of background check do you think exists when buying a fighter jet? It’s kind of a moot point since only one F16 exists in civilian hands but you can buy other jets with weapon hardpoints for a million. Under 3 million if you want to go supersonic too. The cheapest fighter jet is on the order of $250k. There’s zero background check.
    - jacquesm 3y ago
      Sure, but idiots are a thing and the intersection of the sets of libertarians who may believe that and idiots is hopefully empty but it may not be so such outsized power is best dealt with through a chain of command and accountability of sorts.
- r_hoods_ghost 3y ago
  Amoral literally means "lacking a moral sense; unconcerned with the rightness or wrongness of something." Generally this is considered a problem if you are designing something that might influence the way people act.
- ben_w 3y ago
  We don't want immoral output even for immoral input. (We do disagree about what constitutes "immoral", which makes this much harder).
  - int_19h 3y ago
    We absolutely do, though, if we want those things to e.g. write books and scripts in which characters behave immorally.
- bayindirh 3y ago
  Then we should stop teaching Therac-25 incident to developers and remove envelope protection from planes and safety checks from nuclear reactors. Because, users should just input the moral inputs to these things. These are inanimate objects too. Oh, while we're at it, we should also remove battery charge controllers. Just do the moral and civic thing and unplug when your device charges.
  - int_19h 3y ago
    In both of your examples, the result of "immoral inputs" is immediate tangible harm. In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself.
    - bayindirh 3y ago
      > In case of GPT-4 or any other LLM, it's merely "immoral output" - i.e. text. It does not harm anyone by itself. Assuming that you're not running this query over an API and relaying these answers to another control system or a gullible operator. An aircraft control computer or reactor controller won't run my commands regardless of its actuators connected or not. Same for weapon systems. Hall pass given to AI systems just because they're outputting text to a screen is staggering. Nothing prevents me to process this output automatically and actuate things.
      - alexilliamson 3y ago
        Why would anyone give control of air traffic or weapons to AI? That's the key step in AGI, not some tech development. By what social process exactly would we give control of nukes to a chatbot? I can't see it happening.
        bayindirh 3y ago
        > Why would anyone give control of air traffic or weapons to AI? Simplified operations, faster reaction time, eliminating human resistance for obeying killing orders. See "War Games" [0] for a hypothetical exploration of the concept. > a chatbot. Some claim it's self-aware. Some say it called for airstrikes. Some say it gave a hit list for them. It might be a glorified Markov-chain, and I don't use it, but there's a hoard of people who follows it like it's the second Jesus, and believe what it emits. > I can't see it happening. Because, it already happened. Turkey is claimed to use completely autonomous drones in a war [1]. South Korea has autonomous sentry guns which defend DMZ [2]. [0]: https://www.imdb.com/title/tt0086567/ https://www.imdb.com/title/tt0086567/ [1]: https://www.popularmechanics.com/military/weapons/a36559508/drones-autonomously-attacked-humans-libya-united-nations-report/ https://www.popularmechanics.com/military/weapons/a36559508/... [2]: https://en.wikipedia.org/wiki/SGR-A1 https://en.wikipedia.org/wiki/SGR-A1
logicchains 3y ago
The cost of planning an assassination is not the same thing as the cost (and risk) of carrying out an assassination, what a stupid take.
- esafak 3y ago
  A would be assassin would obviously ask the algorithm to suggest a low risk and cost way of assassinating.
  - SXX 3y ago
    Except the reason why we dont all just killed each other yet have nothing to do with risk or cost of killing someone. And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. Yeah very very dumb criminal can certainly benefit from it, but he can as well go on 4chan and ask about assassination there. Or on some detective book discussion club or forum.
    - 0xDEAFBEAD 3y ago
      >And everything LLM can come up with will be exactly the same information you can find in any fiction detective book or TV series about crime. As Nathan states: >And further, I argued that the Red Team project that I participated in did not suggest that they were on-track to achieve the level of control needed >Without safety advances, I warned that the next generation of models might very well be too dangerous to release Seems like each generation of models is getting more capable of thinking, beyond just regurgitating.
      - SXX 3y ago
        I dont disagree with his points, but you completely miss the point of my post. People dont need an AI advise to commit crime and kill others. Nonestly humans they're pretty good at it using technology of 1941. You don't have bunch of cold blood killers going around not just because police is so good and killers are dumb and need AI help. It's because you live in functioning state where society have enough resources so people happy enough to instead go and kill each other in Counter Strike or Fortnite. I totally agree that AGI could be a dangerous tech, but it's will require autonomity where it can manipulate real world. So far GPT with API access is very far from that point.
creato 3y ago
> Reducing the cost of assassination to ~$0 is quite another. It is reducing the cost of developing an assassination plan from ~$0 to ~$0. The cost of actually executing the plan itself is not affected.
- 0xDEAFBEAD 3y ago
  See here: https://news.ycombinator.com/item?id=38470768 https://news.ycombinator.com/item?id=38470768
- SilasX 3y ago
  Good planning necessarily reduces the cost of something relative to the unplanned or poorly planned version. If it identifies a non-obvious means of assassination that is surprisingly easy, then it has done something "close enough" to reducing the cost to $0.
phpisthebest 3y ago
This is where I have issues with OpenAIs stated mission I want AI to be amoral, or rather I should say I do not want the board of OpenAI, or even the employee of OpenAI choosing what "moral" is and what is "immoral" especially given that OpenAI may be superficially "diverse" in race, gender, etc, but they sure as hell are not politically diverse, and sure has hell do not share a moral philosophy that is aligned with the vast majority of the population of humanity given the vast majority of humanity is religious in someway and I would guess the majority of OpenAI is at best agnostic if not atheist I do not want a AI Wikipedia.... aka politically biased to only 1 worldview and only useful for basic fact regurgitation like what is the speed of light

djhn 3y ago

Medical transcriptionists out of jobs? As far as I'm aware, medical transcription is still very much the domain of human experts, since getting doctors to cater their audio notes to the whims of software turned out to be impossible (at least in my corner of the EU).

consp 3y ago
My mom had to do this for her job and apparently some of the docs are so mumbly they have to infer a lot of the words from context and type of procedure but there is a lot of crossover everywhere so it depends a lot on which doc is mumbeling what. And yes you need special training for it (no medical degree though)

tomjakubowski 3y ago

FWIW I had a doctor's appointment just this year with a transcriptionist present. (USA)

vikramkr 3y ago

Did they really create a ton more jobs? The past few rounds of industrialization and automation have coinved with plagues/the black death that massively reduced the population, mass agitation, increasing inequality, and recently a major opioid epidemic in regions devastated by the economic changes of globalization and industrialization. I think these tools are very good and we should develop, I also think it's delusional to think it'll just balance out magically and dangerous to expect our already failed systems to protect people left behind. Doesnt exactly look like they worked any of the previous times!

ben_w 3y ago

Would you rather: 1) be surprised by and unprepared for AGI and every step on the path to that goal, or 2) have the developers of every AI examine their work for its potential impact, both when used as intended and when misused, with regard to all the known ways even non-G AI can already go wrong: bugs, making stuff up, reward hacking, domain shift, etc.; or economically speaking how many people will be made unemployed by just a fully [G]eneral self-driving AI? What happens if this is deployed over one year? Do existing LLMs get used by SEO to systematically undermine the assumptions behind Page rank and thus web search?; and culturally: how much economic harm do artists really suffer from Diffusion models? Are harms caused by AI porn unavoidable thanks to human psychology, or artefacts of out milieu that will disappear as people become accustomed to it? There's also a definition problem for AGI, with a lot of people using a standard I'd reserve for ASI. Also some people think an AGI would have to be conscious, I don't know why. The best outcome is Fully Automated Luxury Communism, but assuming the best outcome is the median outcome is how actual Communism turned into gulags and secret police.

Towaway69 3y ago

It seems that silicon valley is developing not a conscious sentient piece of software rather a conscience, a moral compass is beginning to emerge. After giving us Facebook, insta, Twitter, and ego-search, influencing many people negatively, suddenly there are moral values being discussed amongst those that decide our collective tech futures. AI will have even more influence on humankind and some are questioning the morality of money (hint: money had no morals).