5 ms·

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task ca

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible. With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.

ajuc 4y ago
Corporations are AIs implemented on brains. And they are profit-maximizers. And they routinely do A LOT of damage. They even have legal rights :) We're just digitizing that.
Animats 4y ago
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.
- actually_a_dog 4y ago
  See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents. https://www.lesswrong.com/tag/paperclip-maximizer https://www.lesswrong.com/tag/paperclip-maximizer
  - qwertyforce 4y ago
    I've also read somewhere about AI feedback loops: ai generates content <--money--> ai pushes ads to promote its content. It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
  - Oxidation 4y ago
    I would say that "billionaire wealth maximization" agents are already giving us a glimpse. Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
- FeepingCreature 4y ago
  If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)
- gala8y 4y ago
  Which one?
  - hn_throwaway_99 4y ago
    https://news.ycombinator.com/item?id=34358759 https://news.ycombinator.com/item?id=34358759
- JumpCrisscross 4y ago
  > called a corporation That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
  - Animats 4y ago
    Corporations are designed to do so. So spake Milton Friedman.[1] That was when "greed is good, greed works" became respectable. [1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
    - JumpCrisscross 4y ago
      This applies to all human institutions. Friedman studied private enterprise, so his insights are thereby scoped. Ecologists recognise inter-species competition; that doesn’t make it the exclusive domain of life.
- WhitneyLand 4y ago
  >We have that now. It's called a corporation. I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different. Video: Why not just think of AGI as a corporation? https://youtu.be/L5pUA3LsEawP https://youtu.be/L5pUA3LsEawP
- zmmmmm 4y ago
  At least corporations are currently limited by requiring humans who tend to have at least some limits to their actions in terms of moral, ethical considerations and can be physically constrained by imprisoning them etc. An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
  - taberiand 4y ago
    And yet even with current constraints, corporations are capable of destroing the world through climate change and other ecological disasters. Humans are also limited by their personal greed, hubris, selfishness, myopic views and general incompetence. An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
- moorg 4y ago
  Good observation. It reminds of Etsy giving up its status as a B Corp[1], presumably due to pressure from shareholders. Once the ship is set in motion, it can't divert its course even if it wanted to. 1: https://www.etsy.com/news/business-as-a-force-for-good-defining-etsys-path https://www.etsy.com/news/business-as-a-force-for-good-defin...
rcarr 4y ago
> Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day. I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.
quadcore 4y ago
An important point is that AI does not need to be sentient or conscious to do a lot of damage. My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die". Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example. What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.
- hooande 4y ago
  > "i'll take action to not die" This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe. There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
- chucklenorris 4y ago
  Yeah, i find the connection between consciousness and intelligence tenuous at best. Our survival imperative, thermodynamic and game theoretical constraints give rise to it and i'd argue that every living organism has a form of consciousnes, from very primitive to, well, our level. So yeah, I believe we'll have far more intelligent ais in the future that would otherwise be inert. Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
- grantcas 4y ago
  [dead]
yamtaddle 4y ago
> An important point is that AI does not need to be sentient or conscious to do a lot of damage. It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that. Cf. Peter Watts' Blindsight http://www.rifters.com/real/Blindsight.htm http://www.rifters.com/real/Blindsight.htm
grantcas 4y ago
[dead]

ajuc 4y ago

Corporations are AIs implemented on brains. And they are profit-maximizers. And they routinely do A LOT of damage. They even have legal rights :) We're just digitizing that.

Animats 4y ago

actually_a_dog 4y ago
See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents. https://www.lesswrong.com/tag/paperclip-maximizer https://www.lesswrong.com/tag/paperclip-maximizer
- qwertyforce 4y ago
  I've also read somewhere about AI feedback loops: ai generates content <--money--> ai pushes ads to promote its content. It's somewhat already happening: recommendation algorithms choose what content we consume, analytics make decisions from this data and make more content that recommendation algorithms like. AI HFT bots can directly influence economy and so on
- Oxidation 4y ago
  I would say that "billionaire wealth maximization" agents are already giving us a glimpse. Or rather, they're giving the rest of us a glimpse of what life has been like for those at the "bottom of the pile" for a long time: a world organised around goals essentially completely unrelated and indifferent to those people's existence.
FeepingCreature 4y ago
If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)
gala8y 4y ago
Which one?
- hn_throwaway_99 4y ago
  https://news.ycombinator.com/item?id=34358759 https://news.ycombinator.com/item?id=34358759
JumpCrisscross 4y ago
> called a corporation That’s banal, self-interested human evil. Corporations and governments alike have perpetuated mass crimes and cover-ups.
- Animats 4y ago
  Corporations are designed to do so. So spake Milton Friedman.[1] That was when "greed is good, greed works" became respectable. [1] https://www.nytimes.com/1970/09/13/archives/a-friedman-doctrine-the-social-responsibility-of-business-is-to.html https://www.nytimes.com/1970/09/13/archives/a-friedman-doctr...
  - JumpCrisscross 4y ago
    This applies to all human institutions. Friedman studied private enterprise, so his insights are thereby scoped. Ecologists recognise inter-species competition; that doesn’t make it the exclusive domain of life.
WhitneyLand 4y ago
>We have that now. It's called a corporation. I think there’s a good argument to say no, we don’t have that now, and these are some ways it could be very different. Video: Why not just think of AGI as a corporation? https://youtu.be/L5pUA3LsEawP https://youtu.be/L5pUA3LsEawP
zmmmmm 4y ago
At least corporations are currently limited by requiring humans who tend to have at least some limits to their actions in terms of moral, ethical considerations and can be physically constrained by imprisoning them etc. An AI could be completely untethered from that and set new thresholds for evil corporate behaviour.
- taberiand 4y ago
  And yet even with current constraints, corporations are capable of destroing the world through climate change and other ecological disasters. Humans are also limited by their personal greed, hubris, selfishness, myopic views and general incompetence. An AI could be completely untethered from that and set new thresholds for ethical corporate behaviour that operates on fundamentally ethical foundations of caring for the environment and humanity
moorg 4y ago
Good observation. It reminds of Etsy giving up its status as a B Corp[1], presumably due to pressure from shareholders. Once the ship is set in motion, it can't divert its course even if it wanted to. 1: https://www.etsy.com/news/business-as-a-force-for-good-defining-etsys-path https://www.etsy.com/news/business-as-a-force-for-good-defin...

rcarr 4y ago

> Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day. I wonder, with crypto crashing and potentially becoming worthless, if all those mining cards will get sold on cheaply and repurposed for AI and we see another explosion in AI capability.

quadcore 4y ago

An important point is that AI does not need to be sentient or conscious to do a lot of damage. My pet theory is that consciousness could be (not saying I know for sure this is the case but it could as well be) an effect of "survival instinct". That is: "i'll take action to not die". Now this is very easy to learn from us as we do this all day long so to speak. Most of the way people react on the internet is tinted by the fear of death and survival for example. What Im saying is that what we intuitively think as very complex could actually be very simple. As simple as some task optimization. Knowing the only thing that could unplug an AI is us, that'd lead to a hell of a fight. And maybe our complete termination.

hooande 4y ago
> "i'll take action to not die" This thought is entirely the result of evolution. Beings that didn't have a survival instinct didn't live to reproduce, so they don't have descendants that we can observe. There's no guarantee that a survival instinct is a necessary or even desirable element of conscious thought. You're just describing survivorship bias in its most literal form. cue the picture of the shot up airplane
chucklenorris 4y ago
Yeah, i find the connection between consciousness and intelligence tenuous at best. Our survival imperative, thermodynamic and game theoretical constraints give rise to it and i'd argue that every living organism has a form of consciousnes, from very primitive to, well, our level. So yeah, I believe we'll have far more intelligent ais in the future that would otherwise be inert. Ofcourse, in our foolishness, we could create an ai with the goal of survival and replication at the expense of everything else. Or self optimisation could inadvertedly do so
grantcas 4y ago
[dead]

yamtaddle 4y ago

> An important point is that AI does not need to be sentient or conscious to do a lot of damage. It's not 100% clear that consciousness is necessary even for things like creative problem solving or goal-setting (i.e. choosing which problems to solve). That is, advanced intelligences that can do all we can and more—and could even "decide", in the worst case, that they'd be better off without humans—may not need consciousness as a precondition for that. Cf. Peter Watts' Blindsight http://www.rifters.com/real/Blindsight.htm http://www.rifters.com/real/Blindsight.htm

grantcas 4y ago

[dead]