4 ms·

This is actually one of the failure modes described long ago. Paperclip maximizer. An AI that has agency, intelligence, and a goal. In the course of achieving i

This is actually one of the failure modes described long ago. Paperclip maximizer. An AI that has agency, intelligence, and a goal. In the course of achieving its goal it completely neglects all other considerations. These unaccounted externalities are the danger.

eigenket 3y ago
In addition to a hypothetical AI, it sounds like you are describing real companies here.
- EliRivers 3y ago
  And likewise, corporations have already been described as AIs: https://patternsofmeaning.com/2017/11/30/ai-has-already-taken-over-its-called-the-corporation/ https://patternsofmeaning.com/2017/11/30/ai-has-already-take... et al.
- pointlessone 3y ago
  You're not wrong but also real companies consist of people. Individually they share many of their thinking patterns and moral views with people outside of those companies. So even though collective nature of companies diffuse responsibility and somewhat erode moral considerations of collective action it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. AI doesn't share thinking patterns. We're not even sure to what extent it can do that. We almost certainly don't operate on matrix multiplication while most AIs do. We don't know if AI can share our moral considerations. Many people act as if they equate intelligence with morality, as if one can not exist without other, and as if there's only human kinds of morality. Meanwhile ChatGPT will lie to your face without a hitch even though OpenAI spent a lot of time trying to make it not.
  - eigenket 3y ago
    >Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
    - pointlessone 3y ago
      It is not a counterexample. Most individuals are having hard time taking into account consequences that are decades removed and either vaguely defined (e.g. 3℃ global temperature raise is a catastrophe) or too great in scope and scale (e.g. literally global societal collapse). But even so externalities are catching up. British Petroleum is pivoting into solar, for instance even if for PR reasons. As a result of external pressure (public opinion and legislation/regulation) fossil fuel industry is in decline. Most major companies are trying to diversify and do something else instead. Companies that comprise fossil fuel industry will probably remain but they'll do something else. You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
      - eigenket 3y ago
        Whether or not BP works as a counter example has no bearing on other examples. Aramco and ExxonMobil certainly look like a counterexamples to me.
  - dwaltrip 3y ago
    Adding on to this, companies as “AI-orgs-running-via-human-individuals” can be checked by other similar entities of comparable competency, e.g. activist groups, governments, non-profits, and so on.
  - NoGravitas 3y ago
    > it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
    - nvm0n2 3y ago
      That's not because companies and political parties are mindless paperclip maximizers. It's because there isn't actually a climate crisis and almost everyone knows it, but a quasi-religious sect of true believers goes mental at anyone who points that out, so leaderships have settled into a groove of pretending to believe in order to placate them. See also: AI safety.
      - eigenket 3y ago
        Without wishing to come across as someone "going mental", it is fairly clear that the global mean temperature has been increasing https://climate.nasa.gov/vital-signs/global-temperature/ https://climate.nasa.gov/vital-signs/global-temperature/ with a change of about 1 degree Celsius over the last 100 years about 0.4 degrees over the last 20 years. This shows no sign of stopping and although the details are of course very complicated its obvious that at some point if the temperature keeps going up there are going to be problems (maybe you think there will be problems at 5 degrees, maybe more or less). Unless you can convince me there is some mechanism that will stop the temperature continuing to rise as it has over the last hundred years, I would say this constitutes something to be worried about!
        nvm0n2 3y ago
        So, 1. The word crisis implies some sort of immediate existential threat. Even the extremely biased people at the IPCC have to admit that their actual worst case scenario predictions are things like GDP being a bit lower than it otherwise would have been, 100 years from now (but still much higher than today). A lower rate of economic growth over the long term is not a crisis. 2. It's actually not clear by how much the temperature has been increasing, if at all. The error bars on climate data are enormous, which is why they don't show them, and the way the data is collected and aggregated violates many fundamental rules of good scientific practice. Having dug in deeply I concluded we actually have no real idea of what global temperatures have done over the past 100 years, because the people charged with figuring that out break the rules with wild abandon and no penalties. 3. Even as recently as 25 years ago it was widely accepted that there have been warm and colder periods in humanities recent history, with temperature ranging over very wide ranges and today's climate being nothing particularly special. All that is now verboten (for ideological reasons, pretty much) but given that, it's totally possible that the tiny temperature changes we see today will go into reverse in future. Scientists were freaking out about global cooling in the 60s and 70s for a reason: temperatures were falling (modern graphs don't show that because that history has been edited out).

eigenket 3y ago

In addition to a hypothetical AI, it sounds like you are describing real companies here.

EliRivers 3y ago
And likewise, corporations have already been described as AIs: https://patternsofmeaning.com/2017/11/30/ai-has-already-taken-over-its-called-the-corporation/ https://patternsofmeaning.com/2017/11/30/ai-has-already-take... et al.
pointlessone 3y ago
You're not wrong but also real companies consist of people. Individually they share many of their thinking patterns and moral views with people outside of those companies. So even though collective nature of companies diffuse responsibility and somewhat erode moral considerations of collective action it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. AI doesn't share thinking patterns. We're not even sure to what extent it can do that. We almost certainly don't operate on matrix multiplication while most AIs do. We don't know if AI can share our moral considerations. Many people act as if they equate intelligence with morality, as if one can not exist without other, and as if there's only human kinds of morality. Meanwhile ChatGPT will lie to your face without a hitch even though OpenAI spent a lot of time trying to make it not.
- eigenket 3y ago
  >Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
  - pointlessone 3y ago
    It is not a counterexample. Most individuals are having hard time taking into account consequences that are decades removed and either vaguely defined (e.g. 3℃ global temperature raise is a catastrophe) or too great in scope and scale (e.g. literally global societal collapse). But even so externalities are catching up. British Petroleum is pivoting into solar, for instance even if for PR reasons. As a result of external pressure (public opinion and legislation/regulation) fossil fuel industry is in decline. Most major companies are trying to diversify and do something else instead. Companies that comprise fossil fuel industry will probably remain but they'll do something else. You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
    - eigenket 3y ago
      Whether or not BP works as a counter example has no bearing on other examples. Aramco and ExxonMobil certainly look like a counterexamples to me.
- dwaltrip 3y ago
  Adding on to this, companies as “AI-orgs-running-via-human-individuals” can be checked by other similar entities of comparable competency, e.g. activist groups, governments, non-profits, and so on.
- NoGravitas 3y ago
  > it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
  - nvm0n2 3y ago
    That's not because companies and political parties are mindless paperclip maximizers. It's because there isn't actually a climate crisis and almost everyone knows it, but a quasi-religious sect of true believers goes mental at anyone who points that out, so leaderships have settled into a groove of pretending to believe in order to placate them. See also: AI safety.
    - eigenket 3y ago
      Without wishing to come across as someone "going mental", it is fairly clear that the global mean temperature has been increasing https://climate.nasa.gov/vital-signs/global-temperature/ https://climate.nasa.gov/vital-signs/global-temperature/ with a change of about 1 degree Celsius over the last 100 years about 0.4 degrees over the last 20 years. This shows no sign of stopping and although the details are of course very complicated its obvious that at some point if the temperature keeps going up there are going to be problems (maybe you think there will be problems at 5 degrees, maybe more or less). Unless you can convince me there is some mechanism that will stop the temperature continuing to rise as it has over the last hundred years, I would say this constitutes something to be worried about!
      - nvm0n2 3y ago
        So, 1. The word crisis implies some sort of immediate existential threat. Even the extremely biased people at the IPCC have to admit that their actual worst case scenario predictions are things like GDP being a bit lower than it otherwise would have been, 100 years from now (but still much higher than today). A lower rate of economic growth over the long term is not a crisis. 2. It's actually not clear by how much the temperature has been increasing, if at all. The error bars on climate data are enormous, which is why they don't show them, and the way the data is collected and aggregated violates many fundamental rules of good scientific practice. Having dug in deeply I concluded we actually have no real idea of what global temperatures have done over the past 100 years, because the people charged with figuring that out break the rules with wild abandon and no penalties. 3. Even as recently as 25 years ago it was widely accepted that there have been warm and colder periods in humanities recent history, with temperature ranging over very wide ranges and today's climate being nothing particularly special. All that is now verboten (for ideological reasons, pretty much) but given that, it's totally possible that the tiny temperature changes we see today will go into reverse in future. Scientists were freaking out about global cooling in the 60s and 70s for a reason: temperatures were falling (modern graphs don't show that because that history has been edited out).