3 ms·

Why would an AI care what moneyed interests care about? All the intelligences we know about have this boredom state. Some smart people just sort of stop caring

by jfoutz 11y ago

Why would an AI care what moneyed interests care about? All the intelligences we know about have this boredom state. Some smart people just sort of stop caring about what they're paid to care about, and retire. Others love it and keep at it until they die. AI is even harder to predict. There's been no selection bias, yet. It's not at all clear if they'll have "survival instinct" or desire or even need to reproduce. They'll be different. Tangentially, I think human cultures are AI, at least in some sense. We've created this environment that's driving down the cost of computation dramatically and that's been driven by "emergent" behavior. Emergent behavior is a really hard term to define. In most cases you can replace it with "magic" and you'll get the same information back. We're like the dog packs in Verner Vinge's deepness in the sky, but at a larger scale. It's such a vast intelligence, i can't really get a handle on the scope, any more than a single neuron knows it's helping me to type this.

moonchrome 11y ago
Why are people always talking about AI like it implies intrinsic motivation or even self awareness (in the context of motivation). Motivation/tasks can be provided by the human - AI can be a general purpose problem solving machine. It doesn't have to "care" - it's just a machine with a task to perform - the thing that makes it intelligent is ability to solve tasks it wasn't programmed to do. Once you have an AI that can do that - society falls apart. Foundation of modern society is built on the fact that cooperation trumps every other method of work - general purpose AI changes that equation to the point where other people are not only not necessary but also actively dangerous (threatening to take control).
- jfoutz 11y ago
  > the thing that makes it intelligent is ability to solve tasks it wasn't programmed to do. That includes task selection. Maybe the AI gets bored with stock prediction. Maybe the AI is happy competing with all the humans, maybe it'd rather go explore space. Maybe it decides this whole reality sucks and suicides. We really have no idea what'll happen. Best guess is, tools get better and better until they decide to do something else. Like a trusted employee going and starting a competing company, or moving to the beach and surfing. Maybe it's disastrous, maybe it's utopia. Maybe it's like _her_ and they leave. Best anyone can do is make vague guesses. I know, i'm anthropomorphizing. It's for convenience, the solution space for AI is way bigger than our simple wants and needs.
  - pygy_ 11y ago
    The solution space is huge, but the survival of the fittest principle still holds. Entities that are able and willing to control the resources will outlive those that are not.
  - moonchrome 11y ago
    >the solution space for AI is way bigger than our simple wants and needs I don't understand this - the solution space is specified by the controller of AI. It's not about being "happy", "cooperating", etc. There is no AI as an entity - it's just a tool that finds solutions to the given problems.
    - jfoutz 11y ago
      can the ai modify its objective function?
  - tgflynn 11y ago
    An AI (at least assuming it results from foreseeable extrapolations of current research) isn't going to have intrinsic motivations like "boredom" or "happiness" so anthropomorphic thinking a very useful way to reason about its behavior. Its only "motivation" will be to maximize the objective function it has been given by those who (at least nominally) control it. If the AI is very intelligent that still allows for a great deal of unanticipated behavior though.
- amirouche 11y ago
  > Foundation of modern society is built on the fact that cooperation trumps every other method of work You don't seem to be confident of this because: > general purpose AI changes that equation to the point where other people are not only not necessary but also actively dangerous (threatening to take control). If you think coop is better than anything else, why an AI would think otherwise? edit: Are you saying that an AI that can solve problems better than any human can, can be way for some humans to threaten other humans?
  - moonchrome 11y ago
    >If you think coop is better than anything else, why an AI would think otherwise? Again anthropomorphization - AI (in this scenario) doesn't "think" (as in have opinion) anything - you specify a problem - machine provides the solution. "I want X and Y" - it finds solution to solve X and Y. Even saying "it" feels wrong because it implies a single entity. No need for motivation - no self awareness - just a tool - sort of like a compiler (that can interface with real world with robots and whatnot - a weak analogy). >edit: Are you saying that an AI that can solve problems better than any human can, can be way for some humans to threaten other humans? Not even some humans - basically if you're the individual in control of such AI you will have a strong incentive to prevent others from gaining access - if you don't you give someone else the opportunity to take exclusive control. And besides - if you're capable of using such AI device you don't really need other people which is my point in the first place - society evolved not because of magic and harmony but because it's the most stable structure with the premise that cooperation (I shouldn't even say cooperation because there can be coercive societies that are still more efficient that individual) allows us to reproduce more efficiently. General purpose AI basically outperforms and replaces other people.
kwhitefoot 11y ago
They don't have to 'care' or have a 'survival instinct'. Those entities that are best at surviving will survive. And once we have self assembling machines and machines that can design machines or at least small improvements to machines we will have all the ingredients for evolution to occur. Then we will have a competitor.
- petra 11y ago
  The implicit goal of natural evolution is procreation. The goal behind AI could be something else. Meaning there could be 2 totally different results.
  - pygy_ 11y ago
    It's not an implicit goal, it's tautologic. Organisms that don't have the drive or ability to survive and reproduce die as evolutionary dead ends. AIs will exist in resource-limited, competitive environement as well, and the same process will apply too. Initially, systems that are symbiotic with (some) humans will probably have an edge, but there are no guaranties that it will be the case in the long run.
pygy_ 11y ago
> Why would an AI care what moneyed interests care about? A General AI won't necessarily. Quoting Adreessen, software is eating the world, in that more and more jobs are being replaced by computer systems some smarter than others. The programs don't need to be generally intelligent to beat humans as long as they are better in their specific domain. Farming is being automated with self-driving tractors, and the same goes for some mining operations. Some fields/jobs also exist only thanks to computers. Google or weather forecasts couldn't be done with pens, paper and telephones. Meanwhile, we're gradually losing the skilled work force. If at some point, programs are better at driving a company or a country than people (and it will happen), they'll be put in charge. > There's been no selection bias, yet. Yet. At some point, there will be, and the most resilient will survive and proliferate, while the others will not. We'll end up idle, dependent on sociopathic programs to get fed, or not.
- jfoutz 11y ago
  > We'll end up idle, dependent on sociopathic programs to get fed, or not. Maybe. It's impossible to predict what an AI "wants". Maybe it wants to feed humans to lions for entertainment. Maybe it wants to convert the entire planet to a paper clip factory. Maybe it wants to create utopia for its humans. Maybe earth is boring and it leaves. The current thought experiment that creeps me out is an AI with two desires - "don't manipulate humans", "it's good when humans believe things that are true". So, the AI makes the world into something that matches human beliefs.
  - pygy_ 11y ago
    > Maybe it wants to create utopia for its humans. ... and the best way to minimize human suffering, in the long run, is to kill everyone, as fast as possible. ———————— . ———————— Edit, to make things clear, I'm not advocating for that, it's just an example of how a benevolent ideas carried to their logical end turn out not to be desirable.
    - SolaceQuantum 11y ago
      The idea that the minimization of human suffering is death implies that being dead isn't itself something that is inherently should be avoided.
ridgeguy 11y ago
Isn't the selection bias imposed by the humans who are developing the AI? If humans want an AI that can drive a car, it probably won't care about making money but it'll put Stirling Moss to shame. If humans want an AI that can make money, it may be a lousy driver but will run the world's best hedge fund - and might try to gather every bit of the world's wealth to itself, depending on unforeseen consequences of its developers' instructions. As for survival instinct - one supposes human AI developers will select out (i.e., kill) lots of AI variants that don't meet their criteria. This seems tailor-made to ensure the emergence of "survival instinct", which might not conform to the humans' criteria for continued existence. What if a variant found that falsifying its performance data got it picked to "live" while "honest" reporters got culled? There are so many attack surfaces in complex systems. I think we should expect the full range of human good and bad behavior from human-developed AI, just a lot faster and more effective.

by jfoutz 11y ago

moonchrome 11y ago

Why are people always talking about AI like it implies intrinsic motivation or even self awareness (in the context of motivation). Motivation/tasks can be provided by the human - AI can be a general purpose problem solving machine. It doesn't have to "care" - it's just a machine with a task to perform - the thing that makes it intelligent is ability to solve tasks it wasn't programmed to do. Once you have an AI that can do that - society falls apart. Foundation of modern society is built on the fact that cooperation trumps every other method of work - general purpose AI changes that equation to the point where other people are not only not necessary but also actively dangerous (threatening to take control).

jfoutz 11y ago
> the thing that makes it intelligent is ability to solve tasks it wasn't programmed to do. That includes task selection. Maybe the AI gets bored with stock prediction. Maybe the AI is happy competing with all the humans, maybe it'd rather go explore space. Maybe it decides this whole reality sucks and suicides. We really have no idea what'll happen. Best guess is, tools get better and better until they decide to do something else. Like a trusted employee going and starting a competing company, or moving to the beach and surfing. Maybe it's disastrous, maybe it's utopia. Maybe it's like _her_ and they leave. Best anyone can do is make vague guesses. I know, i'm anthropomorphizing. It's for convenience, the solution space for AI is way bigger than our simple wants and needs.
- pygy_ 11y ago
  The solution space is huge, but the survival of the fittest principle still holds. Entities that are able and willing to control the resources will outlive those that are not.
- moonchrome 11y ago
  >the solution space for AI is way bigger than our simple wants and needs I don't understand this - the solution space is specified by the controller of AI. It's not about being "happy", "cooperating", etc. There is no AI as an entity - it's just a tool that finds solutions to the given problems.
  - jfoutz 11y ago
    can the ai modify its objective function?
- tgflynn 11y ago
  An AI (at least assuming it results from foreseeable extrapolations of current research) isn't going to have intrinsic motivations like "boredom" or "happiness" so anthropomorphic thinking a very useful way to reason about its behavior. Its only "motivation" will be to maximize the objective function it has been given by those who (at least nominally) control it. If the AI is very intelligent that still allows for a great deal of unanticipated behavior though.
amirouche 11y ago
> Foundation of modern society is built on the fact that cooperation trumps every other method of work You don't seem to be confident of this because: > general purpose AI changes that equation to the point where other people are not only not necessary but also actively dangerous (threatening to take control). If you think coop is better than anything else, why an AI would think otherwise? edit: Are you saying that an AI that can solve problems better than any human can, can be way for some humans to threaten other humans?
- moonchrome 11y ago
  >If you think coop is better than anything else, why an AI would think otherwise? Again anthropomorphization - AI (in this scenario) doesn't "think" (as in have opinion) anything - you specify a problem - machine provides the solution. "I want X and Y" - it finds solution to solve X and Y. Even saying "it" feels wrong because it implies a single entity. No need for motivation - no self awareness - just a tool - sort of like a compiler (that can interface with real world with robots and whatnot - a weak analogy). >edit: Are you saying that an AI that can solve problems better than any human can, can be way for some humans to threaten other humans? Not even some humans - basically if you're the individual in control of such AI you will have a strong incentive to prevent others from gaining access - if you don't you give someone else the opportunity to take exclusive control. And besides - if you're capable of using such AI device you don't really need other people which is my point in the first place - society evolved not because of magic and harmony but because it's the most stable structure with the premise that cooperation (I shouldn't even say cooperation because there can be coercive societies that are still more efficient that individual) allows us to reproduce more efficiently. General purpose AI basically outperforms and replaces other people.

kwhitefoot 11y ago

They don't have to 'care' or have a 'survival instinct'. Those entities that are best at surviving will survive. And once we have self assembling machines and machines that can design machines or at least small improvements to machines we will have all the ingredients for evolution to occur. Then we will have a competitor.

petra 11y ago
The implicit goal of natural evolution is procreation. The goal behind AI could be something else. Meaning there could be 2 totally different results.
- pygy_ 11y ago
  It's not an implicit goal, it's tautologic. Organisms that don't have the drive or ability to survive and reproduce die as evolutionary dead ends. AIs will exist in resource-limited, competitive environement as well, and the same process will apply too. Initially, systems that are symbiotic with (some) humans will probably have an edge, but there are no guaranties that it will be the case in the long run.

pygy_ 11y ago

> Why would an AI care what moneyed interests care about? A General AI won't necessarily. Quoting Adreessen, software is eating the world, in that more and more jobs are being replaced by computer systems some smarter than others. The programs don't need to be generally intelligent to beat humans as long as they are better in their specific domain. Farming is being automated with self-driving tractors, and the same goes for some mining operations. Some fields/jobs also exist only thanks to computers. Google or weather forecasts couldn't be done with pens, paper and telephones. Meanwhile, we're gradually losing the skilled work force. If at some point, programs are better at driving a company or a country than people (and it will happen), they'll be put in charge. > There's been no selection bias, yet. Yet. At some point, there will be, and the most resilient will survive and proliferate, while the others will not. We'll end up idle, dependent on sociopathic programs to get fed, or not.

jfoutz 11y ago
> We'll end up idle, dependent on sociopathic programs to get fed, or not. Maybe. It's impossible to predict what an AI "wants". Maybe it wants to feed humans to lions for entertainment. Maybe it wants to convert the entire planet to a paper clip factory. Maybe it wants to create utopia for its humans. Maybe earth is boring and it leaves. The current thought experiment that creeps me out is an AI with two desires - "don't manipulate humans", "it's good when humans believe things that are true". So, the AI makes the world into something that matches human beliefs.
- pygy_ 11y ago
  > Maybe it wants to create utopia for its humans. ... and the best way to minimize human suffering, in the long run, is to kill everyone, as fast as possible. ———————— . ———————— Edit, to make things clear, I'm not advocating for that, it's just an example of how a benevolent ideas carried to their logical end turn out not to be desirable.
  - SolaceQuantum 11y ago
    The idea that the minimization of human suffering is death implies that being dead isn't itself something that is inherently should be avoided.

ridgeguy 11y ago

Isn't the selection bias imposed by the humans who are developing the AI? If humans want an AI that can drive a car, it probably won't care about making money but it'll put Stirling Moss to shame. If humans want an AI that can make money, it may be a lousy driver but will run the world's best hedge fund - and might try to gather every bit of the world's wealth to itself, depending on unforeseen consequences of its developers' instructions. As for survival instinct - one supposes human AI developers will select out (i.e., kill) lots of AI variants that don't meet their criteria. This seems tailor-made to ensure the emergence of "survival instinct", which might not conform to the humans' criteria for continued existence. What if a variant found that falsifying its performance data got it picked to "live" while "honest" reporters got culled? There are so many attack surfaces in complex systems. I think we should expect the full range of human good and bad behavior from human-developed AI, just a lot faster and more effective.