6 ms·

> His big concern about AI isn’t Judgment Day, but rather ‘that people will believe machines actually understand things’. I think there's a third concern that

> His big concern about AI isn’t Judgment Day, but rather ‘that people will believe machines actually understand things’. I think there's a third concern that is far more sinister and more likely than those two. Advances in AI, regardless of whether machines "surpass" or even come close to human intelligence, will be exploited by a few against the many. It will further exacerbate power imbalance and it is already happening. Forget about whether machines get "agency". Humans, in many cases, don't have agency or are rapidly losing it. They are losing it to other humans which are gaining power, financially, politically and perhaps now computationally. That's disturbing and it's part of what Jaron Lanier has been warning about for quite a while now.

agumonkey 6y ago
There's no scenario where AI become sensient through realization that one group of human is harming the other and refuses to act ?
- crispyambulance 6y ago
  I'm sure that would make provocative sci-fi. We, as a people, haven't figured out to how to get along with each other even though we're all sentient. Why should machines behave differently as they become sentient? Whatever the case, I'm less worried about machines than I am about people with machines. At least for the foreseeable future. Heck, if we're talking sci-fi, the _truly_ destructive beings in Blade Runner weren't the synthetic humans (even though they became sentient), it was the "real" humans with their organizations, their power, and their machines all at their disposal.
  - pault 6y ago
    If they had just given the replicants a visa and a software update there wouldn't have been any fuss. Of course, then we wouldn't have had that great final monolog. :)
- mhermher 6y ago
  We are all sentient, and many have had that realization. But how many have refused to act? No, we just mostly participate in that harm in order to survive.
  - agumonkey 6y ago
    The idea is that every scenario has AI is above humans as a basis.. yet it never occurs that an above level of intelligence would reach difference objectives than obedient destruction.
- morlockabove 6y ago
  Why would an AI start with a human utility function? Why would it care about that if it wasn't explicitly built in?
  - agumonkey 6y ago
    self similarity recognition ? intelligent agent, as the AI itself, would represent a form a sibling.
    - morlockabove 6y ago
      Unless the AI was formed under a selection mechanism that rewarded something like kin preference, there's no reason to think it would care. The orthogonality principle is that you could have any level of intelligence furthering any particular goal/utility function; there isn't some level of computational complexity where e.g. a paperclip maximiser's utility function magically changes. In the particular case of humans, our utility function(s) is(are) so complex that what we think of as our 'true values' can change (because our True values are some inscrutable tug-of-war between all the parts of the brain), and also we're hacked together by evolution, so just blindly trying to make a human smarter might change their values (or turn them mad, or cause seizures, or...). But this doesn't have to be the case in general for intelligent agents. In principle, you can build an AI whose terminal values remain stable as it improves its intelligence. (If this is impossible, we're doomed.) So unless you explicitly built an AI to care about all humans, there's no reason to think it magically would.
      - agumonkey 6y ago
        Can't an advanced AI produce emergent concepts and integrate them on (and as) its own ?
        morlockabove 6y ago
        What exactly does 'emergent concepts' mean, in this context?
protonfish 6y ago
I find this argument unconvincing and harmful. The "will be exploited by few against the many" could be applied to any and all technologies. Our social systems have problems, and should be improved, but this has nothing to do with science and technology. I can't see how suppressing innovation and scientific discovery as a misguided solution won't make all of our lives even worse. If we have a breakthrough that leads to a deep understanding of intelligence, it could be expected to also give us insight into our own behavior. And isn't that where most of our problems originate?
- crispyambulance 6y ago
  > I can't see how suppressing innovation... I'm not talking about "suppressing" anything. These are concerns. Valid ones. And yes, AI like all technologies is deeply, inseparably, mixed in with social and political problems. We just have to be wise with this stuff. These are tools with "big boy" consequences, powerful like nukes but in a different way. I'm not optimistic we can handle it.
  - protonfish 6y ago
    That would only be true if there were reasonable safeguards that could be suggested. But I don't see any - just the sowing of fear.
    - meowface 6y ago
      There are many people like Nick Bostrom and Eliezer Yudkowsky who are much more on the "finding possible solutions" side than the fear side. I'd consider myself a major futurist and techno-optimist who eagerly anticipates near and far AI advances, but I think some dose of fear is very healthy here, too, though. I want the top AI and AI risk researchers to constantly consider and fear worst-case scenarios, so that they're hopefully less likely to occur. Kind of like nuclear research, even if only for energy purposes: very exciting, but you should still fear accidents and their consequences. You just need to be rational about the fear and let it guide you towards developing fail-safes rather than paralyzing in despair. Pascal's Wager-style, the possibility of infinite negative utility should instill visceral terror and drive behavior almost no matter how low the probability is; except this one isn't a mugging because creating a god turns out to be a lot more plausible than a vengeful one already watching.
dleslie 6y ago
> They are losing it to other humans which are gaining power, financially, politically and perhaps now computationally. That's disturbing and it's part of what Jaron Lanier has been warning about for quite a while now. AFAICT, the average EU, Commonwealth or American citizen had their agency increased since the turn of the 20th century. Advancements in health care, widespread access to education and sturdy labour laws have vastly improved individual agency. Maybe, as a non-American, I'm missing a key factor?
- carapace 6y ago
  Hong Kong. It was the first major test of popular urban unrest vs. centralized authority (kind of ironic that an ostensibly Communist regime was the first) with the Internet in full play. The story may not be over yet, but so far it looks like the differential advantage of tech is in favor of the central authority over the mob.
  - dleslie 6y ago
    Ah, I excluded Asia on purpose. Foremost because I do not hold as much cultural understanding for its recent history, but also because I understand that there are key differences in long standing societal values. HK is certainly an interesting point to consider; I'll have to think on it. Appropriately it was a Commonwealth member for some time.
- mistermann 6y ago
  You are missing counterfactuals, at least.
carapace 6y ago
I think this is the important point. Large corporations and governments can be considered AI already, with entire humans as the neural nodes, augmented by silicon co-processors. (I like to point out that Turing was observing his own mind when he created computers, so essentially they have always been reified mechanized thought. Computers are AI already. Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.) I don't have any solutions, I just want to point out that rogue AIs are already a thing from a certain POV. What are the FAANG but a bunch of complex entities that are too large for anyone to fully control or understand? The high-frequency trading nexus is another point of dynamics where human motivation and automatic systems form a system of feedback loops beyond understanding or control. Because humans are part of these machines they cannot be considered inanimate, they are living, breathing systems: cyborgs.
- FartyMcFarter 6y ago
  > Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.) That is a very interesting way to think about it! However I disagree that it is foolish, simply because there's at least some amount of belief on the part of scientists that computers can simulate physics to any arbitrary degree of precision. If this is the case, then it follows that computers can imitate all of human thought (not just one kind of it), since humans are physical beings.
  - carapace 6y ago
    Well, it seems foolish to me to argue over whether something a computer does "really is" AI if they have been AI the whole time. Really, we're arguing about how the logical/rational part of the brain can emulate the other capabilities of the brain, which seems less interesting than the more general question of how to build any machine that can do that. Consider BEAM robotics: > BEAM robotics (from biology, electronics, aesthetics and mechanics) is a style of robotics that primarily uses simple analogue circuits, such as comparators, instead of a microprocessor in order to produce an unusually simple design. While not as flexible as microprocessor based robotics, BEAM robotics can be robust and efficient in performing the task for which it was designed. https://en.wikipedia.org/wiki/BEAM_robotics https://en.wikipedia.org/wiki/BEAM_robotics > ...computers can imitate all of human thought (not just one kind of it), since humans are physical beings. Leaving aside the metaphysical questions it raises, we may well be able to build virtual humans someday by emulation of physics of the biology of the neurology of the psychology of people. It just might not be the most efficient way to do it, eh?

agumonkey 6y ago

There's no scenario where AI become sensient through realization that one group of human is harming the other and refuses to act ?

crispyambulance 6y ago
I'm sure that would make provocative sci-fi. We, as a people, haven't figured out to how to get along with each other even though we're all sentient. Why should machines behave differently as they become sentient? Whatever the case, I'm less worried about machines than I am about people with machines. At least for the foreseeable future. Heck, if we're talking sci-fi, the _truly_ destructive beings in Blade Runner weren't the synthetic humans (even though they became sentient), it was the "real" humans with their organizations, their power, and their machines all at their disposal.
- pault 6y ago
  If they had just given the replicants a visa and a software update there wouldn't have been any fuss. Of course, then we wouldn't have had that great final monolog. :)
mhermher 6y ago
We are all sentient, and many have had that realization. But how many have refused to act? No, we just mostly participate in that harm in order to survive.
- agumonkey 6y ago
  The idea is that every scenario has AI is above humans as a basis.. yet it never occurs that an above level of intelligence would reach difference objectives than obedient destruction.
morlockabove 6y ago
Why would an AI start with a human utility function? Why would it care about that if it wasn't explicitly built in?
- agumonkey 6y ago
  self similarity recognition ? intelligent agent, as the AI itself, would represent a form a sibling.
  - morlockabove 6y ago
    Unless the AI was formed under a selection mechanism that rewarded something like kin preference, there's no reason to think it would care. The orthogonality principle is that you could have any level of intelligence furthering any particular goal/utility function; there isn't some level of computational complexity where e.g. a paperclip maximiser's utility function magically changes. In the particular case of humans, our utility function(s) is(are) so complex that what we think of as our 'true values' can change (because our True values are some inscrutable tug-of-war between all the parts of the brain), and also we're hacked together by evolution, so just blindly trying to make a human smarter might change their values (or turn them mad, or cause seizures, or...). But this doesn't have to be the case in general for intelligent agents. In principle, you can build an AI whose terminal values remain stable as it improves its intelligence. (If this is impossible, we're doomed.) So unless you explicitly built an AI to care about all humans, there's no reason to think it magically would.
    - agumonkey 6y ago
      Can't an advanced AI produce emergent concepts and integrate them on (and as) its own ?
      - morlockabove 6y ago
        What exactly does 'emergent concepts' mean, in this context?

protonfish 6y ago

I find this argument unconvincing and harmful. The "will be exploited by few against the many" could be applied to any and all technologies. Our social systems have problems, and should be improved, but this has nothing to do with science and technology. I can't see how suppressing innovation and scientific discovery as a misguided solution won't make all of our lives even worse. If we have a breakthrough that leads to a deep understanding of intelligence, it could be expected to also give us insight into our own behavior. And isn't that where most of our problems originate?

crispyambulance 6y ago
> I can't see how suppressing innovation... I'm not talking about "suppressing" anything. These are concerns. Valid ones. And yes, AI like all technologies is deeply, inseparably, mixed in with social and political problems. We just have to be wise with this stuff. These are tools with "big boy" consequences, powerful like nukes but in a different way. I'm not optimistic we can handle it.
- protonfish 6y ago
  That would only be true if there were reasonable safeguards that could be suggested. But I don't see any - just the sowing of fear.
  - meowface 6y ago
    There are many people like Nick Bostrom and Eliezer Yudkowsky who are much more on the "finding possible solutions" side than the fear side. I'd consider myself a major futurist and techno-optimist who eagerly anticipates near and far AI advances, but I think some dose of fear is very healthy here, too, though. I want the top AI and AI risk researchers to constantly consider and fear worst-case scenarios, so that they're hopefully less likely to occur. Kind of like nuclear research, even if only for energy purposes: very exciting, but you should still fear accidents and their consequences. You just need to be rational about the fear and let it guide you towards developing fail-safes rather than paralyzing in despair. Pascal's Wager-style, the possibility of infinite negative utility should instill visceral terror and drive behavior almost no matter how low the probability is; except this one isn't a mugging because creating a god turns out to be a lot more plausible than a vengeful one already watching.

dleslie 6y ago

> They are losing it to other humans which are gaining power, financially, politically and perhaps now computationally. That's disturbing and it's part of what Jaron Lanier has been warning about for quite a while now. AFAICT, the average EU, Commonwealth or American citizen had their agency increased since the turn of the 20th century. Advancements in health care, widespread access to education and sturdy labour laws have vastly improved individual agency. Maybe, as a non-American, I'm missing a key factor?

carapace 6y ago
Hong Kong. It was the first major test of popular urban unrest vs. centralized authority (kind of ironic that an ostensibly Communist regime was the first) with the Internet in full play. The story may not be over yet, but so far it looks like the differential advantage of tech is in favor of the central authority over the mob.
- dleslie 6y ago
  Ah, I excluded Asia on purpose. Foremost because I do not hold as much cultural understanding for its recent history, but also because I understand that there are key differences in long standing societal values. HK is certainly an interesting point to consider; I'll have to think on it. Appropriately it was a Commonwealth member for some time.
mistermann 6y ago
You are missing counterfactuals, at least.

carapace 6y ago

I think this is the important point. Large corporations and governments can be considered AI already, with entire humans as the neural nodes, augmented by silicon co-processors. (I like to point out that Turing was observing his own mind when he created computers, so essentially they have always been reified mechanized thought. Computers are AI already. Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.) I don't have any solutions, I just want to point out that rogue AIs are already a thing from a certain POV. What are the FAANG but a bunch of complex entities that are too large for anyone to fully control or understand? The high-frequency trading nexus is another point of dynamics where human motivation and automatic systems form a system of feedback loops beyond understanding or control. Because humans are part of these machines they cannot be considered inanimate, they are living, breathing systems: cyborgs.

FartyMcFarter 6y ago
> Ergo, what we think of as AI is really the attempt to make one kind of human thought imitate all the others. From that POV it's kind of foolish, almost a fetish.) That is a very interesting way to think about it! However I disagree that it is foolish, simply because there's at least some amount of belief on the part of scientists that computers can simulate physics to any arbitrary degree of precision. If this is the case, then it follows that computers can imitate all of human thought (not just one kind of it), since humans are physical beings.
- carapace 6y ago
  Well, it seems foolish to me to argue over whether something a computer does "really is" AI if they have been AI the whole time. Really, we're arguing about how the logical/rational part of the brain can emulate the other capabilities of the brain, which seems less interesting than the more general question of how to build any machine that can do that. Consider BEAM robotics: > BEAM robotics (from biology, electronics, aesthetics and mechanics) is a style of robotics that primarily uses simple analogue circuits, such as comparators, instead of a microprocessor in order to produce an unusually simple design. While not as flexible as microprocessor based robotics, BEAM robotics can be robust and efficient in performing the task for which it was designed. https://en.wikipedia.org/wiki/BEAM_robotics https://en.wikipedia.org/wiki/BEAM_robotics > ...computers can imitate all of human thought (not just one kind of it), since humans are physical beings. Leaving aside the metaphysical questions it raises, we may well be able to build virtual humans someday by emulation of physics of the biology of the neurology of the psychology of people. It just might not be the most efficient way to do it, eh?