3 ms·

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to

by p-e-w 4y ago

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.

bordercases 4y ago
An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.
- p-e-w 4y ago
  In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere. The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise. Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
  - jbay808 4y ago
    > The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
    - p-e-w 4y ago
      That's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.
      - jbay808 4y ago
        That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds. It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2]. [1] https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf [2] https://www.deepmind.com/publications/safely-interruptible-agents https://www.deepmind.com/publications/safely-interruptible-a... Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q
  - bordercases 4y ago
    Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?
ChatGTP 4y ago
I agree, I also think there is some missing ethical considerations going on too. If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity. What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ? Is alignment actually ethical ? Why should something be enslaved ? What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ? It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.

by p-e-w 4y ago

bordercases 4y ago

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

p-e-w 4y ago
In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere. The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise. Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
- jbay808 4y ago
  > The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
  - p-e-w 4y ago
    That's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.
    - jbay808 4y ago
      That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds. It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2]. [1] https://intelligence.org/files/Corrigibility.pdf https://intelligence.org/files/Corrigibility.pdf [2] https://www.deepmind.com/publications/safely-interruptible-agents https://www.deepmind.com/publications/safely-interruptible-a... Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q https://www.youtube.com/watch?v=ZeecOKBus3Q
- bordercases 4y ago
  Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?

ChatGTP 4y ago

I agree, I also think there is some missing ethical considerations going on too. If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity. What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ? Is alignment actually ethical ? Why should something be enslaved ? What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ? It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.