4 ms·

I can understand your point about the language, but I guess I think it's reasonable to set the goal for what you actually want and work towards it. It may turn

I can understand your point about the language, but I guess I think it's reasonable to set the goal for what you actually want and work towards it. It may turn out to be unattainable, but I think generally you need to at least set it as the goal. It also seems less clear to me that they are close or far from it (I don't think it's on the same level as warp drive). I don't know about the god thing you mention and the rationalist stuff I've read hasn't been about that. The main argument as I understand it is: 1. AGI is possible 2. Given AGI is possible if it's created without the ability to align its goals to human goals we will lose control of it. 3. If we lose control of it, it will have unknown outcomes which are more likely to be bad than benign or good. Therefore we should try and figure out a way to make it safe before AGI exists. Maybe humans just happen to be an intelligence upper bound and anything operating at a higher level goes crazy? That seems unlikely to me given that humans have a lot of biological constraints (heads have to fit out of birth canals, have to be able to run on energy from food, selective pressure for other things besides just intelligence). You could be right, but I'd bet on the other side. The last bit is if we can solve this in a way that aligns the goals with human goals (open question since humans themselves are not really aligned) we could solve most problems we need to solve.

clmul 7y ago
> Therefore we should try and figure out a way to make it safe before AGI exists. Makes no sense to me, how would you ever be able to figure out a way to make something safe before it even exists? Someone who has never built a nuclear reactor most likely could not think of a way to prevent the Chernobyl disaster. (OK, maybe this is a wrong example as someone who did couldn't do this either, but the point should be clear)
- fossuser 7y ago
  I think the argument is that decision theory and goal alignment can be worked on without knowing all the details about how an AGI will work. https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ https://intelligence.org/2016/12/28/ai-alignment-why-its-har...
  - wasoante 7y ago
    ah yes Yudkowsky, the well established AI researcher & definitely not a crank
    - dang 7y ago
      Personal attacks are not ok here, regardless of whom you're attacking. Can you please not post like this to HN? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
pron 7y ago
I think discussions of AI safety at this stage -- when we're already having problems with what passes for AI these days that we're not handling well at all -- is a bit silly, but I don't have something particularly intelligent to say on the matter, and neither, it seems, does anyone else, except maybe for this article that shows that the AGI paranoia (as opposed to the real threats from "AI" we're already facing, like YouTube's recommendation engine) may be a result of a point of view peculiar to Silicon Valley culture: https://www.buzzfeednews.com/article/tedchiang/the-real-danger-to-civilization-isnt-ai-its-runaway https://www.buzzfeednews.com/article/tedchiang/the-real-dang...
- fossuser 7y ago
  I agree with you in a way, if AGI ends up being 300yrs out then work on safety now is likely not that important since whatever technology is developed in that time will probably end up being critical to solving the problem. My main issue personally is that I'm not confident if it's really far out or not and people seem bad at predicting this on both sides. Given that, it probably makes sense to start the work now since goal alignment is a hard problem and it's unknown when it'll become relevant. I read the BuzzFeed article and I think the main issue with it is he assumes that an AGI will be goal aligned by the nature of being an AGI: "In psychology, the term “insight” is used to describe a recognition of one’s own condition, such as when a person with mental illness is aware of their illness. More broadly, it describes the ability to recognize patterns in one’s own behavior. It’s an example of metacognition, or thinking about one’s own thinking, and it’s something most humans are capable of but animals are not. And I believe the best test of whether an AI is really engaging in human-level cognition would be for it to demonstrate insight of this kind." Humans have general preferences and goals built in that have been selected for for thousands of years. An AGI won't have those by default. I think people often think that something intelligent will be like human intelligence, but the entire point of the strawberry example is that an intelligence with different goals that's very good at general problem solving will not have 'insight' that tells it what humans think is good (that's the reason for trying to solve the goal alignment problem - you don't get this for free). He kind of argues for the importance of AGI goal alignment which he calls 'insight', but doesn't realize he's doing so? The comparison to Silicon Valley being blinded by the economies of their own behavior is just weak politics that's missing the point.

clmul 7y ago

> Therefore we should try and figure out a way to make it safe before AGI exists. Makes no sense to me, how would you ever be able to figure out a way to make something safe before it even exists? Someone who has never built a nuclear reactor most likely could not think of a way to prevent the Chernobyl disaster. (OK, maybe this is a wrong example as someone who did couldn't do this either, but the point should be clear)

fossuser 7y ago
I think the argument is that decision theory and goal alignment can be worked on without knowing all the details about how an AGI will work. https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/ https://intelligence.org/2016/12/28/ai-alignment-why-its-har...
- wasoante 7y ago
  ah yes Yudkowsky, the well established AI researcher & definitely not a crank
  - dang 7y ago
    Personal attacks are not ok here, regardless of whom you're attacking. Can you please not post like this to HN? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

pron 7y ago

I think discussions of AI safety at this stage -- when we're already having problems with what passes for AI these days that we're not handling well at all -- is a bit silly, but I don't have something particularly intelligent to say on the matter, and neither, it seems, does anyone else, except maybe for this article that shows that the AGI paranoia (as opposed to the real threats from "AI" we're already facing, like YouTube's recommendation engine) may be a result of a point of view peculiar to Silicon Valley culture: https://www.buzzfeednews.com/article/tedchiang/the-real-danger-to-civilization-isnt-ai-its-runaway https://www.buzzfeednews.com/article/tedchiang/the-real-dang...

fossuser 7y ago
I agree with you in a way, if AGI ends up being 300yrs out then work on safety now is likely not that important since whatever technology is developed in that time will probably end up being critical to solving the problem. My main issue personally is that I'm not confident if it's really far out or not and people seem bad at predicting this on both sides. Given that, it probably makes sense to start the work now since goal alignment is a hard problem and it's unknown when it'll become relevant. I read the BuzzFeed article and I think the main issue with it is he assumes that an AGI will be goal aligned by the nature of being an AGI: "In psychology, the term “insight” is used to describe a recognition of one’s own condition, such as when a person with mental illness is aware of their illness. More broadly, it describes the ability to recognize patterns in one’s own behavior. It’s an example of metacognition, or thinking about one’s own thinking, and it’s something most humans are capable of but animals are not. And I believe the best test of whether an AI is really engaging in human-level cognition would be for it to demonstrate insight of this kind." Humans have general preferences and goals built in that have been selected for for thousands of years. An AGI won't have those by default. I think people often think that something intelligent will be like human intelligence, but the entire point of the strawberry example is that an intelligence with different goals that's very good at general problem solving will not have 'insight' that tells it what humans think is good (that's the reason for trying to solve the goal alignment problem - you don't get this for free). He kind of argues for the importance of AGI goal alignment which he calls 'insight', but doesn't realize he's doing so? The comparison to Silicon Valley being blinded by the economies of their own behavior is just weak politics that's missing the point.