14 ms·

It’d be a longer conversation that’s hard to do via HN comments, but I think the main divide is I get the impression you’re giving the AGI implicit human-like r

It’d be a longer conversation that’s hard to do via HN comments, but I think the main divide is I get the impression you’re giving the AGI implicit human-like reasoning, but the idea behind the orthogonality thesis or alignment generally is that you don’t get these things for free. It’s not that humans hate ants or apes, it’s that we pursue goals without thinking too hard about them. A house being built may destroy an ant hill but it’s not because we hate ants. The core argument is it’s not only possible to have an intelligence that’s a lot more capable than us but with dumb goals because of our failure to align it, but that that’s the default outcome. There is no “reasoning with it” because it’s not a human like intelligence, it has a goal it’s focused on (paperclips) and if it’s a lot smarter than us then that’s game over.

gooseus 3y ago
Yes, I do assume that any AGI that is "a lot smarter than us" such that it is "game over" will have to possess human-like reasoning that would also allow it to adjust its own goals, otherwise it's going to be restricted in a way that makes it less capable than us. It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it; but also so narrow and limited in how its goals are defined that it is incapable of adjusting its own course of action. I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over). I find a lot of these AI concepts lack a rigor that is commonplace in normal computer science... for instance, you can show that one problem can be reduced to another problem which has been proven to have certain limits, therefore the first problem can not break those limits without breaking the proof [comparison sorting will always be O(nlogn)]. When it comes to the "alignment problem" as it pertains to AI that have human-level capabilities, it seems to me that you have a similar situation... doesn't this problem just reduce to the same ethical, moral, and philosophical issues we have in aligning human intelligence with some cultural or civic ideal for behavior? Isn't the "alignment problem" the same problem as "how do you raise a good citizen"?
- homefree 3y ago
  There's some overlap with 'how you raise a good citizen', but we're also aligned somewhat already by our shared evolutionary history (and even then there are still major problems so if anything that suggests a lot of caution). > "I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over)." The specific bit of this was that paperclips just happen to satisfy its reward function really well (vs. it being narrowly constrained to make paperclips intentionally). The example is meant to be about how you can get an unanticipated result when you don't know what the intelligence is solving for. A human looks at that and thinks 'that's a dumb goal', but the point of the alignment problem is that that isn't some universal truth, a different intelligence would not get that shared understanding for free. Humans have a lot of baked in baseline wants and even then like you said we're ourselves not perfectly aligned. > "It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it" Imagine a chimp or ant trying to outsmart or out play a human - and this difference is smaller than the difference we're talking about here. To simplify it further imagine a regular human brain unconstrained by biological energy limitations scaled up to run a billion times faster. It thinks faster than you - there isn't a competition there you're essentially standing still.

gooseus 3y ago

Yes, I do assume that any AGI that is "a lot smarter than us" such that it is "game over" will have to possess human-like reasoning that would also allow it to adjust its own goals, otherwise it's going to be restricted in a way that makes it less capable than us. It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it; but also so narrow and limited in how its goals are defined that it is incapable of adjusting its own course of action. I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over). I find a lot of these AI concepts lack a rigor that is commonplace in normal computer science... for instance, you can show that one problem can be reduced to another problem which has been proven to have certain limits, therefore the first problem can not break those limits without breaking the proof [comparison sorting will always be O(nlogn)]. When it comes to the "alignment problem" as it pertains to AI that have human-level capabilities, it seems to me that you have a similar situation... doesn't this problem just reduce to the same ethical, moral, and philosophical issues we have in aligning human intelligence with some cultural or civic ideal for behavior? Isn't the "alignment problem" the same problem as "how do you raise a good citizen"?

homefree 3y ago
There's some overlap with 'how you raise a good citizen', but we're also aligned somewhat already by our shared evolutionary history (and even then there are still major problems so if anything that suggests a lot of caution). > "I'll be honest, I don't understand how anyone who builds real machines in real life can seriously consider a machine built with such a narrowly defined goal (produce paper-clips) that also possesses the kind of capabilities you're imagining as a side effect (a lot smarter than us, game over)." The specific bit of this was that paperclips just happen to satisfy its reward function really well (vs. it being narrowly constrained to make paperclips intentionally). The example is meant to be about how you can get an unanticipated result when you don't know what the intelligence is solving for. A human looks at that and thinks 'that's a dumb goal', but the point of the alignment problem is that that isn't some universal truth, a different intelligence would not get that shared understanding for free. Humans have a lot of baked in baseline wants and even then like you said we're ourselves not perfectly aligned. > "It seems to me that you want to have it both ways... a machine that is so smart and strategic that there is no way any human intelligence could ever outsmart or outplay it" Imagine a chimp or ant trying to outsmart or out play a human - and this difference is smaller than the difference we're talking about here. To simplify it further imagine a regular human brain unconstrained by biological energy limitations scaled up to run a billion times faster. It thinks faster than you - there isn't a competition there you're essentially standing still.