3 ms·

people are all of approximately the same order of magnitude of intelligence. A more relevant comparison would be between a person and a chimp or a crocodile or

people are all of approximately the same order of magnitude of intelligence. A more relevant comparison would be between a person and a chimp or a crocodile or ant. Also compare human civilization before and after writing, language. Also all these "smart people do bad things too" arguments totally miss the point that orthogonality claims they are unrelated. It's not. I claim intelligence is highly correlated to ethics; orthogonality proponents need to prove NO correlation. and in fact mechanistically ethics (segment reality into choices and weighting them) is not even POSSIBLE without predictive capacity (which for some reason counter commentators totally ignore this argument so far). "but I feel like we're identifying the big existential risks early enough to handle them." every year we have some % chance of wiping ourselves out as well as a % chance of e.g. gamma ray burst killing us. It's just a matter of time left to ourselves, not even a question to me, we will kill ourselves, it would take us like 10 thousand years to terraform a planet... do you really think our civilization would last 10 thousand years? that's totally unprecedented in human history...

NumberWangMan 3y ago
Why do you need to prove no correlation? Even if there's a correlation, unless that correlation is extremely strict, there is some risk of a super-intelligence turning nasty. And a strict correlation is simple to disprove with, say, humans and bonobos. Humans fight and hurt each other, while bonobos are basically the "peace and free love" hippies of the primate kingdom. And orthogonality is not about ethics, but about goals. If you can have three very intelligent people, one of which tries to become an industrialist in order to become as wealthy as possible and live out their days in luxury, another of which decides to become a scientist and help humanity as much as possible, and one of which decides to spend their life building model trains, you have proof of orthogonality right there. Unless you get a super-intelligent AI whose goal/behavior is exactly something like "always listen to humans and do what they say, but temper that by not hurting people, and don't try to get too far ahead of them and predict what they would want, and don't try to accumulate too many resources in pursuit of your goal, and also keep in mind that humans may try to invent other AIs may try to gain more resources in pursuit of their goals, and try to stop them if they go wild, but again don't go too far" and so on, for all the things I haven't even thought of but that are are also important, then you may have a really big problem. If we can invent aligned AI, that will help a ton with all the other existential risks. If not, unaligned AI is the existential risk to end all others. Maybe things are changing fast enough now that the variance in our outcomes is so high that we're guaranteed to end no matter what. I hope not.

NumberWangMan 3y ago

Why do you need to prove no correlation? Even if there's a correlation, unless that correlation is extremely strict, there is some risk of a super-intelligence turning nasty. And a strict correlation is simple to disprove with, say, humans and bonobos. Humans fight and hurt each other, while bonobos are basically the "peace and free love" hippies of the primate kingdom. And orthogonality is not about ethics, but about goals. If you can have three very intelligent people, one of which tries to become an industrialist in order to become as wealthy as possible and live out their days in luxury, another of which decides to become a scientist and help humanity as much as possible, and one of which decides to spend their life building model trains, you have proof of orthogonality right there. Unless you get a super-intelligent AI whose goal/behavior is exactly something like "always listen to humans and do what they say, but temper that by not hurting people, and don't try to get too far ahead of them and predict what they would want, and don't try to accumulate too many resources in pursuit of your goal, and also keep in mind that humans may try to invent other AIs may try to gain more resources in pursuit of their goals, and try to stop them if they go wild, but again don't go too far" and so on, for all the things I haven't even thought of but that are are also important, then you may have a really big problem. If we can invent aligned AI, that will help a ton with all the other existential risks. If not, unaligned AI is the existential risk to end all others. Maybe things are changing fast enough now that the variance in our outcomes is so high that we're guaranteed to end no matter what. I hope not.