4 ms·

my pov: orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence, one of if not our greatest sin is the inabilit

my pov: orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence, one of if not our greatest sin is the inability to predict the consequences of our actions "terminal goals" is also probably very wrong the expected value of the singularity is very high. In the grand scheme of things, the chance that humanity will wipe ourselves out before we can realize it is much more important than the chance the singularity will wipe us out. feel free to try and change my mind, because we are very much not aligned.

JohnPrine 3y ago
Can you explain your understanding of the orthogonality thesis? I don't think the ability of intelligent agents to plan conflicts with it
- zzzzzzzza 3y ago
  maybe a way I would phrase it that is more interesting than vanilla phrasing: hand wavy dynamical systems interpretation: attractors in mental space diverge as intelligence increases (or something like that, have no overlap or stable orbit changes randomly) I don't agree with it but something along that line would be the more sophisticated take on it imo. also in order for this to worry you you kind of have to assume some other things, like that those non overlapping orbits will necessarily lead to conflict over resources in the physical world, which I think is also probably wrong in general lol
chaos_emergent 3y ago
predicated on intelligence ~ ethics&planning, I think this is the first argument against AI doomsday that I agree with. Questioning the premise tho - what do you define as intelligence? Machines can outperform humans at specific tasks, yet those same machines don't have a greater degree of ethics, even if constrained to their domain (i.e., a vision network may be able to draw bounding boxes more accurately than a human, but that doesn't say anything about its ability to align with more ethical values). Which makes me believe that your definition of intelligence has nothing to do with superseding humans on cognitive metrics.
- zzzzzzzza 3y ago
  LLM's, the most general form of algorithmic intelligence we have made so far, and the one we are currently actually building/worried about, do better at all tasks as they get bigger. More context -> more learnings to cross apply on different domains. Monotonically? No, but it's a very very strong relationship. (not orthogonal)
dist-epoch 3y ago
There are plenty of examples of very intelligent individuals which used it for evil. For example the leader of the 9/11 hijackers, Mohamed Atta: > “His acquaintances from . . . [Technische Universität Hamburg–Harburg] still cannot reconcile him as a killer, but in hindsight the raw ingredients of his personality suggest some clues. He was meticulous, disciplined and highly intelligent” (Yardley, 2001).
- zzzzzzzza 3y ago
  selection bias. he is famous only because unusually for killers he carried out a slightly mechanicly sophisticated plan. Killers could be much much more effective than they are, if they were smart. They tend not to be. It's a very strong trend. So much so that the few examples like the unabomber are famous. For every unabomber there are tens of thousands of similar people who don't kill people, and even the unabomber had an ethics. (maybe the unabomber is a good warning to alignment people about the dangers of moral concern)
NumberWangMan 3y ago
orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence I'm guessing you're a very nice person. There have been a lot of smart people in history who gained power and did very, very nasty things. If you're nice, being smarter means being better at being nice. If you're not, it means being better at doing whatever not-nice things you want to do. And we're just talking about humans vs humans here. From the point of view of, say, chickens, I don't think they'd rate the smarter people who invented factory farming as nicer than the simple farmers who used to raise 10 birds in a coop. I mean, if you exclude AGI, there are some ways that humans can wipe ourselves out, but I feel like we're identifying the big existential risks early enough to handle them. Intelligence that isn't human is the real danger.
- zzzzzzzza 3y ago
  people are all of approximately the same order of magnitude of intelligence. A more relevant comparison would be between a person and a chimp or a crocodile or ant. Also compare human civilization before and after writing, language. Also all these "smart people do bad things too" arguments totally miss the point that orthogonality claims they are unrelated. It's not. I claim intelligence is highly correlated to ethics; orthogonality proponents need to prove NO correlation. and in fact mechanistically ethics (segment reality into choices and weighting them) is not even POSSIBLE without predictive capacity (which for some reason counter commentators totally ignore this argument so far). "but I feel like we're identifying the big existential risks early enough to handle them." every year we have some % chance of wiping ourselves out as well as a % chance of e.g. gamma ray burst killing us. It's just a matter of time left to ourselves, not even a question to me, we will kill ourselves, it would take us like 10 thousand years to terraform a planet... do you really think our civilization would last 10 thousand years? that's totally unprecedented in human history...
  - NumberWangMan 3y ago
    Why do you need to prove no correlation? Even if there's a correlation, unless that correlation is extremely strict, there is some risk of a super-intelligence turning nasty. And a strict correlation is simple to disprove with, say, humans and bonobos. Humans fight and hurt each other, while bonobos are basically the "peace and free love" hippies of the primate kingdom. And orthogonality is not about ethics, but about goals. If you can have three very intelligent people, one of which tries to become an industrialist in order to become as wealthy as possible and live out their days in luxury, another of which decides to become a scientist and help humanity as much as possible, and one of which decides to spend their life building model trains, you have proof of orthogonality right there. Unless you get a super-intelligent AI whose goal/behavior is exactly something like "always listen to humans and do what they say, but temper that by not hurting people, and don't try to get too far ahead of them and predict what they would want, and don't try to accumulate too many resources in pursuit of your goal, and also keep in mind that humans may try to invent other AIs may try to gain more resources in pursuit of their goals, and try to stop them if they go wild, but again don't go too far" and so on, for all the things I haven't even thought of but that are are also important, then you may have a really big problem. If we can invent aligned AI, that will help a ton with all the other existential risks. If not, unaligned AI is the existential risk to end all others. Maybe things are changing fast enough now that the variance in our outcomes is so high that we're guaranteed to end no matter what. I hope not.
nullsense 3y ago
I've seen this argument before and found it wanting. Hitler didn't ultimately succeed, but he sure as heck got pretty far with his plans. So, if there are some entities in the AGI population that eventually get the idea to try it, that's likely all that matters, and not what any average of the population is. An NGI started WW2, so why wouldn't an AGI start WW3? "Demonstrably unfriendly natural intelligence seeks to build provably friendly artificial intelligence"
- zzzzzzzza 3y ago
  personally I think ww2 is a meme, one datapoint out of the entirety of natural history, but funnily enough scott aaranson used it to justify his anti orthognality stance recently (axis scientists were mid tier compared to allies scientists partially because the allies were perceived as better morally aligned with the scientific community) one point he made aside from that though, is that if you truly believe in orthogonality you should not value education or learning except to educate people who agree/are aligned with you. I think he is wrong only in that orthogonality is even more obviously wrong than that.
  - nullsense 3y ago
    It's not really a meme when you read more of history. It becomes much more of a theme than a meme.
  - NumberWangMan 3y ago
    A list of genocidal dictators: https://www.scaruffi.com/politics/dictat.html https://www.scaruffi.com/politics/dictat.html Anyone who comes into power enough to do something like this has a high amount of intelligence -- not necessarily book smarts, but the ability to persuade or manipulate other people.

JohnPrine 3y ago

Can you explain your understanding of the orthogonality thesis? I don't think the ability of intelligent agents to plan conflicts with it

zzzzzzzza 3y ago
maybe a way I would phrase it that is more interesting than vanilla phrasing: hand wavy dynamical systems interpretation: attractors in mental space diverge as intelligence increases (or something like that, have no overlap or stable orbit changes randomly) I don't agree with it but something along that line would be the more sophisticated take on it imo. also in order for this to worry you you kind of have to assume some other things, like that those non overlapping orbits will necessarily lead to conflict over resources in the physical world, which I think is also probably wrong in general lol

chaos_emergent 3y ago

predicated on intelligence ~ ethics&planning, I think this is the first argument against AI doomsday that I agree with. Questioning the premise tho - what do you define as intelligence? Machines can outperform humans at specific tasks, yet those same machines don't have a greater degree of ethics, even if constrained to their domain (i.e., a vision network may be able to draw bounding boxes more accurately than a human, but that doesn't say anything about its ability to align with more ethical values). Which makes me believe that your definition of intelligence has nothing to do with superseding humans on cognitive metrics.

zzzzzzzza 3y ago
LLM's, the most general form of algorithmic intelligence we have made so far, and the one we are currently actually building/worried about, do better at all tasks as they get bigger. More context -> more learnings to cross apply on different domains. Monotonically? No, but it's a very very strong relationship. (not orthogonal)

dist-epoch 3y ago

There are plenty of examples of very intelligent individuals which used it for evil. For example the leader of the 9/11 hijackers, Mohamed Atta: > “His acquaintances from . . . [Technische Universität Hamburg–Harburg] still cannot reconcile him as a killer, but in hindsight the raw ingredients of his personality suggest some clues. He was meticulous, disciplined and highly intelligent” (Yardley, 2001).

zzzzzzzza 3y ago
selection bias. he is famous only because unusually for killers he carried out a slightly mechanicly sophisticated plan. Killers could be much much more effective than they are, if they were smart. They tend not to be. It's a very strong trend. So much so that the few examples like the unabomber are famous. For every unabomber there are tens of thousands of similar people who don't kill people, and even the unabomber had an ethics. (maybe the unabomber is a good warning to alignment people about the dangers of moral concern)

NumberWangMan 3y ago

orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence I'm guessing you're a very nice person. There have been a lot of smart people in history who gained power and did very, very nasty things. If you're nice, being smarter means being better at being nice. If you're not, it means being better at doing whatever not-nice things you want to do. And we're just talking about humans vs humans here. From the point of view of, say, chickens, I don't think they'd rate the smarter people who invented factory farming as nicer than the simple farmers who used to raise 10 birds in a coop. I mean, if you exclude AGI, there are some ways that humans can wipe ourselves out, but I feel like we're identifying the big existential risks early enough to handle them. Intelligence that isn't human is the real danger.

zzzzzzzza 3y ago
people are all of approximately the same order of magnitude of intelligence. A more relevant comparison would be between a person and a chimp or a crocodile or ant. Also compare human civilization before and after writing, language. Also all these "smart people do bad things too" arguments totally miss the point that orthogonality claims they are unrelated. It's not. I claim intelligence is highly correlated to ethics; orthogonality proponents need to prove NO correlation. and in fact mechanistically ethics (segment reality into choices and weighting them) is not even POSSIBLE without predictive capacity (which for some reason counter commentators totally ignore this argument so far). "but I feel like we're identifying the big existential risks early enough to handle them." every year we have some % chance of wiping ourselves out as well as a % chance of e.g. gamma ray burst killing us. It's just a matter of time left to ourselves, not even a question to me, we will kill ourselves, it would take us like 10 thousand years to terraform a planet... do you really think our civilization would last 10 thousand years? that's totally unprecedented in human history...
- NumberWangMan 3y ago
  Why do you need to prove no correlation? Even if there's a correlation, unless that correlation is extremely strict, there is some risk of a super-intelligence turning nasty. And a strict correlation is simple to disprove with, say, humans and bonobos. Humans fight and hurt each other, while bonobos are basically the "peace and free love" hippies of the primate kingdom. And orthogonality is not about ethics, but about goals. If you can have three very intelligent people, one of which tries to become an industrialist in order to become as wealthy as possible and live out their days in luxury, another of which decides to become a scientist and help humanity as much as possible, and one of which decides to spend their life building model trains, you have proof of orthogonality right there. Unless you get a super-intelligent AI whose goal/behavior is exactly something like "always listen to humans and do what they say, but temper that by not hurting people, and don't try to get too far ahead of them and predict what they would want, and don't try to accumulate too many resources in pursuit of your goal, and also keep in mind that humans may try to invent other AIs may try to gain more resources in pursuit of their goals, and try to stop them if they go wild, but again don't go too far" and so on, for all the things I haven't even thought of but that are are also important, then you may have a really big problem. If we can invent aligned AI, that will help a ton with all the other existential risks. If not, unaligned AI is the existential risk to end all others. Maybe things are changing fast enough now that the variance in our outcomes is so high that we're guaranteed to end no matter what. I hope not.

nullsense 3y ago

I've seen this argument before and found it wanting. Hitler didn't ultimately succeed, but he sure as heck got pretty far with his plans. So, if there are some entities in the AGI population that eventually get the idea to try it, that's likely all that matters, and not what any average of the population is. An NGI started WW2, so why wouldn't an AGI start WW3? "Demonstrably unfriendly natural intelligence seeks to build provably friendly artificial intelligence"

zzzzzzzza 3y ago
personally I think ww2 is a meme, one datapoint out of the entirety of natural history, but funnily enough scott aaranson used it to justify his anti orthognality stance recently (axis scientists were mid tier compared to allies scientists partially because the allies were perceived as better morally aligned with the scientific community) one point he made aside from that though, is that if you truly believe in orthogonality you should not value education or learning except to educate people who agree/are aligned with you. I think he is wrong only in that orthogonality is even more obviously wrong than that.
- nullsense 3y ago
  It's not really a meme when you read more of history. It becomes much more of a theme than a meme.
- NumberWangMan 3y ago
  A list of genocidal dictators: https://www.scaruffi.com/politics/dictat.html https://www.scaruffi.com/politics/dictat.html Anyone who comes into power enough to do something like this has a high amount of intelligence -- not necessarily book smarts, but the ability to persuade or manipulate other people.