3 ms·

This is not something programmers and other wild HN dwellers are accustomed to hearing, but there is a very strong case to suggest that intelligence, the univer

by xzvf 6y ago

This is not something programmers and other wild HN dwellers are accustomed to hearing, but there is a very strong case to suggest that intelligence, the universal kind, is not possible without empathy. Worry about alignment in machines, not in human-likes.

_greim_ 6y ago
This is why people talk about optimization instead of intelligence, since you side-step the baggage that comes with the word "intelligence". E.g. an optimizer doesn't need to be universal to be a problem, whether it's optimizing for social media addictiveness or paperclip manufacturing.
- xzvf 6y ago
  But optimization is decidedly not intelligence. We’ve known this for decades and have clear proofs this is the case. This is just a collective dance of burying our heads in the sand. I’ll quote this guy called Turing: “If a machine is expected to be infallible, it cannot also be intelligent”.
  - DougMerritt 6y ago
    Optimization is not usually a synonym for intelligence, despite some individual's beliefs, but it can be an effective substitute at times -- chess programs, for instance, play world class chess via optimization rather than via intelligence-as-seen-in-humans.
  - _greim_ 6y ago
    > But optimization is decidedly not intelligence. I don't think anyone is making that claim. That's why the distinction is useful.
yaacov 6y ago
I think you’re using the word “intelligence” to mean something entirely different from what the AI alignment crowd is worried about.
fossuser 6y ago
I think this is objectively wrong even on a human level. I could see some part of it if you're using empathy in a general enough sense to only mean modeling other minds without caring about their goals except as a way to pursue your own (which doesn't sound like what you're saying). It sounds more like you're putting intelligence in some reference class where you're just stating it's not intelligence until it's already aligned with humans (which is not helpful). For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things. This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff. Some other relevant essays: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe... This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8
- xzvf 6y ago
  When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.
  - fossuser 6y ago
    You do align people all the time - what is culture, story telling, arguing over what's good and bad, law? Discussion and changing people's minds? Persuasion and sharing ideas? > "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'" They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.
roywiggins 6y ago
If that were true, sociopaths would be dumb as rocks. But, some sociopaths are actually pretty smart and don't demonstrate the sort of empathy you'd want in an AGI.
luma 6y ago
If there's a strong case, maybe you could put it forward for us? You've made a heck of an assertion there...

by xzvf 6y ago

_greim_ 6y ago

This is why people talk about optimization instead of intelligence, since you side-step the baggage that comes with the word "intelligence". E.g. an optimizer doesn't need to be universal to be a problem, whether it's optimizing for social media addictiveness or paperclip manufacturing.

xzvf 6y ago
But optimization is decidedly not intelligence. We’ve known this for decades and have clear proofs this is the case. This is just a collective dance of burying our heads in the sand. I’ll quote this guy called Turing: “If a machine is expected to be infallible, it cannot also be intelligent”.
- DougMerritt 6y ago
  Optimization is not usually a synonym for intelligence, despite some individual's beliefs, but it can be an effective substitute at times -- chess programs, for instance, play world class chess via optimization rather than via intelligence-as-seen-in-humans.
- _greim_ 6y ago
  > But optimization is decidedly not intelligence. I don't think anyone is making that claim. That's why the distinction is useful.

yaacov 6y ago

I think you’re using the word “intelligence” to mean something entirely different from what the AI alignment crowd is worried about.

fossuser 6y ago

I think this is objectively wrong even on a human level. I could see some part of it if you're using empathy in a general enough sense to only mean modeling other minds without caring about their goals except as a way to pursue your own (which doesn't sound like what you're saying). It sounds more like you're putting intelligence in some reference class where you're just stating it's not intelligence until it's already aligned with humans (which is not helpful). For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things. This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff. Some other relevant essays: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe... This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8

xzvf 6y ago
When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.
- fossuser 6y ago
  You do align people all the time - what is culture, story telling, arguing over what's good and bad, law? Discussion and changing people's minds? Persuasion and sharing ideas? > "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'" They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.

roywiggins 6y ago

If that were true, sociopaths would be dumb as rocks. But, some sociopaths are actually pretty smart and don't demonstrate the sort of empathy you'd want in an AGI.

luma 6y ago

If there's a strong case, maybe you could put it forward for us? You've made a heck of an assertion there...