5 ms·

AI safety seems like one of those topics that trips up even smart people such as Hawkins, because it intuitively seems like it has very obvious solutions. And f

AI safety seems like one of those topics that trips up even smart people such as Hawkins, because it intuitively seems like it has very obvious solutions. And for each objection raised there's usually a "so you just..." continuation, and an objection to that, until you're talking about things like instrumental convergence and you're in the territory of trying to reason in quite a complex way about the behaviour of systems that don't exist yet, and so the temptation is to dismiss it all as theoretical hand-wringing. Personally, while I'm not on board with the board with the direst predictions of the super-intelligence pessimist crowd, I have become more and more convinced that goal misalignment is going to be a significant problem, and that while it might not doom the species, it's something that all AI researchers like Hawkins need to start paying close attention to now.

xzvf 6y ago
This is not something programmers and other wild HN dwellers are accustomed to hearing, but there is a very strong case to suggest that intelligence, the universal kind, is not possible without empathy. Worry about alignment in machines, not in human-likes.
- _greim_ 6y ago
  This is why people talk about optimization instead of intelligence, since you side-step the baggage that comes with the word "intelligence". E.g. an optimizer doesn't need to be universal to be a problem, whether it's optimizing for social media addictiveness or paperclip manufacturing.
  - xzvf 6y ago
    But optimization is decidedly not intelligence. We’ve known this for decades and have clear proofs this is the case. This is just a collective dance of burying our heads in the sand. I’ll quote this guy called Turing: “If a machine is expected to be infallible, it cannot also be intelligent”.
    - DougMerritt 6y ago
      Optimization is not usually a synonym for intelligence, despite some individual's beliefs, but it can be an effective substitute at times -- chess programs, for instance, play world class chess via optimization rather than via intelligence-as-seen-in-humans.
    - _greim_ 6y ago
      > But optimization is decidedly not intelligence. I don't think anyone is making that claim. That's why the distinction is useful.
- yaacov 6y ago
  I think you’re using the word “intelligence” to mean something entirely different from what the AI alignment crowd is worried about.
- fossuser 6y ago
  I think this is objectively wrong even on a human level. I could see some part of it if you're using empathy in a general enough sense to only mean modeling other minds without caring about their goals except as a way to pursue your own (which doesn't sound like what you're saying). It sounds more like you're putting intelligence in some reference class where you're just stating it's not intelligence until it's already aligned with humans (which is not helpful). For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things. This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff. Some other relevant essays: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe... This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8
  - xzvf 6y ago
    When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.
    - fossuser 6y ago
      You do align people all the time - what is culture, story telling, arguing over what's good and bad, law? Discussion and changing people's minds? Persuasion and sharing ideas? > "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'" They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.
- roywiggins 6y ago
  If that were true, sociopaths would be dumb as rocks. But, some sociopaths are actually pretty smart and don't demonstrate the sort of empathy you'd want in an AGI.
- luma 6y ago
  If there's a strong case, maybe you could put it forward for us? You've made a heck of an assertion there...
fighterpilot 6y ago
I also think people get too caught up on the expected time-frame. The large majority of active AI researchers think that AGI will happen at some point in the (sub-1000 year) future. When exactly isn't a very interesting question, relatively speaking. We're going to have to deal with AGI eventually, and whether it's going to do what we want is not something that can be theoretically predicted from the armchair.
- fossuser 6y ago
  Yeah - and people are famously bad about predicting these events: https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no... If it is something that's a hundred plus years out then we'll probably need whatever tech develops in the mean time to help, but since it's hard to know that seems reasonable for people to be working on it now? It's also possible to figure things out before the necessary tech is possible (I think a lot of CS papers in the 60s became more interesting later when the hardware caught up to be more useful, arguably the recent NN stuff falls into this category too).
  - walleeee 6y ago
    > If it is something that's a hundred plus years out then we'll probably need whatever tech develops in the mean time to help Circular justification of technological development is the reason unfriendly AGI is a threat in the first place (and also the reason we are unlikely to see it realized, imo; the internal combustion engine for instance poses existential risks not only to people but to the possibility of machine intelligence). Technology is not a monolith, forms can and do preclude other forms
  - ketzo 6y ago
    Just wanna say I absolutely loved reading that blog post, thanks for the link.
    - fighterpilot 5y ago
      Check out the blog post in the comment that's parallel to /u/fossuser above. It's also great.
MR4D 6y ago
> AI safety.... My personal thought is that since humanity currently can't manage the I(non-A) safety, that we'll fumble through this as well. As long as they can't replicate, we'll probably be ok, but once that changes, we're probably toast.
- gnzoidberg 6y ago
  Thus, eventually we'll be toast.
ilaksh 6y ago
To me it's obvious that if we create something that is really like a digital animal or digital person then we will lose control pretty soon. Because animals are intended to be adaptive in the extreme and 100% autonomous, survive and reproduce. But I still think we can create fairly general purpose systems without those animal-like characteristics such as full autonomy.
- jamesrcole 6y ago
  What if they’re sandboxed in some fashion? There’s zoos and jails and they’re reasonably effective at “keeping them in”.
  - emiliobumachar 6y ago
    That's been conjectured under the name "AI Box". https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box
tshaddox 6y ago
Do you anticipate goal misalignment being a more significant problem for AIs than it already is for humans? If so, why? And either way, why would we need to approach goal alignment differently than we do with humans?
- jimbokun 6y ago
  It's more significant for AIs because we expect them to become super human, thus with potentially unlimited potential for disaster. I would say we have been dealing with goal alignment problems with humans for most of human history.
  - tshaddox 6y ago
    Why would we expect them to become super human? I would expect AIs to be able to use the latest technology and weapons, and also to develop new and better technology and weapons, and to exclude other intelligences from using said technology and weapons, but note that this is already true for humans.
    - pmichaud 6y ago
      The basic answer is that unlike humans, AIs will be able to recursively self improve (in principle).
      - tshaddox 6y ago
        I don’t really buy that. Humans also improve their own abilities using technology, and I don’t see any reason to expect that technological advancements made by AIs won’t be available to humans as well. Yes, an AI group that is hostile to a human group may want to develop technology and keep it to themselves, but again, that’s already the case with different human groups (and tends to apply most prominently to our most destructive technologies).
        TeMPOraL 5y ago
michaelbarton 6y ago
I have thought that we might have a good a grasp on an existential problem slime AI safety, as earlier hominids did on the problem of preventing a fitter out-competing species dominating the landscape. It might have a couple of things in common: a vague sense of danger without a specific grasp on containing it using out rudimentary tools. And once the cat is out of the bag it's near impossible to get back in.
throwaway81523 5y ago
Hawkins worries about AI accidentally going nuts and destroying stuff. He ignores that it's an existential threat for the same reason nuclear weapons are: not because of potential accidents, but because of potential on-purposes. Do you really want Bezos / Musk / Biden / Putin / your favorite baddie being the first to get control of these things? It doesn't seem wise.

xzvf 6y ago

This is not something programmers and other wild HN dwellers are accustomed to hearing, but there is a very strong case to suggest that intelligence, the universal kind, is not possible without empathy. Worry about alignment in machines, not in human-likes.

_greim_ 6y ago
This is why people talk about optimization instead of intelligence, since you side-step the baggage that comes with the word "intelligence". E.g. an optimizer doesn't need to be universal to be a problem, whether it's optimizing for social media addictiveness or paperclip manufacturing.
- xzvf 6y ago
  But optimization is decidedly not intelligence. We’ve known this for decades and have clear proofs this is the case. This is just a collective dance of burying our heads in the sand. I’ll quote this guy called Turing: “If a machine is expected to be infallible, it cannot also be intelligent”.
  - DougMerritt 6y ago
    Optimization is not usually a synonym for intelligence, despite some individual's beliefs, but it can be an effective substitute at times -- chess programs, for instance, play world class chess via optimization rather than via intelligence-as-seen-in-humans.
  - _greim_ 6y ago
    > But optimization is decidedly not intelligence. I don't think anyone is making that claim. That's why the distinction is useful.
yaacov 6y ago
I think you’re using the word “intelligence” to mean something entirely different from what the AI alignment crowd is worried about.
fossuser 6y ago
I think this is objectively wrong even on a human level. I could see some part of it if you're using empathy in a general enough sense to only mean modeling other minds without caring about their goals except as a way to pursue your own (which doesn't sound like what you're saying). It sounds more like you're putting intelligence in some reference class where you're just stating it's not intelligence until it's already aligned with humans (which is not helpful). For some reason people tend to think that general intelligence would generate all these other positive human-like qualities, but a lot of those are not super well aligned even in humans and they are tied to our multi-billion year evolutionary history selecting for certain things. This is basically the orthogonality thesis which I found pretty compelling: https://www.lesswrong.com/tag/orthogonality-thesis https://www.lesswrong.com/tag/orthogonality-thesis - the AGI crowd has a lot of really good writing on this stuff and they've thought a lot about it. If it's something you're curious about it's worth reading the current stuff. Some other relevant essays: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe... This talk is also a decent introduction: https://www.youtube.com/watch?v=EUjc1WuyPT8 https://www.youtube.com/watch?v=EUjc1WuyPT8
- xzvf 6y ago
  When we get intelligent machines they will become beings. Contrary to SV corporate-speak, you don’t align beings. You don’t align your neighbors, you don’t align people from other countries. You can’t hardwire them to follow certain principles, that’s the whole point of beings, otherwise you’re back at automated machines. All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?' and that is not a very interesting question to me.
  - fossuser 6y ago
    You do align people all the time - what is culture, story telling, arguing over what's good and bad, law? Discussion and changing people's minds? Persuasion and sharing ideas? > "All of these lesswrong posts sound so technical and philosophical and so on but in the end they all really ask 'How would you control superheroes and supervillains?'" They explicitly don't ask that because if you get to that point and you don't have them aligned with human goals, you're fucked. The purpose is to understand how to align an AGI's goals to humanity before they reach that level.
roywiggins 6y ago
If that were true, sociopaths would be dumb as rocks. But, some sociopaths are actually pretty smart and don't demonstrate the sort of empathy you'd want in an AGI.
luma 6y ago
If there's a strong case, maybe you could put it forward for us? You've made a heck of an assertion there...

fighterpilot 6y ago

I also think people get too caught up on the expected time-frame. The large majority of active AI researchers think that AGI will happen at some point in the (sub-1000 year) future. When exactly isn't a very interesting question, relatively speaking. We're going to have to deal with AGI eventually, and whether it's going to do what we want is not something that can be theoretically predicted from the armchair.

fossuser 6y ago
Yeah - and people are famously bad about predicting these events: https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no... If it is something that's a hundred plus years out then we'll probably need whatever tech develops in the mean time to help, but since it's hard to know that seems reasonable for people to be working on it now? It's also possible to figure things out before the necessary tech is possible (I think a lot of CS papers in the 60s became more interesting later when the hardware caught up to be more useful, arguably the recent NN stuff falls into this category too).
- walleeee 6y ago
  > If it is something that's a hundred plus years out then we'll probably need whatever tech develops in the mean time to help Circular justification of technological development is the reason unfriendly AGI is a threat in the first place (and also the reason we are unlikely to see it realized, imo; the internal combustion engine for instance poses existential risks not only to people but to the possibility of machine intelligence). Technology is not a monolith, forms can and do preclude other forms
- ketzo 6y ago
  Just wanna say I absolutely loved reading that blog post, thanks for the link.
  - fighterpilot 5y ago
    Check out the blog post in the comment that's parallel to /u/fossuser above. It's also great.

MR4D 6y ago

> AI safety.... My personal thought is that since humanity currently can't manage the I(non-A) safety, that we'll fumble through this as well. As long as they can't replicate, we'll probably be ok, but once that changes, we're probably toast.

gnzoidberg 6y ago
Thus, eventually we'll be toast.

ilaksh 6y ago

To me it's obvious that if we create something that is really like a digital animal or digital person then we will lose control pretty soon. Because animals are intended to be adaptive in the extreme and 100% autonomous, survive and reproduce. But I still think we can create fairly general purpose systems without those animal-like characteristics such as full autonomy.

jamesrcole 6y ago
What if they’re sandboxed in some fashion? There’s zoos and jails and they’re reasonably effective at “keeping them in”.
- emiliobumachar 6y ago
  That's been conjectured under the name "AI Box". https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box

tshaddox 6y ago

Do you anticipate goal misalignment being a more significant problem for AIs than it already is for humans? If so, why? And either way, why would we need to approach goal alignment differently than we do with humans?

jimbokun 6y ago
It's more significant for AIs because we expect them to become super human, thus with potentially unlimited potential for disaster. I would say we have been dealing with goal alignment problems with humans for most of human history.
- tshaddox 6y ago
  Why would we expect them to become super human? I would expect AIs to be able to use the latest technology and weapons, and also to develop new and better technology and weapons, and to exclude other intelligences from using said technology and weapons, but note that this is already true for humans.
  - pmichaud 6y ago
    The basic answer is that unlike humans, AIs will be able to recursively self improve (in principle).
    - tshaddox 6y ago
      I don’t really buy that. Humans also improve their own abilities using technology, and I don’t see any reason to expect that technological advancements made by AIs won’t be available to humans as well. Yes, an AI group that is hostile to a human group may want to develop technology and keep it to themselves, but again, that’s already the case with different human groups (and tends to apply most prominently to our most destructive technologies).
      - TeMPOraL 5y ago

michaelbarton 6y ago

I have thought that we might have a good a grasp on an existential problem slime AI safety, as earlier hominids did on the problem of preventing a fitter out-competing species dominating the landscape. It might have a couple of things in common: a vague sense of danger without a specific grasp on containing it using out rudimentary tools. And once the cat is out of the bag it's near impossible to get back in.

throwaway81523 5y ago

Hawkins worries about AI accidentally going nuts and destroying stuff. He ignores that it's an existential threat for the same reason nuclear weapons are: not because of potential accidents, but because of potential on-purposes. Do you really want Bezos / Musk / Biden / Putin / your favorite baddie being the first to get control of these things? It doesn't seem wise.