3 ms·

Probably you don't, it's likely an unverifiable proposition. But "better at ethics" is a completely different question than "better at implementing a given ethi

Probably you don't, it's likely an unverifiable proposition. But "better at ethics" is a completely different question than "better at implementing a given ethical stance". The people frightened of intelligence explosion are worried about something like an AI version of existentialism: a mind that accepts some moral system without even trying to justify it, and then optimizes accordingly. It's certainly possible to just accept as axiomatic ethical standards which don't come from any intrinsic feature of the world. I've seen lots of essays (not this one) claim that morality will "inherently" emerge from intelligence, which I think is absurd. Shit, my moral views aren't an 'inherent' product of anything except my evolution-shaped brain that feels empathy. That's where I think "AI won't be like humans so it's fine!" essays screw up so catastrophically; "not like humans" is exactly what people are worried about.

akvadrako 9y ago
> I've seen lots of essays (not this one) claim that morality will "inherently" emerge from intelligence, which I think is absurd. Shit, my moral views aren't an 'inherent' product of anything except my evolution-shaped brain that feels empathy. You're evolution-shaped brain is what an AI will become too. It won't start out super-human smart; it will evolve to an equal level. Then keep going. Morals are basically lower-level approximations which are beneficial for survival and AI's will certainly evolve those too.
- Bartweiss 9y ago
  ...huh? AI will reach parity with humans before it exceeds them, agreed. But intelligence parity is not the same as "matching my evolution-shaped brain". Feeling empathy for other humans is not a requirement for intelligence; I routinely make supposedly-irrational choices to help humans in non-reciprocal settings because I feel empathy. There's no reason to think an AI that can write code as well as me will feel the same. > Morals are basically lower-level approximations which are beneficial for survival and AI's will certainly evolve those too. AI will evolve approximations which are beneficial for survival. We agree there. But why would a bunch of approximations that helped my ancestors survive in low-tech communal environments benefit a strong AI trained with high tech, minimal survival needs, and no "peer group"? I fully expect strong AI will pursue some set of real-world goals which aren't justified except by the anthropic principle. But the assumption that those goals will match human goals seems to completely ignore the fact that the first strong AI will live in a vastly different environment than the first human.
  - Veedrac 9y ago
    > AI will reach parity with humans before it exceeds them, agreed. Intelligence is a hugely dimensional space, and AI constraints look very little like those on humans. It is unlikely there will ever be a time when AI is similarly smart, as much as differently smart to a similar context-dependent magnitude.
    - Bartweiss 9y ago
      A good point, and I should have been clearer about that. I do think it's reasonable to talk about intelligence 'growing', and consequently about one intelligence 'surpassing' another. But AI's methods of thinking certainly won't be human, and it may reach human-parity on different metrics at very different times. Hell, we're seeing some of that already: AI can do I/O and data processing at superhuman speeds, but humans can still extract much more knowledge from a small amount of data.
  - akvadrako 9y ago
    Your morals are not universal, so of course I wouldn't expect an AI to adhere to them. But I also don't adhere to your morals and I'm just another human.
    - Retric 9y ago
      Just another human still imply's you don't relax by selling yourself on fire. Specific instincts are not quite universal, but they very much shape behavior.
    - Bartweiss 9y ago
      Oh ok - if your point is simply that an AI will probably have some set of goals which could be called values, we agree. I was attacking the common claim that superintelligence will inherently develop 'friendly' values. The orthogonality thesis doesn't say AI will lack goals, only that those goals may be totally unrelated to the sort of beliefs you or I would recognize as morals.
      - 9y ago

akvadrako 9y ago

> I've seen lots of essays (not this one) claim that morality will "inherently" emerge from intelligence, which I think is absurd. Shit, my moral views aren't an 'inherent' product of anything except my evolution-shaped brain that feels empathy. You're evolution-shaped brain is what an AI will become too. It won't start out super-human smart; it will evolve to an equal level. Then keep going. Morals are basically lower-level approximations which are beneficial for survival and AI's will certainly evolve those too.

Bartweiss 9y ago
...huh? AI will reach parity with humans before it exceeds them, agreed. But intelligence parity is not the same as "matching my evolution-shaped brain". Feeling empathy for other humans is not a requirement for intelligence; I routinely make supposedly-irrational choices to help humans in non-reciprocal settings because I feel empathy. There's no reason to think an AI that can write code as well as me will feel the same. > Morals are basically lower-level approximations which are beneficial for survival and AI's will certainly evolve those too. AI will evolve approximations which are beneficial for survival. We agree there. But why would a bunch of approximations that helped my ancestors survive in low-tech communal environments benefit a strong AI trained with high tech, minimal survival needs, and no "peer group"? I fully expect strong AI will pursue some set of real-world goals which aren't justified except by the anthropic principle. But the assumption that those goals will match human goals seems to completely ignore the fact that the first strong AI will live in a vastly different environment than the first human.
- Veedrac 9y ago
  > AI will reach parity with humans before it exceeds them, agreed. Intelligence is a hugely dimensional space, and AI constraints look very little like those on humans. It is unlikely there will ever be a time when AI is similarly smart, as much as differently smart to a similar context-dependent magnitude.
  - Bartweiss 9y ago
    A good point, and I should have been clearer about that. I do think it's reasonable to talk about intelligence 'growing', and consequently about one intelligence 'surpassing' another. But AI's methods of thinking certainly won't be human, and it may reach human-parity on different metrics at very different times. Hell, we're seeing some of that already: AI can do I/O and data processing at superhuman speeds, but humans can still extract much more knowledge from a small amount of data.
- akvadrako 9y ago
  Your morals are not universal, so of course I wouldn't expect an AI to adhere to them. But I also don't adhere to your morals and I'm just another human.
  - Retric 9y ago
    Just another human still imply's you don't relax by selling yourself on fire. Specific instincts are not quite universal, but they very much shape behavior.
  - Bartweiss 9y ago
    Oh ok - if your point is simply that an AI will probably have some set of goals which could be called values, we agree. I was attacking the common claim that superintelligence will inherently develop 'friendly' values. The orthogonality thesis doesn't say AI will lack goals, only that those goals may be totally unrelated to the sort of beliefs you or I would recognize as morals.
    - 9y ago