4 ms·

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even kn

by nmca 6y ago

I haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “harm humans” or “display superintelligence”) of a Turing machine is undecidable" One man's modus ponens is another man's modus tollens. If their theory says that superintelligence is not recognisable, then they're perhaps not using a good definition of superintelligence, because obviously we will be able to recognise it.

hinkley 6y ago
Look at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.
- nmca 6y ago
  This is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.
  - thatguy0900 6y ago
    What if it was intelligent enough to continuously improve itself?
  - cma 6y ago
    > It will be smart, but contain no infinities. We think we can say that from physics, but we don't know what we don't know.
    - goatlover 6y ago
      Computational limits are already known from physics. But before any device can get close to those limits, there are likely practical ones in terms of cost and energy to construct such devices.
yongjik 6y ago
Heh, that quote reeks of postmodernism - reminds me of the whole Sokal affair. Good old days...
tialaramex 6y ago
Check out the Stanisław Lem story "GOLEM XIV". GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie). Lots of supposedly smart humans try to debate with Golem and eventually they (humans supervising the interaction) have to impose a "rule" to stop people opening their mouths the very first time they see Golem and getting humiliated almost before they've understood what is happening, because it's frustrating for everybody else. Golem is asked if humans could acquire such intelligence and it explains that this is categorically impossible, Golem is doing something that is not just a better way to do the same thing as humans, it's doing something altogether different and superior that humans can't do. It also seems to hint that Annie is, in turn, superior in capability to Golem and that for them such transcendence to further feats is not necessarily impossible. This is one of the stories that Lem wrote by an oblique method, what we have is extracts from an introduction to an imaginary dry scientific record that details the period between GOLEM being constructed and... the eventual conclusion of the incident. Anyway, I was reminded because while Lem has to be careful (he's not superintelligent after all) he's clearly hinting that humans aren't smart enough to recognise the superintelligence of GOLEM and ANNIE. One proposed reason for why ANNIE rather than GOLEM is responsible for the events described near the end of the story is that she doesn't even think about humans, for the same reason humans largely don't think about flies. What's to think about? They're just an annoyance, to be swatted aside.
- Causality1 6y ago
  That relies on the existence of systems too complex for humans to understand. We may eventually build machines that can in turn build other incomprehensible machines, but we haven't passed that point yet. Superintelligence can't surprise us until some point after computers become self-improving.
  - harmoat 6y ago
    Aren't large neural network already black boxes we don't understand built by machines we understand?
qwantim1 6y ago
> obviously we will be able to recognise it. There is no basis for that assumption, as you’ve never encountered a superintelligence knowingly.
masswerk 6y ago
> because obviously we will be able to recognise it Mind that even the oldest definitions of intelligence include the ability to assume a position, which is not necessarily held by the subject. This includes all kinds of imitation and mimicry, i.e., simulating a state. Assuming that super intelligence is an extended state of "normal" intelligence, but excelling in capabilities, we must also assume assume that we would be only able to recognize it, if the superintelligence wants to be recognized or allows for it in any other way (e.g., doesn't care about being recognized, etc). Now we could obviously take simulation out of the equation (or definition), but then, what are we talking about?

by nmca 6y ago

hinkley 6y ago

Look at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.

nmca 6y ago
This is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.
- thatguy0900 6y ago
  What if it was intelligent enough to continuously improve itself?
- cma 6y ago
  > It will be smart, but contain no infinities. We think we can say that from physics, but we don't know what we don't know.
  - goatlover 6y ago
    Computational limits are already known from physics. But before any device can get close to those limits, there are likely practical ones in terms of cost and energy to construct such devices.

yongjik 6y ago

Heh, that quote reeks of postmodernism - reminds me of the whole Sokal affair. Good old days...

tialaramex 6y ago

Check out the Stanisław Lem story "GOLEM XIV". GOLEM is one of a series of machines constructed to plan World War III, as is its sister HONEST ANNIE. But to the frustration of their human creators these more sophisticated machines refuse to plan World War III and instead seem to become philosophers (Golem) or just refuse to communicate with humans at all (Annie). Lots of supposedly smart humans try to debate with Golem and eventually they (humans supervising the interaction) have to impose a "rule" to stop people opening their mouths the very first time they see Golem and getting humiliated almost before they've understood what is happening, because it's frustrating for everybody else. Golem is asked if humans could acquire such intelligence and it explains that this is categorically impossible, Golem is doing something that is not just a better way to do the same thing as humans, it's doing something altogether different and superior that humans can't do. It also seems to hint that Annie is, in turn, superior in capability to Golem and that for them such transcendence to further feats is not necessarily impossible. This is one of the stories that Lem wrote by an oblique method, what we have is extracts from an introduction to an imaginary dry scientific record that details the period between GOLEM being constructed and... the eventual conclusion of the incident. Anyway, I was reminded because while Lem has to be careful (he's not superintelligent after all) he's clearly hinting that humans aren't smart enough to recognise the superintelligence of GOLEM and ANNIE. One proposed reason for why ANNIE rather than GOLEM is responsible for the events described near the end of the story is that she doesn't even think about humans, for the same reason humans largely don't think about flies. What's to think about? They're just an annoyance, to be swatted aside.

Causality1 6y ago
That relies on the existence of systems too complex for humans to understand. We may eventually build machines that can in turn build other incomprehensible machines, but we haven't passed that point yet. Superintelligence can't surprise us until some point after computers become self-improving.
- harmoat 6y ago
  Aren't large neural network already black boxes we don't understand built by machines we understand?

qwantim1 6y ago

> obviously we will be able to recognise it. There is no basis for that assumption, as you’ve never encountered a superintelligence knowingly.

masswerk 6y ago

> because obviously we will be able to recognise it Mind that even the oldest definitions of intelligence include the ability to assume a position, which is not necessarily held by the subject. This includes all kinds of imitation and mimicry, i.e., simulating a state. Assuming that super intelligence is an extended state of "normal" intelligence, but excelling in capabilities, we must also assume assume that we would be only able to recognize it, if the superintelligence wants to be recognized or allows for it in any other way (e.g., doesn't care about being recognized, etc). Now we could obviously take simulation out of the equation (or definition), but then, what are we talking about?