5 ms·

> what if high intelligence doesn't require self-awareness How could that work concretely? I'm struggling to think of how an agent could be intelligent, e.g. a

> what if high intelligence doesn't require self-awareness How could that work concretely? I'm struggling to think of how an agent could be intelligent, e.g. able to effectively achieve its goals, without being aware of itself. I suspect that, if you could communicate with one, any sufficiently intelligent being would be aware that it's a thinking being. What else does "self-awareness" encompass that I'm missing?

snowwrestler 9y ago
> be intelligent, e.g. able to effectively achieve its goals Well, a lot of very intelligent people never achieve their goals either. I'd suggest that intelligence is not the achievement of goals but the formulation of goals. And funny enough, people generally have no self-awareness about how they come to have goals. Some kid grows up wanting to be a fighter pilot. Why? Why does some other kid want to be a writer? Why do I like peach ice cream more than chocolate? Achieving the goal of getting some peach ice cream is a lot easier than understanding why I want to. Intelligent beings want things, on their own. Rational consciousness is just the tool we use to get what we want. The importance of articulated self-awareness to intelligence is probably overstated by humans.
- aeorgnoieang 9y ago
  > Well, a lot of very intelligent people never achieve their goals either. They're not intelligent with respect to the goals they're not achieving. Or they're competing with other people that are more 'intelligent'. There are (of course!) confounding factors but, all else being equal, the more intelligent agent should win. Let's use a concrete example. Consider someone that's very good at solving math problems but is unhappy with their job. What's the problem with just considering that they're intelligent with respect to solving math problems but bad at (all of the skills necessary to be good at) finding a satisfying job? Achieving goals, winning 'games', etc., seems like a really useful and really general way of measuring the effectiveness of things and that seems pretty close to, if not exactly, what intelligence is (if it's any one thing in particular). > I'd suggest that intelligence is not the achievement of goals but the formulation of goals. What's the motivation to define intelligence in that way?
  - Ygg2 9y ago
    > the more intelligent agent should win. Not necessarily. An intelligent agent could lose to an agent that can simply see the future.
    - rocqua 9y ago
      In the sense here, an agent that can see the future is 'infinitely intelligent'. Seeing the future has a tendency of becoming paradoxical when you start acting on that information though.
      - Ygg2 9y ago
        I don't think that's true. A beast with ability to see two days in advance could easily kill a human, but still not be able to do anything else, like beat him at chess or make tools.
        emiliobumachar 9y ago
        Your example seems reasonable, but not as a rebuttal to the paradoxical nature of seeing the future. The paradoxes are the same as in time travel. Imagine your best sees itself hunted down two days in advance; it simply goes somewhere else so the hunters don't find it. Then it didn't really see the future, did it?
        abiox 9y ago
        > it didn't really see the future, did it depends on your model of time and the universe, i suppose. perhaps "a future" vs "the future".
    - aeorgnoieang 9y ago
      Magical powers (and powerful, potentially paradoxical ones at that) are an extremely blatant violation of "all else being equal".
DowsingSpoon 9y ago
Imagine an idiot-savant. He can tell you with complete accuracy the answer to complex mathematical problems, but has no understanding of how he arrives at the result. Now imagine a hyper intelligent alien who is like the savant. It can construct interstellar vessels, but has no self-awareness of how it came up with the designs. Though, it is clear that they do work. The alien can observe a human and know exactly how to best injure him or her, but it cannot understand how it came to know human anatomy, or even why the attacks work. For example, if it performed a Vulcan neck pinch, and you asked it how it figured out how to do that then it might tell you that "it just felt like the right thing to do." It has no conscious notion of the human mind or neurology. The alien in Blindsight also apparently does not care to understand such things, and may lack the capacity for this kind of thought. It runs entirely on instinct, but these instincts bake in extremely complex behaviors that are beyond human capabilities. I'm also reminded of two other aliens from other Sci Fi series: 1) the Orkz from Warhammer 40k. Dumb as a rock, but can build devastatingly effective vehicles and weapons from salvage and garbage. God knows how. They certainly have no idea. 2) The Motie engineers from "The Mote in God's Eye." They can instinctually build and improve on machines but cannot understand, or come close to explaining, How they did it.
- aeorgnoieang 9y ago
  This confuses 'levels', or what exactly is the intelligent 'agent'. The idiot-savant is not generally intelligent. Depending on what you mean by "complex mathematical problems" even the neural components actually responsible for solving the problems might not be intelligent. In the case of actual idiot-savants, I'd suspect those components aren't 'intelligent'; they're too simple. The alien in Blindsight, if it really "runs entirely on instinct", then I'd expect it to do badly in novel situations, unless it's version of "instincts" basically "bake in extremely complex behaviors" that make it functionally equivalent or similar to a human – at the level we're used to dealing with as humans. Also note that most of your examples are fictional, i.e. not actual evidence beyond maybe extremely weak plausibility. AFAIK, every "high intelligence" of which actual people have actual strong evidence is self aware. That seems like it should be an important fact! Also, you're conflating self-awareness with the ability to (verbally) explain, i.e. communicate, thoughts to others. By "self-awareness", I was thinking of an 'intelligent system' including some sub-model of itself in whatever the relevant domain it is we're considering. For a 'general-purpose reproductive-fitness-maximizing-strategy executor' that's intelligent, e.g. us, it seems pretty unlikely that they don't, generally, model themselves. We should probably taboo "self-awareness" as it's pretty obviously too slippery and ambiguous.

snowwrestler 9y ago

> be intelligent, e.g. able to effectively achieve its goals Well, a lot of very intelligent people never achieve their goals either. I'd suggest that intelligence is not the achievement of goals but the formulation of goals. And funny enough, people generally have no self-awareness about how they come to have goals. Some kid grows up wanting to be a fighter pilot. Why? Why does some other kid want to be a writer? Why do I like peach ice cream more than chocolate? Achieving the goal of getting some peach ice cream is a lot easier than understanding why I want to. Intelligent beings want things, on their own. Rational consciousness is just the tool we use to get what we want. The importance of articulated self-awareness to intelligence is probably overstated by humans.

aeorgnoieang 9y ago
> Well, a lot of very intelligent people never achieve their goals either. They're not intelligent with respect to the goals they're not achieving. Or they're competing with other people that are more 'intelligent'. There are (of course!) confounding factors but, all else being equal, the more intelligent agent should win. Let's use a concrete example. Consider someone that's very good at solving math problems but is unhappy with their job. What's the problem with just considering that they're intelligent with respect to solving math problems but bad at (all of the skills necessary to be good at) finding a satisfying job? Achieving goals, winning 'games', etc., seems like a really useful and really general way of measuring the effectiveness of things and that seems pretty close to, if not exactly, what intelligence is (if it's any one thing in particular). > I'd suggest that intelligence is not the achievement of goals but the formulation of goals. What's the motivation to define intelligence in that way?
- Ygg2 9y ago
  > the more intelligent agent should win. Not necessarily. An intelligent agent could lose to an agent that can simply see the future.
  - rocqua 9y ago
    In the sense here, an agent that can see the future is 'infinitely intelligent'. Seeing the future has a tendency of becoming paradoxical when you start acting on that information though.
    - Ygg2 9y ago
      I don't think that's true. A beast with ability to see two days in advance could easily kill a human, but still not be able to do anything else, like beat him at chess or make tools.
      - emiliobumachar 9y ago
        Your example seems reasonable, but not as a rebuttal to the paradoxical nature of seeing the future. The paradoxes are the same as in time travel. Imagine your best sees itself hunted down two days in advance; it simply goes somewhere else so the hunters don't find it. Then it didn't really see the future, did it?
        abiox 9y ago
        > it didn't really see the future, did it depends on your model of time and the universe, i suppose. perhaps "a future" vs "the future".
  - aeorgnoieang 9y ago
    Magical powers (and powerful, potentially paradoxical ones at that) are an extremely blatant violation of "all else being equal".

DowsingSpoon 9y ago

Imagine an idiot-savant. He can tell you with complete accuracy the answer to complex mathematical problems, but has no understanding of how he arrives at the result. Now imagine a hyper intelligent alien who is like the savant. It can construct interstellar vessels, but has no self-awareness of how it came up with the designs. Though, it is clear that they do work. The alien can observe a human and know exactly how to best injure him or her, but it cannot understand how it came to know human anatomy, or even why the attacks work. For example, if it performed a Vulcan neck pinch, and you asked it how it figured out how to do that then it might tell you that "it just felt like the right thing to do." It has no conscious notion of the human mind or neurology. The alien in Blindsight also apparently does not care to understand such things, and may lack the capacity for this kind of thought. It runs entirely on instinct, but these instincts bake in extremely complex behaviors that are beyond human capabilities. I'm also reminded of two other aliens from other Sci Fi series: 1) the Orkz from Warhammer 40k. Dumb as a rock, but can build devastatingly effective vehicles and weapons from salvage and garbage. God knows how. They certainly have no idea. 2) The Motie engineers from "The Mote in God's Eye." They can instinctually build and improve on machines but cannot understand, or come close to explaining, How they did it.

aeorgnoieang 9y ago
This confuses 'levels', or what exactly is the intelligent 'agent'. The idiot-savant is not generally intelligent. Depending on what you mean by "complex mathematical problems" even the neural components actually responsible for solving the problems might not be intelligent. In the case of actual idiot-savants, I'd suspect those components aren't 'intelligent'; they're too simple. The alien in Blindsight, if it really "runs entirely on instinct", then I'd expect it to do badly in novel situations, unless it's version of "instincts" basically "bake in extremely complex behaviors" that make it functionally equivalent or similar to a human – at the level we're used to dealing with as humans. Also note that most of your examples are fictional, i.e. not actual evidence beyond maybe extremely weak plausibility. AFAIK, every "high intelligence" of which actual people have actual strong evidence is self aware. That seems like it should be an important fact! Also, you're conflating self-awareness with the ability to (verbally) explain, i.e. communicate, thoughts to others. By "self-awareness", I was thinking of an 'intelligent system' including some sub-model of itself in whatever the relevant domain it is we're considering. For a 'general-purpose reproductive-fitness-maximizing-strategy executor' that's intelligent, e.g. us, it seems pretty unlikely that they don't, generally, model themselves. We should probably taboo "self-awareness" as it's pretty obviously too slippery and ambiguous.