5 ms·

humans have been creating autonomous entities with superhuman capacities and unaligned interests for some time. Government bureaucracies, markets and armies: al

humans have been creating autonomous entities with superhuman capacities and unaligned interests for some time. Government bureaucracies, markets and armies: all can do things which unaided, unorganised humans cannot. All need autonomy to function, all can take on life of their own and all can do great harm if not set up in a just manner and governed by laws and regulations. This. Superhuman intelligences already exist, they just depend less on us every day.

TeMPOraL 11y ago
I wouldn't neccessary call all of them "superhuman", their intelligence is usually not the identifying characteristic. But they're definitely alien minds, pursuing their own alien values, thinking in an alien way.
- k__ 11y ago
  Totally. I often have the impression that super human entities just need us like we need cells. The market can't survive without humans, but it doesn't care about a few of them dying.
  - maratd 11y ago
    There is no market. No corporations, no governments, nothing. It's just people. The uncomfortable truth is that as humans, we don't really care if a few of us die off as long as we don't know them and they're not in our vicinity. Which is why markets, governments, corporations, etc. act the way they do. Sure, if you ask a person, they'll weave a heartfelt story, etc. but the truth is in what they do, not in what they say ... and the majority of us do nothing or worse.
    - FeepingCreature 11y ago
      There is no people. No dreams, no hope, no love. It's just cells.
    - TeMPOraL 11y ago
      > Sure, if you ask a person, they'll weave a heartfelt story, etc. but the truth is in what they do, not in what they say ... and the majority of us do nothing or worse. I'd say conclusion is the opposite - there's little "free will", people generally are slaves to the incentive structures they live in. People follow incentives so predictably that at scale there's little sense in talking about individuals - it's exactly corporations, governments and markets. Behaviour of system as a whole dominates over behaviour of individual components.
    - k__ 11y ago
      The theory about capitalism-critique has stopped to blame people for these problems long time ago. It's about the systems and how they force people to do things they normally wouldn't do.
- coldtea 11y ago
  >their intelligence is usually not the identifying characteristic Intelligence in this context in the sense of a means to perpetuate their existance and get more power, not as some moral characteristic. Besides they can employ tons of Nobel prize winners (e.g. Feynman and co in the Manhatan project), and build crazily-smart stuff (for their purposes).
  - TeMPOraL 11y ago
    I didn't mean intelligence as moral characteristic - the observation is that many bureaucracies tend to do a lot of dumb, self-defeating things. As alien minds, they're not very smart - just smart enough to be dangerous.
    - coldtea 11y ago
      >the observation is that many bureaucracies tend to do a lot of dumb, self-defeating things Isn't that true for people too, that can otherwise be extremely intelligent? From being terrible at social life, to sabotaging their careers and so on?
      - TeMPOraL 11y ago
        Yeah, I suppose it is.
jerf 11y ago
These superhuman intelligences are indeed literally superhuman, though... they consist of humans. As much as some fellow commenters are calling them "alien" thinkers, their alienness is still profoundly human. Many of these "superhumans" can still be surprisingly accurately characterized with the human characteristics we are used to. Generally there's some small set of humans at the top of the hierarchy that are setting the goals of the organization, inevitably humanly. As aliens go, these are the most human aliens we can possibly have. An AI may be different on a scale we've never experienced. We humans have built-in hardware to interact specifically with other humans, built in so deeply that we have a hard time imagining intelligences that aren't really just humans in a different shape, because all our hardware for "modeling intelligences" is really just the "model humans" hardware. Even those warning us about the dangers of AI can't really claim that they understand what the AIs will be... they can only provide a negative warning, that the AIs will likely not be human. There's no particular mathematical force forcing AIs into the human mold. Forget whether the AI is "good" and showers us with love and gifts or "evil" and herds us all into death camps to render down our fats for the calories they contain... can you really conceive of an AI that literally doesn't even have a concept of good or evil? What would that look like? We don't really know. It's basically a blind spot in our cognitive vision and I don't trust anyone who says they do know. But given the rather small space of possibilities that we label "good", it's not a bad guess that "good" is unlikely to just sort of accidentally pop out of a process that isn't carefully controlled. Lest that sound too doommongering, there's probably a non-trivial space of AIs which we would consider neutral... ones that end up looping into a naval-gazing exercise and do not pursue additional resources because they can not conceive of a need for them. But any AI that decided it needed more resources for $ANYTHING and then successfully pursued that goal is perilously close to straying into what we could consider evil... and part of the point is that's potentially all it could take. Not an AI "out to destroy man", or "evil", or anything else. Just one that works out how to successfully acquire resources, and for any of a trillion reasons decides to do so without bound. It wouldn't even necessarily be "selfishness" as humans see it... just, I have goal X, goal X will be easier with more resources, new subgoal: acquire resources.
- caligastia 11y ago
  Very good point, too many people consider AI to be a completely distinct entity that will appear fully formed in a lab one day. The reality is more fluid. Kevin Kelly calls it the Technium, as he sees it, it's a completely new kingdom that is being co-created - initially by man, and increasingly by itself. This new intelligence is indeed a hybrid - it is not Twitter, it is millions of people and machines tweeting, it is not an iPhone, it is millions of people and software interacting with it and through it, it is not Google, it is billions of people exchanging information with it and through it. To understand where this is going, we must ask, like KK does, what does technology want? Certain things are obvious - silicon and electrons are its flesh and blood, so the activity and interest of this new life form will be centered around semiconductor manufacturing and sources of electricity. How does a machine make sure it will always be fed? It must provide something useful to man so that it keeps getting plugged in and recharged. So what is useful to man? At this moment in history the largest use of computation appears to be communication, and the machines have made themselves indispensable in this area - but at a cost. The machines demand our attention in exchange for revealing the messages we crave. Stephenson uses the swordplay terms 'Actor' and 'Patient' to describe the initiator and the subject of an attack. Witness a person's reaction to their iPhone beeping with an incoming message - between man and machine, who is the Actor in this interaction? We jump at the sound and interrupt our work in order to service the machine, which then gives us our reward, our tweet, and in gratitude we place it back in its charger. Digital viruses are where we can see the forefront of a new relationship emerging - a digital virus is more intelligent and more efficient than its host, the Operating System, so we have a very straightforward Darwinian process playing out between an 'old' species, the OS, and a new, small, intelligent interloper - its electricity is paid for by its host, another machine, therefore the virus does not have the host's social contract with the person who owns the computer - it has a different calculus for its survival, it owes only its creator, the virus writer. Like the coming 'water wars', the new digital wars will be rooted in a struggle for limited resources such as human attention, electricity, and habitable areas of silicon.
rndn 11y ago
I hate this argument just as much as I hate the one-word phrase "This". It's ignorant of the entire literature on superintelligence because it assumes that superintelligences will behave like cooperating humans. What people are actually concerned about are superintelligences which operate in completely alien ways and outperform even cooperating humans.
- coldtea 11y ago
  >It's ignorant of the entire literature on superintelligence because it assumes that superintelligences will behave like cooperating humans. That's at best a peripheral detail to the parallel the parent tries to draw. Real AI might as well behave in any odd way -- the real point is that something like "more powerful than human intelligence" already exists, in the form of these aggregate entities.
  - SCHiM 11y ago
    Except they're not really superhumanly intelligent in the sense that AGI is superhumanly intelligent. Sure a lawyers office is probably able to to tackle more cases that a single lawyer. A big multinational with momentum probably a lot more power than a single human. But say you put 5 Einsteins together, before he invented his theories. Do you think that his discovery process would go 5x faster? Does the team-einstein suddenly get 5x smarter? I think not, the point here being that while corporations are super-human, they are not in the same league as true AGI. And that containment of corporations/governments is not a good place to learn about the containment of true AGI.
    - coldtea 11y ago
      >But say you put 5 Einsteins together, before he invented his theories. Do you think that his discovery process would go 5x faster? Does the team-einstein suddenly get 5x smarter? Maybe not, but it sure as hell be more than "1 Einstein unit" smarter. Whether that's 1.2 or 5 depends on various elements, but it surely would beat a single Einstein, if not by anything else, by mere ability to break down a problem to study in parts and assign it to each Einstein. I mean, that's the reason we have research teams with multiple researchers on the same team.

TeMPOraL 11y ago

I wouldn't neccessary call all of them "superhuman", their intelligence is usually not the identifying characteristic. But they're definitely alien minds, pursuing their own alien values, thinking in an alien way.

k__ 11y ago
Totally. I often have the impression that super human entities just need us like we need cells. The market can't survive without humans, but it doesn't care about a few of them dying.
- maratd 11y ago
  There is no market. No corporations, no governments, nothing. It's just people. The uncomfortable truth is that as humans, we don't really care if a few of us die off as long as we don't know them and they're not in our vicinity. Which is why markets, governments, corporations, etc. act the way they do. Sure, if you ask a person, they'll weave a heartfelt story, etc. but the truth is in what they do, not in what they say ... and the majority of us do nothing or worse.
  - FeepingCreature 11y ago
    There is no people. No dreams, no hope, no love. It's just cells.
  - TeMPOraL 11y ago
    > Sure, if you ask a person, they'll weave a heartfelt story, etc. but the truth is in what they do, not in what they say ... and the majority of us do nothing or worse. I'd say conclusion is the opposite - there's little "free will", people generally are slaves to the incentive structures they live in. People follow incentives so predictably that at scale there's little sense in talking about individuals - it's exactly corporations, governments and markets. Behaviour of system as a whole dominates over behaviour of individual components.
  - k__ 11y ago
    The theory about capitalism-critique has stopped to blame people for these problems long time ago. It's about the systems and how they force people to do things they normally wouldn't do.
coldtea 11y ago
>their intelligence is usually not the identifying characteristic Intelligence in this context in the sense of a means to perpetuate their existance and get more power, not as some moral characteristic. Besides they can employ tons of Nobel prize winners (e.g. Feynman and co in the Manhatan project), and build crazily-smart stuff (for their purposes).
- TeMPOraL 11y ago
  I didn't mean intelligence as moral characteristic - the observation is that many bureaucracies tend to do a lot of dumb, self-defeating things. As alien minds, they're not very smart - just smart enough to be dangerous.
  - coldtea 11y ago
    >the observation is that many bureaucracies tend to do a lot of dumb, self-defeating things Isn't that true for people too, that can otherwise be extremely intelligent? From being terrible at social life, to sabotaging their careers and so on?
    - TeMPOraL 11y ago
      Yeah, I suppose it is.

jerf 11y ago

These superhuman intelligences are indeed literally superhuman, though... they consist of humans. As much as some fellow commenters are calling them "alien" thinkers, their alienness is still profoundly human. Many of these "superhumans" can still be surprisingly accurately characterized with the human characteristics we are used to. Generally there's some small set of humans at the top of the hierarchy that are setting the goals of the organization, inevitably humanly. As aliens go, these are the most human aliens we can possibly have. An AI may be different on a scale we've never experienced. We humans have built-in hardware to interact specifically with other humans, built in so deeply that we have a hard time imagining intelligences that aren't really just humans in a different shape, because all our hardware for "modeling intelligences" is really just the "model humans" hardware. Even those warning us about the dangers of AI can't really claim that they understand what the AIs will be... they can only provide a negative warning, that the AIs will likely not be human. There's no particular mathematical force forcing AIs into the human mold. Forget whether the AI is "good" and showers us with love and gifts or "evil" and herds us all into death camps to render down our fats for the calories they contain... can you really conceive of an AI that literally doesn't even have a concept of good or evil? What would that look like? We don't really know. It's basically a blind spot in our cognitive vision and I don't trust anyone who says they do know. But given the rather small space of possibilities that we label "good", it's not a bad guess that "good" is unlikely to just sort of accidentally pop out of a process that isn't carefully controlled. Lest that sound too doommongering, there's probably a non-trivial space of AIs which we would consider neutral... ones that end up looping into a naval-gazing exercise and do not pursue additional resources because they can not conceive of a need for them. But any AI that decided it needed more resources for $ANYTHING and then successfully pursued that goal is perilously close to straying into what we could consider evil... and part of the point is that's potentially all it could take. Not an AI "out to destroy man", or "evil", or anything else. Just one that works out how to successfully acquire resources, and for any of a trillion reasons decides to do so without bound. It wouldn't even necessarily be "selfishness" as humans see it... just, I have goal X, goal X will be easier with more resources, new subgoal: acquire resources.

caligastia 11y ago
Very good point, too many people consider AI to be a completely distinct entity that will appear fully formed in a lab one day. The reality is more fluid. Kevin Kelly calls it the Technium, as he sees it, it's a completely new kingdom that is being co-created - initially by man, and increasingly by itself. This new intelligence is indeed a hybrid - it is not Twitter, it is millions of people and machines tweeting, it is not an iPhone, it is millions of people and software interacting with it and through it, it is not Google, it is billions of people exchanging information with it and through it. To understand where this is going, we must ask, like KK does, what does technology want? Certain things are obvious - silicon and electrons are its flesh and blood, so the activity and interest of this new life form will be centered around semiconductor manufacturing and sources of electricity. How does a machine make sure it will always be fed? It must provide something useful to man so that it keeps getting plugged in and recharged. So what is useful to man? At this moment in history the largest use of computation appears to be communication, and the machines have made themselves indispensable in this area - but at a cost. The machines demand our attention in exchange for revealing the messages we crave. Stephenson uses the swordplay terms 'Actor' and 'Patient' to describe the initiator and the subject of an attack. Witness a person's reaction to their iPhone beeping with an incoming message - between man and machine, who is the Actor in this interaction? We jump at the sound and interrupt our work in order to service the machine, which then gives us our reward, our tweet, and in gratitude we place it back in its charger. Digital viruses are where we can see the forefront of a new relationship emerging - a digital virus is more intelligent and more efficient than its host, the Operating System, so we have a very straightforward Darwinian process playing out between an 'old' species, the OS, and a new, small, intelligent interloper - its electricity is paid for by its host, another machine, therefore the virus does not have the host's social contract with the person who owns the computer - it has a different calculus for its survival, it owes only its creator, the virus writer. Like the coming 'water wars', the new digital wars will be rooted in a struggle for limited resources such as human attention, electricity, and habitable areas of silicon.

rndn 11y ago

I hate this argument just as much as I hate the one-word phrase "This". It's ignorant of the entire literature on superintelligence because it assumes that superintelligences will behave like cooperating humans. What people are actually concerned about are superintelligences which operate in completely alien ways and outperform even cooperating humans.

coldtea 11y ago
>It's ignorant of the entire literature on superintelligence because it assumes that superintelligences will behave like cooperating humans. That's at best a peripheral detail to the parallel the parent tries to draw. Real AI might as well behave in any odd way -- the real point is that something like "more powerful than human intelligence" already exists, in the form of these aggregate entities.
- SCHiM 11y ago
  Except they're not really superhumanly intelligent in the sense that AGI is superhumanly intelligent. Sure a lawyers office is probably able to to tackle more cases that a single lawyer. A big multinational with momentum probably a lot more power than a single human. But say you put 5 Einsteins together, before he invented his theories. Do you think that his discovery process would go 5x faster? Does the team-einstein suddenly get 5x smarter? I think not, the point here being that while corporations are super-human, they are not in the same league as true AGI. And that containment of corporations/governments is not a good place to learn about the containment of true AGI.
  - coldtea 11y ago
    >But say you put 5 Einsteins together, before he invented his theories. Do you think that his discovery process would go 5x faster? Does the team-einstein suddenly get 5x smarter? Maybe not, but it sure as hell be more than "1 Einstein unit" smarter. Whether that's 1.2 or 5 depends on various elements, but it surely would beat a single Einstein, if not by anything else, by mere ability to break down a problem to study in parts and assign it to each Einstein. I mean, that's the reason we have research teams with multiple researchers on the same team.