4 ms·

The recent articles on Artificial Superintelligence have got me thinking about interstellar MAD. It seems that once you have ASI you really have the capability

The recent articles on Artificial Superintelligence have got me thinking about interstellar MAD. It seems that once you have ASI you really have the capability to do anything within the laws of physics because if there's anything you don't know how to do, you just add intelligence until you do. If so, it seems that primitive civilizations can become enormous, exponentially expanding threats to established civilizations in relatively no time at all, and in fact they are extremely vulnerable if they choose not to, not only to established players but to any new ones that come along.

api 12y ago
A little off topic but... "you just add intelligence until you do" So many assumptions. I'd challenge everyone to look deeply into learning theory and combinatorics. We have no reason to believe that intelligence is something you can just scale up without bound, and many reasons to believe otherwise. For starters: nature "invented" intelligence hundreds of millions of years ago with the evolution of neural nets, and it's a powerful adaptation that confers significant advantage. Why are we the only intelligence with our level of analytical capability on Earth? If it were so easy to "just add intelligence," almost every large animal in nature should be capable of language, mathematics, and logic by now. As someone who spent many years studying AI, machine learning, combinatorics, evolutionary theory, and related topics, some of the stuff that's been flying around about AI recently is making me itch.
- 6d0debc071 12y ago
  > For starters: nature "invented" intelligence hundreds of millions of years ago with the evolution of neural nets, and it's a powerful adaptation that confers significant advantage. Why are we the only intelligence with our level of analytical capability on Earth? Lots of animals seem to lack the ability to use fine tools and have limited vocal ranges that would make communication far more difficult. I'm inclined to question the 'significant advantage' part in its unqualified form.
  - api 12y ago
    Strategic thinking, categorization, and the recognition of complex environmental patterns are among the abilities that would be very valuable even for mutes without opposable thumbs.
    - 6d0debc071 12y ago
      That sounds reasonable, but I'm not sure what those things translate into. Take something that's reasonably intelligent - a wolf for instance. How, for most of its evolutionary history, would being able to recognise complex environmental patterns, as a human would, have made it successful enough to offset the energy cost of doing so? What part of the energy of the system accessible with their abilities are they missing?
- maxpupmax 12y ago
  I'm interested. Could you provide a starting point for reading? Preferably with extensive references!
  - api 12y ago
    I wish I had the time to write a book on it. It's a deeply esoteric subject though, and I am too insanely busy right now. Better yet, I really wish someone far more up to date than I am who is actually working today in the field would take up the cause of airing out some of this silliness. I considered it harmless up to the point I started reading actual calls to regulate or even ban basic CS research. If I see any more of that stuff there is one person I know that I might email and try to convince to do it. The last thing we need is to shut down one of our most productive areas of innovation with baseless fear mongering built on towers of wobbly assumptions and cheesy sci fi scenarios.
- InclinedPlane 12y ago
  Precisely. So much theory around super-intelligences does very little to actually address the nature of intelligence or awareness, it merely treats intelligence like horsepower or computing cycles. This despite the fact that we've spent almost a century proving that intelligence differs greatly from raw computing power, and is a far more complex and subtle thing. We don't know what the returns on computational power are in regards to intelligence levels, it may actually be non-linear (one way or the other). Or it may be mostly uncorrelated.
  - api 12y ago
    All that is true and more. Intelligence is bad enough, but start talking about consciousness and sentience and we don't even know what we don't know. Our understanding there is so spotty there are major camps of people who want to just chicken out on the whole question and claim thes things do not exist. Humans don't like to admit they don't understand things... Threatens the ego.
- RickHull 12y ago
  An analogy might be made in terms of escape velocity. For the longest time, we hurled stones at targets in trees and towers. However, in the times leading up to WWII, we (Germans, mostly) began to sense that we could launch objects at the Moon or Mars with plausibly incremental improvements. A tipping or inflection point is reached where things go all zoomy and nonlinear. 70 years later, we might be entering that era for intelligence. These sorts of things are not easily rejected by looking at historical trajectories.
  - darkmighty 12y ago
    You are using the historical trajectory of rockets in a way to prove your point :) But looking precisely at rockets, there was a boom in technology where the distances and velocities we could go increased exponentially until the tech matured, and now it's settled and still pretty expensive to go up there. In fact, this trend seems common with technologies (rapid growth until a wall is reached). I wish I had a link to the graphs I have in mind that make this argument.
  - api 12y ago
    You can scale a rocket by mostly just making it bigger. I'm not at all convinced you can do this with intelligence. It sssumes many things: intelligence is one thing, no new problems of a fundamentally different order appear with scale, and there are no scaling laws that apply to intelligence. Of those assumptions, the one that seems most common around here is that intelligence is one thing.
- seanmcdirmid 12y ago
  We killed off all animals that could conceivably compete with us on intelligence (Neanderthals, various other hominids). The advantage of being first os that there is no need for second. Also, there are other animals that approach our intelligence and are probably sentient. They are just a bit behind us, but enough so that we dont recognize them as so (the pack of communication is also an issue).

api 12y ago

A little off topic but... "you just add intelligence until you do" So many assumptions. I'd challenge everyone to look deeply into learning theory and combinatorics. We have no reason to believe that intelligence is something you can just scale up without bound, and many reasons to believe otherwise. For starters: nature "invented" intelligence hundreds of millions of years ago with the evolution of neural nets, and it's a powerful adaptation that confers significant advantage. Why are we the only intelligence with our level of analytical capability on Earth? If it were so easy to "just add intelligence," almost every large animal in nature should be capable of language, mathematics, and logic by now. As someone who spent many years studying AI, machine learning, combinatorics, evolutionary theory, and related topics, some of the stuff that's been flying around about AI recently is making me itch.

6d0debc071 12y ago
> For starters: nature "invented" intelligence hundreds of millions of years ago with the evolution of neural nets, and it's a powerful adaptation that confers significant advantage. Why are we the only intelligence with our level of analytical capability on Earth? Lots of animals seem to lack the ability to use fine tools and have limited vocal ranges that would make communication far more difficult. I'm inclined to question the 'significant advantage' part in its unqualified form.
- api 12y ago
  Strategic thinking, categorization, and the recognition of complex environmental patterns are among the abilities that would be very valuable even for mutes without opposable thumbs.
  - 6d0debc071 12y ago
    That sounds reasonable, but I'm not sure what those things translate into. Take something that's reasonably intelligent - a wolf for instance. How, for most of its evolutionary history, would being able to recognise complex environmental patterns, as a human would, have made it successful enough to offset the energy cost of doing so? What part of the energy of the system accessible with their abilities are they missing?
maxpupmax 12y ago
I'm interested. Could you provide a starting point for reading? Preferably with extensive references!
- api 12y ago
  I wish I had the time to write a book on it. It's a deeply esoteric subject though, and I am too insanely busy right now. Better yet, I really wish someone far more up to date than I am who is actually working today in the field would take up the cause of airing out some of this silliness. I considered it harmless up to the point I started reading actual calls to regulate or even ban basic CS research. If I see any more of that stuff there is one person I know that I might email and try to convince to do it. The last thing we need is to shut down one of our most productive areas of innovation with baseless fear mongering built on towers of wobbly assumptions and cheesy sci fi scenarios.
InclinedPlane 12y ago
Precisely. So much theory around super-intelligences does very little to actually address the nature of intelligence or awareness, it merely treats intelligence like horsepower or computing cycles. This despite the fact that we've spent almost a century proving that intelligence differs greatly from raw computing power, and is a far more complex and subtle thing. We don't know what the returns on computational power are in regards to intelligence levels, it may actually be non-linear (one way or the other). Or it may be mostly uncorrelated.
- api 12y ago
  All that is true and more. Intelligence is bad enough, but start talking about consciousness and sentience and we don't even know what we don't know. Our understanding there is so spotty there are major camps of people who want to just chicken out on the whole question and claim thes things do not exist. Humans don't like to admit they don't understand things... Threatens the ego.
RickHull 12y ago
An analogy might be made in terms of escape velocity. For the longest time, we hurled stones at targets in trees and towers. However, in the times leading up to WWII, we (Germans, mostly) began to sense that we could launch objects at the Moon or Mars with plausibly incremental improvements. A tipping or inflection point is reached where things go all zoomy and nonlinear. 70 years later, we might be entering that era for intelligence. These sorts of things are not easily rejected by looking at historical trajectories.
- darkmighty 12y ago
  You are using the historical trajectory of rockets in a way to prove your point :) But looking precisely at rockets, there was a boom in technology where the distances and velocities we could go increased exponentially until the tech matured, and now it's settled and still pretty expensive to go up there. In fact, this trend seems common with technologies (rapid growth until a wall is reached). I wish I had a link to the graphs I have in mind that make this argument.
- api 12y ago
  You can scale a rocket by mostly just making it bigger. I'm not at all convinced you can do this with intelligence. It sssumes many things: intelligence is one thing, no new problems of a fundamentally different order appear with scale, and there are no scaling laws that apply to intelligence. Of those assumptions, the one that seems most common around here is that intelligence is one thing.
seanmcdirmid 12y ago
We killed off all animals that could conceivably compete with us on intelligence (Neanderthals, various other hominids). The advantage of being first os that there is no need for second. Also, there are other animals that approach our intelligence and are probably sentient. They are just a bit behind us, but enough so that we dont recognize them as so (the pack of communication is also an issue).