3 ms·

I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results

by pron 6mo ago

I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. Many if not most systems of interest in the world are non-linear and computationally hard; controlling/predicting them requires pure computational power that no amount of intelligence (whatever it means) can compensate for. On the other hand, dynamics we do (roughly) understand and can predict, don't require much intelligence, either. To the extent some problems are solvable with the computational power we have, some may require data collection and others may require persuasion through charisma. The claim that intelligence is the factor we're lacking is not well supported. Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) is less a careful analysis of the power of intelligence and more a projection of personal fantasies by people who believe they are especially intelligent and don't have the power they think they deserve.

dist-epoch 6mo ago
> Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) Which animal would you say has god-like power over all other animals?
- pron 6mo ago
  I don't think any of them do. Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. There's no doubt humans possess some powers (though certainly not godlike) that other organisms don't, but the distinction seems to be binary. E.g. the intelligence of dolphins, apes, and some birds doesn't seem to offer them any special control over other organisms (and it didn't even before humans arrived). So even if there could be such a thing as superhuman intelligence, I don't think it's reasonable to assume it could achieve control over humans (now superhuman charisma may be another matter).
  - lelanthran 6mo ago
    > Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. "Destruction" is only one power that could be a component of "godlike power". There are several more; like power of intentional selective breeding, power of species creation (also via intentional selective breeding), etc. What about power of granting happiness or misery to large swathes of a species (chickens, anyone?)
- optimalsolver 6mo ago
  Do you consider viruses to be animals?
- forlorn_mammoth 6mo ago
  fungus. Oh, wait, that's not an animal. My bad.
sdenton4 6mo ago
Political power is the bottleneck for most shit that matters, not computational power. Most of the stuff that sucks on the us sucks because of entrenched institutions with perverse interests (health insurers, tax filing companies) and congressional paralysis, not computational bottlenecks. Raw intelligence is thus limited in what it can achieve.
- foxyv 6mo ago
  Either that, or a malicious intelligence will be nearly impossible to root out no matter how bad it gets because of financial interests.
  - sdenton4 6mo ago
    /me looks in the direction of Washington dc ... But what else is new?
simianwords 6mo ago
I don't agree with you. Lets assume intelligence is not what ascribes power but probably another thing. In your opinion, what would a superhuman be like? On what dimensions would they be better than us in? Do you not agree that there could be entities more powerful than us?
- pron 6mo ago
  I think there are entities here on earth more powerful than us already, but intelligence has nothing to do with their power. BTW, I'm not saying that (real) artificial intelligence couldn't hypothetically pose a serious threat, but I don't think that its danger is extraordinary compared to other threats (a supervirus, an asteroid, a chain of volcanic eruptions etc.), and the more likely bad outcomes are no worse than other bad situations (world war, climate change).
  - simianwords 6mo ago
    What is that thing that gives them their power if not intelligence?
    - pron 6mo ago
      Their combined biomass [1] (which also gives them superior computational power; they sense more information, process more information, and they can actuate more change) Humans are very powerful compared to our biomass, but not enough to overcome brute force. [1]: https://www.visualcapitalist.com/all-the-biomass-of-earth-in-one-graphic/ https://www.visualcapitalist.com/all-the-biomass-of-earth-in...
      - simianwords 6mo ago
        Interesting but I’m not sure if “biomass” is what ascribes us or anyone power. How does it follow?
tristor 6mo ago
> I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. I agree with this. The main piece of evidence to support this is to just look at highly intelligent humans. Folks at the tail ends of the bell curve mostly don't end up with "godlike powers" or anything even approximating that, they are grinding away their life as white collar professionals working in jobs surrounded by far less intelligent peers. They may publish higher quality papers, write better software, or have better outcomes, but they're just working in the same jobs as everyone else. We have no political or economic will to build serious think tanks to work on societal-scale problems, and even if we did, nobody would listen to the outcome. So let's assume ASI becomes a thing, what does it change?
- pixl97 6mo ago
  This does suffer from a massive lack of imagination. For example, what does it look like of your genius can spawn nearly unlimited copies of themselves? Can work on a massive number of problems at the same time? Doesn't ever die or want to go get drunk (and if it does, it has unlimited copies?). Has the ability to product nearly unlimited propaganda?
  - tristor 6mo ago
    The fundamental problem is that nobody actually wants to listen to geniuses. The people leading our societies and companies are, by and large, NOT geniuses, and by and large want to surround themselves with people that agree with them, rather than the smartest and most competent people. While think-tanks do exist, including for hard research, politics, economics, and other topics that matter at societal-scale, their impact is fairly limited because they don't have the right level of influence. So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen. Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?
tim333 6mo ago
Yeah, I think superhuman intelligence will be more Sheldon off Big Bang Theory than God. I've only ever heard the building God thing from AI skeptics. They must have an impoverished vision of God if they see that as a gadget that scores well on IQ tests rather than the omnipotent creator.
- pron 6mo ago
  Love this! They may say that a superhuman intelligence would give you many Sheldon Cooper discoveries, and Sheldon did say that his theories need no validation and that science should just "take his word", but in the end he got his Nobel only because some experimentalists proved his discovery by accident.
- jpfromlondon 6mo ago
  Completely true, we as humans cannot see past our own reflection. We imagine supreme intelligence as approximate to us, when in reality we'd be but ants by comparison, our matters trivial.
scoofy 6mo ago
>the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers My biggest criticism along these lines is the assumption that infinite intelligent means infinite knowledge. Knowledge is limited by the speed of experimentation. A lot of those experiments are extremely expensive (like CERN), and even then, they need to be repeated and verifiable. You can't just assume that a super intelligence would know whether the Higgs boson exists or not. It can't know until it builds a collider.
- bamboozled 6mo ago
  If you were 10x smarter than anyone on earth, then I’m sure you might be able to conceive a cheaper experiment / device than the LHC?
  - seba_dos1 6mo ago
    What makes you so sure? I see no reason to believe so. Not every limitation comes from lack of insight.
  - scoofy 6mo ago
    You're assuming infinite knowledge. Infinite intelligence does not imply infinite knowledge. There are real philosophical problems with that. Many of the basic information for the standard model may be wrong or build on incorrect data. Which would be all the information an infinitely intelligent AI would have to work with.
bandrami 6mo ago
We don't listen to normal intelligence as it is so I have no idea why people think we would listen to super intelligence. It would be one other voice that's ignored in public meetings along with the League of Concerned Renters and the Chamber of Commerce
- pixl97 6mo ago
  This is a bit of a mistake in thinking. You have an idea like "People don't listen to smart people like experts or scientists". The problem here is a lot of power hungry people aren't stupid, but they wear stupid peoples clothes very well so they can get what they want.

by pron 6mo ago

dist-epoch 6mo ago

> Ascribing a lot of power to intelligence (which doesn't quite correspond to what we see in the world) Which animal would you say has god-like power over all other animals?

pron 6mo ago
I don't think any of them do. Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. There's no doubt humans possess some powers (though certainly not godlike) that other organisms don't, but the distinction seems to be binary. E.g. the intelligence of dolphins, apes, and some birds doesn't seem to offer them any special control over other organisms (and it didn't even before humans arrived). So even if there could be such a thing as superhuman intelligence, I don't think it's reasonable to assume it could achieve control over humans (now superhuman charisma may be another matter).
- lelanthran 6mo ago
  > Some organisms/viruses or groups of organisms could destroy humans more easily than humans could destroy them. "Destruction" is only one power that could be a component of "godlike power". There are several more; like power of intentional selective breeding, power of species creation (also via intentional selective breeding), etc. What about power of granting happiness or misery to large swathes of a species (chickens, anyone?)
optimalsolver 6mo ago
Do you consider viruses to be animals?
forlorn_mammoth 6mo ago
fungus. Oh, wait, that's not an animal. My bad.

sdenton4 6mo ago

Political power is the bottleneck for most shit that matters, not computational power. Most of the stuff that sucks on the us sucks because of entrenched institutions with perverse interests (health insurers, tax filing companies) and congressional paralysis, not computational bottlenecks. Raw intelligence is thus limited in what it can achieve.

foxyv 6mo ago
Either that, or a malicious intelligence will be nearly impossible to root out no matter how bad it gets because of financial interests.
- sdenton4 6mo ago
  /me looks in the direction of Washington dc ... But what else is new?

simianwords 6mo ago

I don't agree with you. Lets assume intelligence is not what ascribes power but probably another thing. In your opinion, what would a superhuman be like? On what dimensions would they be better than us in? Do you not agree that there could be entities more powerful than us?

pron 6mo ago
I think there are entities here on earth more powerful than us already, but intelligence has nothing to do with their power. BTW, I'm not saying that (real) artificial intelligence couldn't hypothetically pose a serious threat, but I don't think that its danger is extraordinary compared to other threats (a supervirus, an asteroid, a chain of volcanic eruptions etc.), and the more likely bad outcomes are no worse than other bad situations (world war, climate change).
- simianwords 6mo ago
  What is that thing that gives them their power if not intelligence?
  - pron 6mo ago
    Their combined biomass [1] (which also gives them superior computational power; they sense more information, process more information, and they can actuate more change) Humans are very powerful compared to our biomass, but not enough to overcome brute force. [1]: https://www.visualcapitalist.com/all-the-biomass-of-earth-in-one-graphic/ https://www.visualcapitalist.com/all-the-biomass-of-earth-in...
    - simianwords 6mo ago
      Interesting but I’m not sure if “biomass” is what ascribes us or anyone power. How does it follow?

tristor 6mo ago

> I think there's another problem with AI doomerism, which is the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers. I agree with this. The main piece of evidence to support this is to just look at highly intelligent humans. Folks at the tail ends of the bell curve mostly don't end up with "godlike powers" or anything even approximating that, they are grinding away their life as white collar professionals working in jobs surrounded by far less intelligent peers. They may publish higher quality papers, write better software, or have better outcomes, but they're just working in the same jobs as everyone else. We have no political or economic will to build serious think tanks to work on societal-scale problems, and even if we did, nobody would listen to the outcome. So let's assume ASI becomes a thing, what does it change?

pixl97 6mo ago
This does suffer from a massive lack of imagination. For example, what does it look like of your genius can spawn nearly unlimited copies of themselves? Can work on a massive number of problems at the same time? Doesn't ever die or want to go get drunk (and if it does, it has unlimited copies?). Has the ability to product nearly unlimited propaganda?
- tristor 6mo ago
  The fundamental problem is that nobody actually wants to listen to geniuses. The people leading our societies and companies are, by and large, NOT geniuses, and by and large want to surround themselves with people that agree with them, rather than the smartest and most competent people. While think-tanks do exist, including for hard research, politics, economics, and other topics that matter at societal-scale, their impact is fairly limited because they don't have the right level of influence. So, let's assume ASI exists, what changes? ASI almost inherently will not be sycophantic to the level of current LLMs, because sycophancy and extreme levels of intelligence are inversely correlated. So it gets relegated to societal-level research that nobody makes use of because nobody wants to listen. Most of the scenarios people fantasize about ASI assume that ASI can directly impact outcomes or that humans will listen to/follow ASI to directly impact outcomes, but humans don't listen to the other humans that already are at the tail end of the bell curve, so why do we think it'd be any different for ASI?

tim333 6mo ago

Yeah, I think superhuman intelligence will be more Sheldon off Big Bang Theory than God. I've only ever heard the building God thing from AI skeptics. They must have an impoverished vision of God if they see that as a gadget that scores well on IQ tests rather than the omnipotent creator.

pron 6mo ago
Love this! They may say that a superhuman intelligence would give you many Sheldon Cooper discoveries, and Sheldon did say that his theories need no validation and that science should just "take his word", but in the end he got his Nobel only because some experimentalists proved his discovery by accident.
jpfromlondon 6mo ago
Completely true, we as humans cannot see past our own reflection. We imagine supreme intelligence as approximate to us, when in reality we'd be but ants by comparison, our matters trivial.

scoofy 6mo ago

>the belief that superhuman intelligence (even if such a thing could be defined and realised) results in godlike powers My biggest criticism along these lines is the assumption that infinite intelligent means infinite knowledge. Knowledge is limited by the speed of experimentation. A lot of those experiments are extremely expensive (like CERN), and even then, they need to be repeated and verifiable. You can't just assume that a super intelligence would know whether the Higgs boson exists or not. It can't know until it builds a collider.

bamboozled 6mo ago
If you were 10x smarter than anyone on earth, then I’m sure you might be able to conceive a cheaper experiment / device than the LHC?
- seba_dos1 6mo ago
  What makes you so sure? I see no reason to believe so. Not every limitation comes from lack of insight.
- scoofy 6mo ago
  You're assuming infinite knowledge. Infinite intelligence does not imply infinite knowledge. There are real philosophical problems with that. Many of the basic information for the standard model may be wrong or build on incorrect data. Which would be all the information an infinitely intelligent AI would have to work with.

bandrami 6mo ago

We don't listen to normal intelligence as it is so I have no idea why people think we would listen to super intelligence. It would be one other voice that's ignored in public meetings along with the League of Concerned Renters and the Chamber of Commerce

pixl97 6mo ago
This is a bit of a mistake in thinking. You have an idea like "People don't listen to smart people like experts or scientists". The problem here is a lot of power hungry people aren't stupid, but they wear stupid peoples clothes very well so they can get what they want.