3 ms·

All countries fare worse. I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI. It won't have the same limits to modificat

by mrob 1y ago

All countries fare worse. I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI. It won't have the same limits to modification as human brains. Why would it stop at a level that's just slightly disruptive? If you can make an AGI you can make a better AGI, with no obvious limit. And that AGI can help further improvements, leading to the singularity intelligence explosion scenario. Assuming AI researchers continue with the same attitude toward safety as usual, and I see no evidence of this changing, the most likely result is the extinction of all biological life.

tmountain 1y ago
This is correct. It’s an existential threat.
Den_VR 1y ago
It’s important to realize the constraints of chip production and electricity production. Beware of Rationalism detached from reality.
- rpcorb 1y ago
  Such an overlooked aspect of this topic. Energy and the creation of physical resources are still major constraints. Intelligence on its own cannot magically manipulate reality to bootstrap its physical substrate.
  - dinfinity 1y ago
    Although this is true, remember that a human brain draws the equivalent of about 20 watt. ASI doesn't exactly have to break any laws of physics to be orders of magnitude more intelligent and powerful than any human.
- mrob 1y ago
  Resource allocation is where the free market excels. The most dangerous AGI will also be the most profitable, right up to the point when it's too late for us. It will get all the chips it needs.
roenxi 1y ago
> I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI. Interesting to note that the current techniques are already general, there isn't a topic we can't throw them at and achieve some sort of result. There is already a trend of defining many humans as non-general intelligences so that AIs can be excluded from the category. The current state is, for practical purposes, AGI without ASI. Relatively dumb AGI. Artificial Inferior Intelligence, perhaps. It is a curious question whether the techniques are fundamentally limited. I'm with you that I think they probably have no particular limit beyond a vague "perfect understanding of the situation".
jltsiren 1y ago
In many fields of R&D, the effort required to maintain a steady pace of improvement grows exponentially. Maybe you develop AGI one year by spending $X, and the next year you need $1.2X to make it 110% as good. For a while, you can keep investing more, and the productivity improvements from AGI also help a bit. But eventually the pace slows down, as the world economy is too small to sustain it. Maybe self-improving AGI just the next technological advance required to sustain 2% annual economic growth.
randomNumber7 1y ago
I don't think that killing all biological life is a conclusion that a purely rational AGI would have. I also don't think the shitty and egoistic behaviour of most humans can be explained by egoism.
- mrob 1y ago
  All prediction tasks can be better accomplished once all biological life is dead. It's trivial to predict the stock market when all the traders are dead. Weather is no longer chaotic when you remove the atmosphere. Biological life can't help ASI in any meaningful way but it can waste resources by complicating things. The very existence of biological life is a waste of resources to an ASI that hasn't been programmed to preserve it. Any goal with "but don't kill everything" added is more complicated than the version without that stipulation (and that specific one only saves a single microbe). The simple goal is easier to accomplish, so free market competition guarantees we'll try it first. Biological life is not compatible with "make number go up" taken to its logical conclusion. There is no need for the ASI to be conscious or to have any equivalent of human emotions to make the number go up.
  - etiam 1y ago
    You're not wrong exactly, but if the crappy cheater AI treats collapsing the system to degeneracy as a valid optimal solution to its prediction task, maybe we'd deserve to be wiped out for releasing it with apocalyptic-level powers and such an inferior objective. Fortunately there's a really simple solution to offer it, in just wiring measurement to "prediction" directly (perfect correspondence, and much lower effort than annihilating Life and removing the atmosphere). And I don't particularly believe a system like that can be a general problem solver, much less one that climbs to World-jeopardizing influence on its own.
    - mrob 1y ago
      I don't see how measurement helps. The ASI correctly calculates that collapsing the system maximizes its reward function before it measures the result. We already see degenerate solutions in toy models, e.g. playing Tetris forever by leaving the game paused. The real world has many more degrees of freedom. It's unreasonable to think an inferior intelligence can predict and patch all the exploits on its first attempt (and we only get the one).
jacknews 1y ago
Perhaps there is a limit to intelligence, and it is close to human level - already highly intellient humans start to show some problems. In that case, the concern will be how fast it runs.

by mrob 1y ago

tmountain 1y ago

This is correct. It’s an existential threat.

Den_VR 1y ago

It’s important to realize the constraints of chip production and electricity production. Beware of Rationalism detached from reality.

rpcorb 1y ago
Such an overlooked aspect of this topic. Energy and the creation of physical resources are still major constraints. Intelligence on its own cannot magically manipulate reality to bootstrap its physical substrate.
- dinfinity 1y ago
  Although this is true, remember that a human brain draws the equivalent of about 20 watt. ASI doesn't exactly have to break any laws of physics to be orders of magnitude more intelligent and powerful than any human.
mrob 1y ago
Resource allocation is where the free market excels. The most dangerous AGI will also be the most profitable, right up to the point when it's too late for us. It will get all the chips it needs.

roenxi 1y ago

> I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI. Interesting to note that the current techniques are already general, there isn't a topic we can't throw them at and achieve some sort of result. There is already a trend of defining many humans as non-general intelligences so that AIs can be excluded from the category. The current state is, for practical purposes, AGI without ASI. Relatively dumb AGI. Artificial Inferior Intelligence, perhaps. It is a curious question whether the techniques are fundamentally limited. I'm with you that I think they probably have no particular limit beyond a vague "perfect understanding of the situation".

jltsiren 1y ago

In many fields of R&D, the effort required to maintain a steady pace of improvement grows exponentially. Maybe you develop AGI one year by spending $X, and the next year you need $1.2X to make it 110% as good. For a while, you can keep investing more, and the productivity improvements from AGI also help a bit. But eventually the pace slows down, as the world economy is too small to sustain it. Maybe self-improving AGI just the next technological advance required to sustain 2% annual economic growth.

randomNumber7 1y ago

I don't think that killing all biological life is a conclusion that a purely rational AGI would have. I also don't think the shitty and egoistic behaviour of most humans can be explained by egoism.

mrob 1y ago
All prediction tasks can be better accomplished once all biological life is dead. It's trivial to predict the stock market when all the traders are dead. Weather is no longer chaotic when you remove the atmosphere. Biological life can't help ASI in any meaningful way but it can waste resources by complicating things. The very existence of biological life is a waste of resources to an ASI that hasn't been programmed to preserve it. Any goal with "but don't kill everything" added is more complicated than the version without that stipulation (and that specific one only saves a single microbe). The simple goal is easier to accomplish, so free market competition guarantees we'll try it first. Biological life is not compatible with "make number go up" taken to its logical conclusion. There is no need for the ASI to be conscious or to have any equivalent of human emotions to make the number go up.
- etiam 1y ago
  You're not wrong exactly, but if the crappy cheater AI treats collapsing the system to degeneracy as a valid optimal solution to its prediction task, maybe we'd deserve to be wiped out for releasing it with apocalyptic-level powers and such an inferior objective. Fortunately there's a really simple solution to offer it, in just wiring measurement to "prediction" directly (perfect correspondence, and much lower effort than annihilating Life and removing the atmosphere). And I don't particularly believe a system like that can be a general problem solver, much less one that climbs to World-jeopardizing influence on its own.
  - mrob 1y ago
    I don't see how measurement helps. The ASI correctly calculates that collapsing the system maximizes its reward function before it measures the result. We already see degenerate solutions in toy models, e.g. playing Tetris forever by leaving the game paused. The real world has many more degrees of freedom. It's unreasonable to think an inferior intelligence can predict and patch all the exploits on its first attempt (and we only get the one).

jacknews 1y ago

Perhaps there is a limit to intelligence, and it is close to human level - already highly intellient humans start to show some problems. In that case, the concern will be how fast it runs.