3 ms·

> Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that

> Humans can and have contained human intelligence ASI is not human intelligence. As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem. > The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths. (Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)

RandomLensman 3y ago
Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
- JoshTriplett 3y ago
  > Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply". > Similar, it is unclear if humans could not contain superhuman intelligence. If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'". The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes. It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
  - RandomLensman 3y ago
    We do pretty much nothing to mitigate other, actual extinction level risks - why should AI be special given that its risk has an unknown probability and it could even be zero.
breuleux 3y ago
> As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. I'm fairly certain that what you describe is physically impossible. Organic brains may not be fully optimal, but they are not that many orders of magnitude off.
- Xirgil 3y ago
  Not only is such an entity possible, it's already here, the only difference is clock speed. Corportations (really any large organization devoted to some kind of intellectual work, or you could say human society as a whole) can be thought of as a superintelligent entity composed of biological processors running in parallel. They that can store, process, and create information at superhuman speeds compared to the individual. And given that our brains evolved restricted by the calories available to our ancient ancestors, the size of the human pelvis, and a multitude of other obsolete factors, that individual processing unit clearly far from optimal. But even if you just imagine the peak demonstrated potential of the human brain in the form of an AI process it's easy to see how that could be scaled to massively superhuman levels. Imagine a corporation that could recruit from a pool of candidates (only bound by their computing power) each with the intellect of John von Neumann, with a personality fine tuned for their role, that can think faster by increasing their clock speed, that can access, learn and communicate information near instantly, happily work 24/7 with zero complaint or fatigue, etc, etc. Imagine how much more efficient (how much more superintelligent) that company would be compared to its competition.
  - breuleux 3y ago
    The parent was positing, at a minimum, a 30 million fold increase in clock speed. This entails a proportional increase in energy consumption, which would likely destroy any thermal envelope the size of a brain. The only reason current processors run that fast is that there are very few of them: millions of synaptic calculations therefore have to be multiplexed into each, leading to an effective clock rate that's far closer to a human brain's than you would assume. As for your corporation example, I do not think the effectiveness of a corporation is necessarily bottlenecked by the number or intelligence of its employees. Notwithstanding the problem of coordinating many agents, there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two. The chaotic nature of reality also entails a fundamental difficulty in predicting complex systems: you can only think so far ahead before the expected deviation between your plan and reality becomes too large. You need a feedback loop where you test your designs against reality and adjust accordingly, and this also acts as a bottleneck on the effectiveness of intelligence. I'm not saying "superintelligent" AI couldn't be an order of magnitude better, mind you. I just think the upside is far, far less than the 7+ orders of magnitude the parent is talking about.

RandomLensman 3y ago

Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?

JoshTriplett 3y ago
> Even for your last example, two hypotheses need to be true: (1) such information exists, and (2) the AI has access to such information/can generate it. EDIT: actually at least three: (3) the human and/or the AI can apply that information. Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply". > Similar, it is unclear if humans could not contain superhuman intelligence. If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'". The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes. It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
- RandomLensman 3y ago
  We do pretty much nothing to mitigate other, actual extinction level risks - why should AI be special given that its risk has an unknown probability and it could even be zero.

breuleux 3y ago

> As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. I'm fairly certain that what you describe is physically impossible. Organic brains may not be fully optimal, but they are not that many orders of magnitude off.

Xirgil 3y ago
Not only is such an entity possible, it's already here, the only difference is clock speed. Corportations (really any large organization devoted to some kind of intellectual work, or you could say human society as a whole) can be thought of as a superintelligent entity composed of biological processors running in parallel. They that can store, process, and create information at superhuman speeds compared to the individual. And given that our brains evolved restricted by the calories available to our ancient ancestors, the size of the human pelvis, and a multitude of other obsolete factors, that individual processing unit clearly far from optimal. But even if you just imagine the peak demonstrated potential of the human brain in the form of an AI process it's easy to see how that could be scaled to massively superhuman levels. Imagine a corporation that could recruit from a pool of candidates (only bound by their computing power) each with the intellect of John von Neumann, with a personality fine tuned for their role, that can think faster by increasing their clock speed, that can access, learn and communicate information near instantly, happily work 24/7 with zero complaint or fatigue, etc, etc. Imagine how much more efficient (how much more superintelligent) that company would be compared to its competition.
- breuleux 3y ago
  The parent was positing, at a minimum, a 30 million fold increase in clock speed. This entails a proportional increase in energy consumption, which would likely destroy any thermal envelope the size of a brain. The only reason current processors run that fast is that there are very few of them: millions of synaptic calculations therefore have to be multiplexed into each, leading to an effective clock rate that's far closer to a human brain's than you would assume. As for your corporation example, I do not think the effectiveness of a corporation is necessarily bottlenecked by the number or intelligence of its employees. Notwithstanding the problem of coordinating many agents, there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two. The chaotic nature of reality also entails a fundamental difficulty in predicting complex systems: you can only think so far ahead before the expected deviation between your plan and reality becomes too large. You need a feedback loop where you test your designs against reality and adjust accordingly, and this also acts as a bottleneck on the effectiveness of intelligence. I'm not saying "superintelligent" AI couldn't be an order of magnitude better, mind you. I just think the upside is far, far less than the 7+ orders of magnitude the parent is talking about.