4 ms·

> Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! If that was your takeaway, then you missed the point.

> Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable. > As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.

Jach 4y ago
Your argument that it is actually impossible is not convincing, I think my example of a pitfall trap in 4 is sufficient to demonstrate that. More precisely, if we grant that any AGI no matter how intelligent and agent-y is still subject to physics and logic, it doesn't seem to be an actually impossible problem, just very hard, and maybe not solvable before time runs out. If we don't grant physics and logic, very well, but Fermi considerations raise a puzzle about why we're even here to ask the question. Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.) You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29 https://wiki.lesswrong.com/wiki/Free_will_%28solution%29
- 13years 4y ago
  > I think my example of a pitfall trap in 4 is sufficient to demonstrate that It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test. Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment. > Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components We are already having challenges in this area. Example... “The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output” from - https://arxiv.org/pdf/2302.03494.pdf https://arxiv.org/pdf/2302.03494.pdf
  - Jach 4y ago
    "ASI" is still limited by physics and logic. Or if not, we're just more screwed, maybe. I suppose we're in agreement that in the long run (maybe longer than we think with some really cool stuff like Ems in between...), the likely scenario is we're screwed one way or another? I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, logic is logic independent of intelligence. Similarly, I'm very sure that no one else and no superintelligent AGI can ever reproduce the original string I used to produce the sha512 hash of a5136df53e73c4fdcde80d9ba41f2526c6e144e344c85a355f518b161f5b5171a6de831908ec32f6a6e503ae055219027d8e9890517977bbcd16a6cd9f36c858 (perhaps they'll find a collision). (Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.) Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.
    - 13years 4y ago
      > I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views. > Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason. This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.

Jach 4y ago

Your argument that it is actually impossible is not convincing, I think my example of a pitfall trap in 4 is sufficient to demonstrate that. More precisely, if we grant that any AGI no matter how intelligent and agent-y is still subject to physics and logic, it doesn't seem to be an actually impossible problem, just very hard, and maybe not solvable before time runs out. If we don't grant physics and logic, very well, but Fermi considerations raise a puzzle about why we're even here to ask the question. Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.) You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29 https://wiki.lesswrong.com/wiki/Free_will_%28solution%29

13years 4y ago
> I think my example of a pitfall trap in 4 is sufficient to demonstrate that It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test. Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment. > Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components We are already having challenges in this area. Example... “The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output” from - https://arxiv.org/pdf/2302.03494.pdf https://arxiv.org/pdf/2302.03494.pdf
- Jach 4y ago
  "ASI" is still limited by physics and logic. Or if not, we're just more screwed, maybe. I suppose we're in agreement that in the long run (maybe longer than we think with some really cool stuff like Ems in between...), the likely scenario is we're screwed one way or another? I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, logic is logic independent of intelligence. Similarly, I'm very sure that no one else and no superintelligent AGI can ever reproduce the original string I used to produce the sha512 hash of a5136df53e73c4fdcde80d9ba41f2526c6e144e344c85a355f518b161f5b5171a6de831908ec32f6a6e503ae055219027d8e9890517977bbcd16a6cd9f36c858 (perhaps they'll find a collision). (Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason.) Yes, there's "difficulty comprehending" current deep learning models, put modestly. There's also interesting work in interpretability/explainability going on to work on that. Nothing there says "impossible". However, even granting that it is actually impossible to explain and reason about current AI methods to the degree we need to in order to create chains of logic for alignment purposes, that just says that it's impossible to reason about an AGI produced using such methods, if it's even possible to produce an AGI that way. It says nothing about it being impossible to reason about an AGI produced using different methods.
  - 13years 4y ago
    > I'm just glad people are working on trying to discover the chains of logic that let us not be so screwed, hard as that problem may be. I don't see any paradox there that makes the problem impossible, I understand you don't perceive the paradox. Nonetheless, my point is rather that when a paradox presents an impossible problem, then something about our premise is not understood. Or we need to find an entirely different concept to solve our problems. For example, an entirely different concept of intelligence not modelled strictly on how we perceive our brains. In summary, I think we are trying to solve the wrong problem. I might write something more depth at some point on that to explain my views. > Incidentally there are many secure systems and schemes that have not fallen. But you'd probably write them off as too narrow for some reason. This is precisely my point. We can build what is logically infallible to only be subverted by different means. If you can't hack the system, trick a human to give you access. This is how every such system has been defeated.