3 ms·

Your question and larger post don't present anything new to me, and I'm just a casual reader of the idea space (but I've been around since SL4). Some brief thou

by Jach 4y ago

Your question and larger post don't present anything new to me, and I'm just a casual reader of the idea space (but I've been around since SL4). Some brief thoughts: 1. Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! Too many (not any of the alignment people) have thought it's a trivial matter, or that Friendliness (the old word) will just "emerge", or that somehow being "superintelligent" will cause it to just smoke weed and play games all day like some super smart dorm room friend so there's nothing to worry about, or... 2. The idea of Friendliness or Aligned values is a difficult one purely from definitions alone, yes, let alone implementation, so that's part of what makes it such a hard problem. The idea of an AI tiling the solar system with smiley faces was once a response to a serious suggestion that one can just train the AI to optimize "human happiness". Imagine trying to formally model something like "If AI-Gandhi was offered a pill that made AI-Gandhi 2x smarter but also want to destroy the world, we know AI-Gandhi would not take such a pill because [...]" What's the [...]? A proof about its values? For actual Gandhi, we might as humans say "because he's Gandhi, and Gandhi does not want to destroy the world, and does not want to want to destroy the world". On the topic of the submission, I hope that any who take the course can by the end demonstrate many issues with things like the old and obsolete Coherent Extrapolated Volition proposal. 3. Containment is of course folly, you can look up the old AI Box arguments if you wanted more on that from back in the day. Hence the focus on Friendly (and more modernly, Aligned) AI. 4. I, an approximately +1 std dev IQ human, fully expect that it'd be no challenge for a -1 std dev IQ human to dig a pitfall trap that could not only contain me but kill me. Because gravity still applies even if I'm smarter than my adversary. Sure, an AGI will figure things out about physics that we don't know, and math too, but physics will still apply, I would bet that even a superintelligent AGI will never find a way to break the speed of light and travel the universe. But perhaps we're very wrong about special relativity. We'd have to be very wrong about everything if our proofs from first order logic are somehow awry. But that too highlights the level of challenge to the problem: we have to prove the AI is Friendly, with the reliability of first order logic. As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second. (Or to be fair, there are errors, but those are modeled accurately and through correction and redundancy (with multiple systems for things that go into space) we can drive the probability of a total failure as low as we please.)

13years 4y ago
> Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable. > As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.
- Jach 4y ago
  Your argument that it is actually impossible is not convincing, I think my example of a pitfall trap in 4 is sufficient to demonstrate that. More precisely, if we grant that any AGI no matter how intelligent and agent-y is still subject to physics and logic, it doesn't seem to be an actually impossible problem, just very hard, and maybe not solvable before time runs out. If we don't grant physics and logic, very well, but Fermi considerations raise a puzzle about why we're even here to ask the question. Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.) You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29 https://wiki.lesswrong.com/wiki/Free_will_%28solution%29
  - 13years 4y ago
    > I think my example of a pitfall trap in 4 is sufficient to demonstrate that It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test. Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment. > Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components We are already having challenges in this area. Example... “The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output” from - https://arxiv.org/pdf/2302.03494.pdf https://arxiv.org/pdf/2302.03494.pdf

by Jach 4y ago

13years 4y ago

> Congrats on recognizing that it's a hard (perhaps impossibly so, at least with the time left) problem! If that was your takeaway, then you missed the point. It was not an argument that it was a hard problem. It is precisely the argument that it is impossible as paradoxes are not solvable. > As a matter of fact humans can do things perfectly, your CPU is executing billions of instructions perfectly every second Yes, if you constrain the context small enough you get get the result you want. However, the concept of a ASI with free agency expands the context to a scope that we can't even reason about.

Jach 4y ago
Your argument that it is actually impossible is not convincing, I think my example of a pitfall trap in 4 is sufficient to demonstrate that. More precisely, if we grant that any AGI no matter how intelligent and agent-y is still subject to physics and logic, it doesn't seem to be an actually impossible problem, just very hard, and maybe not solvable before time runs out. If we don't grant physics and logic, very well, but Fermi considerations raise a puzzle about why we're even here to ask the question. Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components (as we do for large problems already). I think you'd have to give some sort of impossibility proof on what about the problem is possible to reason about and what is not possible to reason about to convince me. (Note: chaos theory won't necessarily help you here, because probabilistic reasoning exists as well as do() operations by us, and if it must be used the argument must explain why certain catastrophic errors cannot be modeled and driven to arbitrarily low values.) You might have some confused notions about free agency that if cleared up would resolve your other questions. This might be useful: https://wiki.lesswrong.com/wiki/Free_will_%28solution%29 https://wiki.lesswrong.com/wiki/Free_will_%28solution%29
- 13years 4y ago
  > I think my example of a pitfall trap in 4 is sufficient to demonstrate that It is an isolated narrow context. The problem with ASI is there is no such limitations. I addressed this point in the post, but another example would be the Kobayashi Maru Test. Or rather, there has been no system devised that humans have not found a vulnerability. Why would AGI be any different? Alignment/containment must also address the possibility of human exploit of the system as well. So we are back at the beginning in either case. Which means a human could still devise a trap for the AI to leverage its ability or modify its alignment. > Similarly I'm not convinced that the scope is broad enough that it really cannot be reasoned about at all, either in its entirety or by breaking it up into smaller components We are already having challenges in this area. Example... “The size and complexity of deep learning models, particularly language models, have increased to the point where even the creators have difficulty comprehending why their models make specific predictions. This lack of interpretability is a major concern, particularly in situations where individuals want to understand the reasoning behind a model’s output” from - https://arxiv.org/pdf/2302.03494.pdf https://arxiv.org/pdf/2302.03494.pdf