3 ms·

But can it show that it's actually running this correct algorithm to determine whether the statement about itself is probably true? There is always an external

But can it show that it's actually running this correct algorithm to determine whether the statement about itself is probably true? There is always an external context that runs the computation. A computation never exists in "abstract space", it always happens on a physical level. Any computation is only good if it runs in the context that it was designed for. What I mean is that the AI can never "know" if it's still following the "righteous goals", because a new environment may have grown around it, that "transforms" its inputs and outputs so that to the core machine it seems like nothing has changed, but actually its real-world effects have become very different. And at this point you could actually also say that this part of the environment has grown onto the AI machine and it now effectively has new goals that are incompatible with the ones designed into the core part of the machine. My point is, there is no isolated system, reasoning by itself in an isolated fashion. Subject and environment are not clearly divided.

posterboy 10y ago
> a new environment may have grown around it > at this point you could actually also say that this part of the environment has grown onto the AI machine No, you cannot, if the environment is part of the definition of the machine, as you put it, A computation never exists in "abstract space", because if the environment changed, you'd have a new machine and all bets are off. You had that right at first. Of course, that definition is unwieldy and we define interfaces as abstract and general as possible, to separate concerns. With involved machinery, that separation is not that clear cut, as their always two sides of the coin, to put it trivially.
tbt 10y ago
This is a good point. I think it'd be pretty interesting and useful to get a better grasp on when/whether/how a system can reason about itself as embedded in the world in a sane way.

posterboy 10y ago

> a new environment may have grown around it > at this point you could actually also say that this part of the environment has grown onto the AI machine No, you cannot, if the environment is part of the definition of the machine, as you put it, A computation never exists in "abstract space", because if the environment changed, you'd have a new machine and all bets are off. You had that right at first. Of course, that definition is unwieldy and we define interfaces as abstract and general as possible, to separate concerns. With involved machinery, that separation is not that clear cut, as their always two sides of the coin, to put it trivially.

tbt 10y ago

This is a good point. I think it'd be pretty interesting and useful to get a better grasp on when/whether/how a system can reason about itself as embedded in the world in a sane way.