3 ms·

In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decis

by panic 12y ago

In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decision, to run off with it? In other words, why restrict ourselves to self-aware agents?

Eliezer 12y ago
1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)
- JoshTriplett 12y ago
  Seems like it would help to reformulate Parfit's Hitchhiker not in terms of observation and imprecise prediction of the agent making the decision (such as "facial microexpressions") but in terms of perfect predictability (the predictor has access to the agent's source). In which case your own self-awareness can only be an advantage, since otherwise you're voluntarily choosing to not know something that the predictor does know. You should never regret having access to additional information, since you can always behave identically to an agent without that information. With a perfect predictor, the hitchhiker problem seems simpler. Even if the agent participating in the hitchhiker problem is intentionally not self-aware (e.g. it was constructed by an agent tasked with submitting an agent to participate in the hitchhiker problem), the predictor can predict the actual behavior of that agent even if the agent can't, even if the actual behavior of the agent would be to "honestly" say that it intends to pay in the city and not decide otherwise until it reaches the city. So a non-self-aware program that would refuse to pay in the city would still die stranded in the desert. (In other words, just because you're intentionally non-self-aware doesn't prevent other agents from being you-aware.) In the imperfect predictor version, such as with facial microexpressions that indicate lying, "not self-aware" seems equivalent to "self-aware but trained to not show the facial microexpressions that indicate lying"; both are cheats that ignore the purpose of the problem, and the problem can be reformulated to eliminate both such cheats.

by panic 12y ago

Eliezer 12y ago

1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)

JoshTriplett 12y ago
Seems like it would help to reformulate Parfit's Hitchhiker not in terms of observation and imprecise prediction of the agent making the decision (such as "facial microexpressions") but in terms of perfect predictability (the predictor has access to the agent's source). In which case your own self-awareness can only be an advantage, since otherwise you're voluntarily choosing to not know something that the predictor does know. You should never regret having access to additional information, since you can always behave identically to an agent without that information. With a perfect predictor, the hitchhiker problem seems simpler. Even if the agent participating in the hitchhiker problem is intentionally not self-aware (e.g. it was constructed by an agent tasked with submitting an agent to participate in the hitchhiker problem), the predictor can predict the actual behavior of that agent even if the agent can't, even if the actual behavior of the agent would be to "honestly" say that it intends to pay in the city and not decide otherwise until it reaches the city. So a non-self-aware program that would refuse to pay in the city would still die stranded in the desert. (In other words, just because you're intentionally non-self-aware doesn't prevent other agents from being you-aware.) In the imperfect predictor version, such as with facial microexpressions that indicate lying, "not self-aware" seems equivalent to "self-aware but trained to not show the facial microexpressions that indicate lying"; both are cheats that ignore the purpose of the problem, and the problem can be reformulated to eliminate both such cheats.