3 ms·

This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pd

by Eliezer 12y ago

This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pdf http://intelligence.org/files/ProblemClassDominance.pdf * http://arxiv.org/abs/1401.5577 http://arxiv.org/abs/1401.5577 (The original TDT manuscript occurred via pleas for 'something, anything to be published' that were satisfied by cleaning up an unfinished book draft and publishing it as-was.)

panic 12y ago
In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decision, to run off with it? In other words, why restrict ourselves to self-aware agents?
- Eliezer 12y ago
  1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)
  - JoshTriplett 12y ago
    Seems like it would help to reformulate Parfit's Hitchhiker not in terms of observation and imprecise prediction of the agent making the decision (such as "facial microexpressions") but in terms of perfect predictability (the predictor has access to the agent's source). In which case your own self-awareness can only be an advantage, since otherwise you're voluntarily choosing to not know something that the predictor does know. You should never regret having access to additional information, since you can always behave identically to an agent without that information. With a perfect predictor, the hitchhiker problem seems simpler. Even if the agent participating in the hitchhiker problem is intentionally not self-aware (e.g. it was constructed by an agent tasked with submitting an agent to participate in the hitchhiker problem), the predictor can predict the actual behavior of that agent even if the agent can't, even if the actual behavior of the agent would be to "honestly" say that it intends to pay in the city and not decide otherwise until it reaches the city. So a non-self-aware program that would refuse to pay in the city would still die stranded in the desert. (In other words, just because you're intentionally non-self-aware doesn't prevent other agents from being you-aware.) In the imperfect predictor version, such as with facial microexpressions that indicate lying, "not self-aware" seems equivalent to "self-aware but trained to not show the facial microexpressions that indicate lying"; both are cheats that ignore the purpose of the problem, and the problem can be reformulated to eliminate both such cheats.
JoshTriplett 12y ago
Is there a canonical list of "remaining open problems" that need solving in order to construct a useful AI (or rather, a more-powerful-than-human optimization process), other than the construction and validation of a sufficiently robust decision theory?
- Eliezer 12y ago
  Producing this is my major current writing project. Meanwhile, http://intelligence.org/files/TechnicalAgenda.pdf http://intelligence.org/files/TechnicalAgenda.pdf
- jimrandomh 12y ago
  In addition to the document Eliezer linked to, check out the Future of Life Institute's Research Priorities (http://futureoflife.org/static/data/documents/research_priorities.pdf http://futureoflife.org/static/data/documents/research_prior...) and Survey of Research Questions (http://futureoflife.org/static/data/documents/research_survey.pdf http://futureoflife.org/static/data/documents/research_surve...). Note that these are focused on how to make AGI safe and beneficial, not on how to make AGI come sooner.

by Eliezer 12y ago

panic 12y ago

In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decision, to run off with it? In other words, why restrict ourselves to self-aware agents?

Eliezer 12y ago
1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)
- JoshTriplett 12y ago
  Seems like it would help to reformulate Parfit's Hitchhiker not in terms of observation and imprecise prediction of the agent making the decision (such as "facial microexpressions") but in terms of perfect predictability (the predictor has access to the agent's source). In which case your own self-awareness can only be an advantage, since otherwise you're voluntarily choosing to not know something that the predictor does know. You should never regret having access to additional information, since you can always behave identically to an agent without that information. With a perfect predictor, the hitchhiker problem seems simpler. Even if the agent participating in the hitchhiker problem is intentionally not self-aware (e.g. it was constructed by an agent tasked with submitting an agent to participate in the hitchhiker problem), the predictor can predict the actual behavior of that agent even if the agent can't, even if the actual behavior of the agent would be to "honestly" say that it intends to pay in the city and not decide otherwise until it reaches the city. So a non-self-aware program that would refuse to pay in the city would still die stranded in the desert. (In other words, just because you're intentionally non-self-aware doesn't prevent other agents from being you-aware.) In the imperfect predictor version, such as with facial microexpressions that indicate lying, "not self-aware" seems equivalent to "self-aware but trained to not show the facial microexpressions that indicate lying"; both are cheats that ignore the purpose of the problem, and the problem can be reformulated to eliminate both such cheats.

JoshTriplett 12y ago

Is there a canonical list of "remaining open problems" that need solving in order to construct a useful AI (or rather, a more-powerful-than-human optimization process), other than the construction and validation of a sufficiently robust decision theory?

Eliezer 12y ago
Producing this is my major current writing project. Meanwhile, http://intelligence.org/files/TechnicalAgenda.pdf http://intelligence.org/files/TechnicalAgenda.pdf
jimrandomh 12y ago
In addition to the document Eliezer linked to, check out the Future of Life Institute's Research Priorities (http://futureoflife.org/static/data/documents/research_priorities.pdf http://futureoflife.org/static/data/documents/research_prior...) and Survey of Research Questions (http://futureoflife.org/static/data/documents/research_survey.pdf http://futureoflife.org/static/data/documents/research_surve...). Note that these are focused on how to make AGI safe and beneficial, not on how to make AGI come sooner.