4 ms·

Timeless Decision Theory [pdf]

61 points by anacleto 12y ago

DennisP 12y ago
Most of the argument I've seen for this involves some very theoretical and unlikely circumstances. Can anybody boil down (a) the decision process that leads to cooperating in the one-shot prisoners' dilemma, and (b) the reason an agent would prefer to use that decision process? That would be a remarkable result. I confess I haven't read the whole 100-page paper, but I've scanned it and I'm having trouble finding an algorithm concrete enough to be expressed in code.
- Eliezer 12y ago
  http://arxiv.org/abs/1401.5577 http://arxiv.org/abs/1401.5577
- judk 12y ago
  http://lesswrong.com/lw/bxi/hofstadters_superrationality/ http://lesswrong.com/lw/bxi/hofstadters_superrationality/ http://en.m.wikipedia.org/wiki/Superrationality http://en.m.wikipedia.org/wiki/Superrationality
comboy 12y ago
[2010]
Moshe_Silnorin 12y ago
I believe the folks at MIRI spend most their time on Wei Dai's Updateless Decision Theory now. MIRI is strange. Five years ago, I was convinced they were very misguided or cranks, but I've been reading some of their logic stuff and much of it is pretty neat. I also attended a talk by Yudkowsky at MIT and it was also very good. I'm not sure what to think now.
- Eliezer 12y ago
  I affirm that UDT has replaced TDT. Updateless Decision Theory is by Wei Dai and Vladimir Nesov with major contributions from several others.
  - maaku 12y ago
    Why?
aspirin 12y ago
There is something wery odd about Newcomb’s Problem. If the Predictor can predict my choice with 100% accuracy, it's not really a choice, is it? Free will does not exist in this scenario, and therefore I cannot meaningfully choose between one or both boxes. The question is nonsensical.
- mziel 12y ago
  Also if you include a mixed (randomized) strategies the postulated Newcomb's problem is idiotic, because the other player is not an oracle. That's why they included a dirty fix that "if the Predictor predicts that the player will choose randomly, then box B will contain nothing".
  - Houshalter 12y ago
    Invoking nondeterminism doesn't make the thought experiment easier, just more complicated. From http://www.scottaaronson.com/blog/?p=30 http://www.scottaaronson.com/blog/?p=30 : >(Incidentally, don’t imagine you can wiggle out of this by basing your decision on a coin flip! For suppose the Predictor predicts you’ll open only the first box with probability p. Then he’ll put the $1,000,000 in that box with the same probability p. So your expected payoff is 1,000,000p2 + 1,001,000p(1-p) + 1,000(1-p)2 = 1,000,000p + 1,000(1-p), and you’re stuck with the same paradox as before.) Even if you have "free will" and your actions are semi-random, that's not different than flipping a coin to make your decisions.
- danbruc 12y ago
  I see it in a similar way and think about it in the following two steps. First convince the predictor that you will either choose AB or B - and he will always predict exactly this - then actually choose either AB or B, but both choices are not necessarily the same. Of the four possibilities you can safely ignore AB-B because it is worse then unproblematic AB-AB. If you want the million you have to convince the predictor that you will choose B and that means you have to either do B-B or B-AB. But you can only perform B-AB if you manage to fool the predictor and especially you can not do B-AB in a deterministic universe without free will (let's ignore that you don't even have control over the first step in that scenario). B is definitely the right choice in the first step and then changing to AB is the best choice in the second step, but whether you can actually do this depends on your stance on free will and the like.
miguelrochefort 12y ago
Let's say a person is faced with Newcomb's Problem every day, and its decision is made public every time. If the person has always taken B, then the Predictor will probably guess that the person will take B again, and therefore place $1,000,000 in B. By always choosing B, the person "wins". If the person doesn't always take B, then the Predictor probably can't guess what the person will take, and B will be empty. By taking A and B, the person can only get $1,000. The mechanism of Trust is exactly this. Whenever you're faced with a question, lying (to your advantage) will always increase the odds of a better outcome (like taking both A and B). However, that trick doesn't scale, and you can't use it anymore once a lie is discovered, once trust is lost.
Eliezer 12y ago
This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pdf http://intelligence.org/files/ProblemClassDominance.pdf * http://arxiv.org/abs/1401.5577 http://arxiv.org/abs/1401.5577 (The original TDT manuscript occurred via pleas for 'something, anything to be published' that were satisfied by cleaning up an unfinished book draft and publishing it as-was.)
- panic 12y ago
  In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decision, to run off with it? In other words, why restrict ourselves to self-aware agents?
  - Eliezer 12y ago
    1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)
jonathankoren 12y ago
Never mention http://rationalwiki.org/wiki/Roko's_basilisk http://rationalwiki.org/wiki/Roko's_basilisk
- deleted 12y ago
  [deleted]

61 points by anacleto 12y ago

Most of the argument I've seen for this involves some very theoretical and unlikely circumstances. Can anybody boil down (a) the decision process that leads to cooperating in the one-shot prisoners' dilemma, and (b) the reason an agent would prefer to use that decision process? That would be a remarkable result. I confess I haven't read the whole 100-page paper, but I've scanned it and I'm having trouble finding an algorithm concrete enough to be expressed in code.

Eliezer 12y ago
http://arxiv.org/abs/1401.5577 http://arxiv.org/abs/1401.5577
judk 12y ago
http://lesswrong.com/lw/bxi/hofstadters_superrationality/ http://lesswrong.com/lw/bxi/hofstadters_superrationality/ http://en.m.wikipedia.org/wiki/Superrationality http://en.m.wikipedia.org/wiki/Superrationality

comboy 12y ago

[2010]

Moshe_Silnorin 12y ago

I believe the folks at MIRI spend most their time on Wei Dai's Updateless Decision Theory now. MIRI is strange. Five years ago, I was convinced they were very misguided or cranks, but I've been reading some of their logic stuff and much of it is pretty neat. I also attended a talk by Yudkowsky at MIT and it was also very good. I'm not sure what to think now.

Eliezer 12y ago
I affirm that UDT has replaced TDT. Updateless Decision Theory is by Wei Dai and Vladimir Nesov with major contributions from several others.
- maaku 12y ago
  Why?

aspirin 12y ago

There is something wery odd about Newcomb’s Problem. If the Predictor can predict my choice with 100% accuracy, it's not really a choice, is it? Free will does not exist in this scenario, and therefore I cannot meaningfully choose between one or both boxes. The question is nonsensical.

mziel 12y ago
Also if you include a mixed (randomized) strategies the postulated Newcomb's problem is idiotic, because the other player is not an oracle. That's why they included a dirty fix that "if the Predictor predicts that the player will choose randomly, then box B will contain nothing".
- Houshalter 12y ago
  Invoking nondeterminism doesn't make the thought experiment easier, just more complicated. From http://www.scottaaronson.com/blog/?p=30 http://www.scottaaronson.com/blog/?p=30 : >(Incidentally, don’t imagine you can wiggle out of this by basing your decision on a coin flip! For suppose the Predictor predicts you’ll open only the first box with probability p. Then he’ll put the $1,000,000 in that box with the same probability p. So your expected payoff is 1,000,000p2 + 1,001,000p(1-p) + 1,000(1-p)2 = 1,000,000p + 1,000(1-p), and you’re stuck with the same paradox as before.) Even if you have "free will" and your actions are semi-random, that's not different than flipping a coin to make your decisions.
danbruc 12y ago
I see it in a similar way and think about it in the following two steps. First convince the predictor that you will either choose AB or B - and he will always predict exactly this - then actually choose either AB or B, but both choices are not necessarily the same. Of the four possibilities you can safely ignore AB-B because it is worse then unproblematic AB-AB. If you want the million you have to convince the predictor that you will choose B and that means you have to either do B-B or B-AB. But you can only perform B-AB if you manage to fool the predictor and especially you can not do B-AB in a deterministic universe without free will (let's ignore that you don't even have control over the first step in that scenario). B is definitely the right choice in the first step and then changing to AB is the best choice in the second step, but whether you can actually do this depends on your stance on free will and the like.

miguelrochefort 12y ago

Let's say a person is faced with Newcomb's Problem every day, and its decision is made public every time. If the person has always taken B, then the Predictor will probably guess that the person will take B again, and therefore place $1,000,000 in B. By always choosing B, the person "wins". If the person doesn't always take B, then the Predictor probably can't guess what the person will take, and B will be empty. By taking A and B, the person can only get $1,000. The mechanism of Trust is exactly this. Whenever you're faced with a question, lying (to your advantage) will always increase the odds of a better outcome (like taking both A and B). However, that trick doesn't scale, and you can't use it anymore once a lie is discovered, once trust is lost.

Eliezer 12y ago

This is a relatively old manuscript, and I would recommend instead reading, or at least first reading: * http://intelligence.org/files/ProblemClassDominance.pdf http://intelligence.org/files/ProblemClassDominance.pdf * http://arxiv.org/abs/1401.5577 http://arxiv.org/abs/1401.5577 (The original TDT manuscript occurred via pleas for 'something, anything to be published' that were satisfied by cleaning up an unfinished book draft and publishing it as-was.)

panic 12y ago
In the "Parfit's Hitchhiker" problem, wouldn't it improve the agent's outcome to believe it will give Paul the gas money but, when actually faced with the decision, to run off with it? In other words, why restrict ourselves to self-aware agents?
- Eliezer 12y ago
  1) Even if Paul can't predict you perfectly, it may be worth exploiting a probabilistic relation. 2) Can Paul see that you're not very self-aware? If Paul can't predict you at all, why does he believe that he can? 3) One of the applications of logical decision theories (TDT/UDT/etc) is between advanced agents or computer-based agents that (a) are very smart and might well be able to predict each other better than humans can and/or (b) might be able to look at each other's source code. 4) One of the other major applications is for the reflective consistency of self-modifying agents. The sort of self-modifying agents we're concerned with aren't human and can look directly at their own code; we should expect advanced forms of such agents to be correspondingly self-aware. 5) Many existing analyses, e.g. of Nash equilibria, assume that you know enough about the other agent to believe that, e.g., they will also be looking for a Nash equilibrium. 6) With that said, if you assume sufficiently great ignorance to break or prevent exploitation of any logical correlations, even correlations of the sort involved in "I know I try to maximize my utility" or "This agent and I are mutually ignorant of each other, and therefore will gravitate to a Nash equilibrium if there exists a unique Nash equilibrium", then you can end up outside the applicable domain of logical decision theories. (This may well be the case for actual humans! We can't read each other's source code, couldn't understand the brains if we saw them, don't know which decision theory the other person actually believes in, etcetera.)

jonathankoren 12y ago

Never mention http://rationalwiki.org/wiki/Roko's_basilisk http://rationalwiki.org/wiki/Roko's_basilisk

deleted 12y ago
[deleted]