4 ms·

Could someone please explain the explanation of equ. 18 in the appendix to the paper? It seems key, but isn't clear to me. > if one greedily maximize the sum o

Could someone please explain the explanation of equ. 18 in the appendix to the paper? It seems key, but isn't clear to me. > if one greedily maximize the sum of entropy and value with one-step look-ahead, then one obtains π˜ from π

skndr 9y ago
Hi Alex! I don't have the answer, but I just sent this over to Seth. Looking forward to seeing you all again in December!
- deleted 9y ago
  [deleted]
- AlexCoventry 9y ago
  Hey, Skander! Hope SF's treating you well.
jkabrg 9y ago
It's probably referring to policy iteration[1]. It works by iterating two steps: - Keep the policy $\pi$ fixed, and determine the $Q$ values for that policy - Keep the $Q$ value estimates fixed, and create a policy $\pi'$ such that -- $\pi'(s) = \arg\max_a Q(s,a)$ for the ordinary greedy algorithm, -- $\pi'(s,a) \propto \exp Q(s, a)$ if using greedy soft-max. [1] - Lecture 3 of http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html
clickok 9y ago
It wasn't clear to me either, and I really dislike when papers expect the reader to 'notice' some rather non-trivial manipulations. So I wrote up my unpacking of the equivalence: http://rl.ai/posts/max-entropy-rl-explanation.html http://rl.ai/posts/max-entropy-rl-explanation.html Please forgive the possibly screwy LaTeX, as I am in a battle with my static site generator. If you're just wondering about the meaning of (18), it claims that for a given state, the entropy of the original policy plus its expected Q-value is less than the entropy + expected Q-value of the π-tilde policy. Where they are using Q_{soft}^{\pi} to denote the state-action values (with entropy term) for an arbitrary policy. In (17) they introduce a new policy, \tilde{\pi} defined through the Q_{soft} values for some policy π, with the probability of taking action 'a' in state 's' proportional to the exponentiated Q(s,a).
- AlexCoventry 9y ago
  Thanks, this is really generous of you.

skndr 9y ago

Hi Alex! I don't have the answer, but I just sent this over to Seth. Looking forward to seeing you all again in December!

deleted 9y ago
[deleted]
AlexCoventry 9y ago
Hey, Skander! Hope SF's treating you well.

jkabrg 9y ago

It's probably referring to policy iteration[1]. It works by iterating two steps: - Keep the policy $\pi$ fixed, and determine the $Q$ values for that policy - Keep the $Q$ value estimates fixed, and create a policy $\pi'$ such that -- $\pi'(s) = \arg\max_a Q(s,a)$ for the ordinary greedy algorithm, -- $\pi'(s,a) \propto \exp Q(s, a)$ if using greedy soft-max. [1] - Lecture 3 of http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

clickok 9y ago

It wasn't clear to me either, and I really dislike when papers expect the reader to 'notice' some rather non-trivial manipulations. So I wrote up my unpacking of the equivalence: http://rl.ai/posts/max-entropy-rl-explanation.html http://rl.ai/posts/max-entropy-rl-explanation.html Please forgive the possibly screwy LaTeX, as I am in a battle with my static site generator. If you're just wondering about the meaning of (18), it claims that for a given state, the entropy of the original policy plus its expected Q-value is less than the entropy + expected Q-value of the π-tilde policy. Where they are using Q_{soft}^{\pi} to denote the state-action values (with entropy term) for an arbitrary policy. In (17) they introduce a new policy, \tilde{\pi} defined through the Q_{soft} values for some policy π, with the probability of taking action 'a' in state 's' proportional to the exponentiated Q(s,a).

AlexCoventry 9y ago
Thanks, this is really generous of you.