3 ms·
I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we los
by vn0m 10y ago
I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we losing some key ingredient for generalization? I mean could this algorithm be used in a fully non deterministic environment ? Like human vs machine?
- alfa02 10y agoI am not sure at the moment. I guess the main problem of ES in a complex non-deterministic environment is that it would average over multiple local minima that occur in one generation which would result in a non-optimum solution. There are policy search methods that address this problem (e.g. VIPS https://scholar.google.com/citations?view_op=view_citation&hl=de&user=GL360kMAAAAJ&citation_for_view=GL360kMAAAAJ:IjCSPb-OGe4C https://scholar.google.com/citations?view_op=view_citation&h... , hierarchichal REPS https://scholar.google.com/citations?view_op=view_citation&hl=en&user=PxgVi0cAAAAJ&citation_for_view=PxgVi0cAAAAJ:u5HHmVD_uO8C https://scholar.google.com/citations?view_op=view_citation&h...). Temporal credit assignment is another problem: the policy is updated after a full episode and there is no way to use the information which step was responsible for which reward. Policy search usually works well if the value function is very complex and the optimal policy is simple.
- dimatura 10y agoIt does not, at least not explicitly, which is a distinguishing feature of policy optimization algorithms. I don't think there's anything about this that makes it worse for non-deterministic problems. Note that the policy can still be stochastic, if desired (not sure if that's a good idea in general). A nice feature of black box policy optimization is that it's almost trivial to apply to either stochastic or deterministic problems.