I am not sure at the moment. I guess the main problem of ES in a complex non-deterministic environment is that it would average over multiple local minima that

I am not sure at the moment. I guess the main problem of ES in a complex non-deterministic environment is that it would average over multiple local minima that occur in one generation which would result in a non-optimum solution. There are policy search methods that address this problem (e.g. VIPS https://scholar.google.com/citations?view_op=view_citation&hl=de&user=GL360kMAAAAJ&citation_for_view=GL360kMAAAAJ:IjCSPb-OGe4C https://scholar.google.com/citations?view_op=view_citation&h... , hierarchichal REPS https://scholar.google.com/citations?view_op=view_citation&hl=en&user=PxgVi0cAAAAJ&citation_for_view=PxgVi0cAAAAJ:u5HHmVD_uO8C https://scholar.google.com/citations?view_op=view_citation&h...). Temporal credit assignment is another problem: the policy is updated after a full episode and there is no way to use the information which step was responsible for which reward. Policy search usually works well if the value function is very complex and the optimal policy is simple.