"Within the scope of developing reinforcement learning (RL) algorithms, the goal is to build agents that learn to achieve the given objective." This is the fun

"Within the scope of developing reinforcement learning (RL) algorithms, the goal is to build agents that learn to achieve the given objective." This is the fundamental problem. That is not the goal. As long as people believe that is the goal we will not advance. Put another way, "the given objective" to date has always been wrong. All of the desired behavior that we are trying to get machines to emulate is a tiny subset of the learned behaviors of the capable agents (us). The primary object we have is to survive. On a moment to moment basis our object is to fulfill a set of competing needs (hunger, tiredness, boredom etc). It is the interplay of these multiple objectives with an complex environment which necessitates the kind of intelligence that we have. And to be absolutely clear the answer to "How do we faithfully capture the human concept of a given task in a reward function?" is "You don't" and trying to do so is harmful to the research effort. The correct path is to evolve a reward system which functions to increase the frequencies of behaviors which meet the needs of agents in the moment and fulfill the general goal of survival. Then through a curriculum of environmental challenges* allow for those needs to only be met by accomplishing certain tasks which have value, or are of interest, to us. *Simulator fidelity / hacking will continue to be a problem, but it is orthogonal to the problem of RL. RL finds the bugs, you fix them.