3 ms·
I think you are hiding a key assumption - that an AGI must be capable of performing optimally on a task with Archimedean measure. This is a very strong assumpti
by TTPrograms 6y ago
I think you are hiding a key assumption - that an AGI must be capable of performing optimally on a task with Archimedean measure. This is a very strong assumption - performing eps-close to optimal may certainly fall in the definition of AGI, and this might be achieved by a non-Archimedean approximation. Defining AGI as performing optimally on any set of tasks is problematic from a computational theory perspective in general - even with real reward signals.
Furthermore, infinite rewards are not compatible with human behavior. Humans never optimize for a single event at infinite expense w.r.t. other goals.
- Animats 6y agoThat an AGI must be capable of performing optimally on a task with Archimedean measure. If that's correct, this may be just one of those many problems where the optimal solution is far harder than a near-optimal solution. Examples include linear programming and the traveling salesman problem, where a true optimum is NP-hard to find, but you can get very close with far less work.
- xamuel 6y agoHi, thanks for looking at my paper. I do not assume that an AGI must be capable of performing optimally on all tasks in general--indeed, that's quite impossible. When measuring the performance of RL agents, one must come up with some way of aggregating performance across many environments, but that's beside the point of this paper. The point of this paper is that if you're forced to use real numbers as rewards, you can't even communicate all environments to the agent without misleading the agent. Whether the agent could perform well or poorly in the environments once communicated, is beside the point.