11 ms·
Most RL agents are 'jumpy' or jitter a lot, because it makes no difference to the reward, and where it does make a difference, that tends to only matter close t
by gwern 3y ago
Most RL agents are 'jumpy' or jitter a lot, because it makes no difference to the reward, and where it does make a difference, that tends to only matter close to convergence where slight speedups are all that's left. If you want to reduce that, you have to reward-shape it to penalize excess movement. (Which is relevant in robotics, where 'jitter' can be very bad for the machinery in a way not reflected in the simple tasks' reward functions.)
- anyfoo 3y agoCase in point: Watch an actual Mario speed running world record video, and the human players there are jumping around a lot while running around flat sections with no obstacles. As I understand, the jumping (at least in this case) does nothing to Mario's horizontal speed at all, so they basically just do it for "fun". The difference is that the human players know that they're doing something inconsequential for variety, while the fuzzer has no "idea" (i.e. no concept) that the jumping does not matter, or even that it is "jumping". More generally, it does not know nor care that this particular part of the input is irrelevant. It just found an input that works, and sometimes that happens to include a useless jump.
- gwern 3y agoOr they might be doing it just to keep their fingers 'warm'. I recall during AlphaStar, there were a lot of questions about why the human pro APMs were so high when it seemed like a lot of them were null ops, and apparently the answer is that they do that just to keep their fingers going and reduce reflex reaction time. (As you can imagine, this would be very bad for imitation learning...)