Whenever I see an ML paper with surprising results I'm usually sceptical, as the results often come down to one of the following: 1. The assumptions don't gene

Whenever I see an ML paper with surprising results I'm usually sceptical, as the results often come down to one of the following: 1. The assumptions don't generalize 2. The observations don't imply the conclusions from the paper 3. Hyper-parameter shenanigans: It's usually possible to choose hyper-parameters where method A is better than B, even if generally B is better. In this case the lower-left part of figure 4 makes me suspicious, since forward mode seems to be better than reverse mode even on a per-iteration basis, which makes me suspect that they chose hyper-parameters where reverse mode has a disadvantage.