None of these experiments actually do anything to show feature learning - if this is the claim, I would like to see a transfer learning experiment. I would be

None of these experiments actually do anything to show feature learning - if this is the claim, I would like to see a transfer learning experiment. I would be surprised if this works well, since they can't jointly optimize their layers (so you can't just use ImageNet to induce a good representation). Not quite clear why we should think that trees will turn out to be inherently cheaper that a DNN with similar accuracy, unless perhaps the model structure encodes a prior which matches the distribution of the problem?