Is the relevant model here simply a sum of 2nd order terms? It's widely known to practitioners vanilla Neural Nets don't do very well on small datasets because

Is the relevant model here simply a sum of 2nd order terms? It's widely known to practitioners vanilla Neural Nets don't do very well on small datasets because of overfitting issues -- which can probably be corrected with very careful regularization and parameter optimization, but where generally it's best to just use simpler methods. Polynomial regression is one of those, but I'm not sure how it compares to other methods such as random forests and KNN variations. Those methods don't scale very well, in particular PR (if I'm interpreting correctly) doesn't scale, because depth is essential to solving complex problems with computational efficiency. The researchers conveniently refer to those more complex tasks as a 'work in progress'... when they are what NNs are mostly used for (i.e. it starts getting interesting with CIFAR-10). That said, I would be curious to see performance of deeper polynomial regression networks. They become almost identical to NNs then, but maybe some methods of PR could be extended to give an edge in some cases (e.g. perhaps better regularization techniques than dropout/etc).