>If I am being pedantic, "x^2" are still nonlinear though yes, and this is where this paper should have been stopped, with no ill intention towards the authors

>If I am being pedantic, "x^2" are still nonlinear though yes, and this is where this paper should have been stopped, with no ill intention towards the authors. Polynomial networks have gotten good results (my colleague https://scholar.google.com/citations?user=1bU041kAAAAJ https://scholar.google.com/citations?user=1bU041kAAAAJ has done extensive work on them) and there have been multiple papers studying multiplicative interactions and the effects of feature engineering, dozens of small tweaks on activation functions, not to start with the NAS papers automating the whole process. But higher numbers get a paper in I guess