4 ms·

Now I'm curious. What was wrong with that paper?

• bigbillheck 5y ago
  The central conceit of the paper (https://arxiv.org/abs/1806.06850 https://arxiv.org/abs/1806.06850) is the "NN<->PR correspondence" from section 6, which argues that since nice nonlinear activation functions like tanh can be arbitrarily well-approximated by polynomials, then the whole stack of convolutional layers can be arbitrarily well-approximated by polynomials, and that means the whole network is just a polynomial. And sure, ```technically''' that's true but there are good reasons that numerics aren't just all polynomials all the time (check out Lloyd Trefethen's 'Approximation Theory and Approximation Practice', for example). tanh is nice and smooth but its Taylor series doesn't converge outside a small region, and the whole reason you'd use tanh in the first place is because you'll have things outside that region. And tanh is probably the best case for Matloff's argument! ReLU has that corner, and I don't even want to think about approximating softmax with a polynomial. The blog has comments by, among others, Radford Neal and Kenghagho which I find compelling.