3 ms·

The core observation here is that NN nonlinearities can be approximated by polynomials. And sure, that's true, but especially with low degree polynomials the ap

The core observation here is that NN nonlinearities can be approximated by polynomials. And sure, that's true, but especially with low degree polynomials the approximation might suck, or have very different properties from the actual function in question. In particular, RELU activation doesn't explode towards infinity the way a high degree polynomial activation would. OTOH, no one cares what the nonlinearity actually is so long as the net trains, and there's a lot of effort to keep layer inputs in a neighborhood near zero (batch normalization), so polynomial explosion may not be such an issue. Feel like I would like a more serious comparison of their model's results with best of class NNs. I'm suspicious that their NN character detector was basically failing...

contravariant 8y ago
Well this is getting in some rather unfamiliar territory to me, but a function like max(0,x) could be considered a polynomial in the max-plus algebra [1]. In this algebra the values kind of behave like the base-b log of a real number, with b taken to infinity [2]. If you replace b by a really big number you're just dealing with regular polynomials. In this sense I think you could consider a big neural network with RELU activation a (really big) polynomial. This is about the extent of my knowledge on the subject though, for more info search for tropical geometry [3]. [1]: https://en.wikipedia.org/wiki/Max-plus_algebra https://en.wikipedia.org/wiki/Max-plus_algebra [2]: https://mathoverflow.net/questions/83624/why-tropical-geometry https://mathoverflow.net/questions/83624/why-tropical-geomet... [3]: https://en.wikipedia.org/wiki/Tropical_geometry https://en.wikipedia.org/wiki/Tropical_geometry
- sdenton4 8y ago
  Totally. I'm unsure, though, how much people have looked at regression in tropical geometry, though... There's likely not a whole lot of results to draw on. (Similarly, I looked real hard at hyperplane arrangements arising from relu activations at one point, and found the hyper plane arrangement literature to be pay much completely useless for statistical questions... Though it's still quite beautiful!)
- rors 8y ago
  I wrote my thesis on tropical geometry for machine translation[1]. You can use it to train a a linear model for structured prediction, which used to be the dominant model before RNNs. [1]: http://www.aclweb.org/anthology/N15-1041 http://www.aclweb.org/anthology/N15-1041
deleted 8y ago
[deleted]
geezerjay 8y ago
> In particular, RELU activation doesn't explode towards infinity the way a high degree polynomial activation would. Isn't the ReLU a piecewise linear polynomial? Moreocer, if you convolve piecewise linear polynomials you get a higher-degree piecewise polynomial, typically referred to as a spline.

contravariant 8y ago

Well this is getting in some rather unfamiliar territory to me, but a function like max(0,x) could be considered a polynomial in the max-plus algebra [1]. In this algebra the values kind of behave like the base-b log of a real number, with b taken to infinity [2]. If you replace b by a really big number you're just dealing with regular polynomials. In this sense I think you could consider a big neural network with RELU activation a (really big) polynomial. This is about the extent of my knowledge on the subject though, for more info search for tropical geometry [3]. [1]: https://en.wikipedia.org/wiki/Max-plus_algebra https://en.wikipedia.org/wiki/Max-plus_algebra [2]: https://mathoverflow.net/questions/83624/why-tropical-geometry https://mathoverflow.net/questions/83624/why-tropical-geomet... [3]: https://en.wikipedia.org/wiki/Tropical_geometry https://en.wikipedia.org/wiki/Tropical_geometry

sdenton4 8y ago
Totally. I'm unsure, though, how much people have looked at regression in tropical geometry, though... There's likely not a whole lot of results to draw on. (Similarly, I looked real hard at hyperplane arrangements arising from relu activations at one point, and found the hyper plane arrangement literature to be pay much completely useless for statistical questions... Though it's still quite beautiful!)
rors 8y ago
I wrote my thesis on tropical geometry for machine translation[1]. You can use it to train a a linear model for structured prediction, which used to be the dominant model before RNNs. [1]: http://www.aclweb.org/anthology/N15-1041 http://www.aclweb.org/anthology/N15-1041

deleted 8y ago

[deleted]

geezerjay 8y ago

> In particular, RELU activation doesn't explode towards infinity the way a high degree polynomial activation would. Isn't the ReLU a piecewise linear polynomial? Moreocer, if you convolve piecewise linear polynomials you get a higher-degree piecewise polynomial, typically referred to as a spline.