Technically, it's not a polynomial as you can have nonlinear activation functions, such as the ReLU function. There's no polynomial that's equal to a network wi

Technically, it's not a polynomial as you can have nonlinear activation functions, such as the ReLU function. There's no polynomial that's equal to a network with ReLU activations (although, of course, a sufficiently large polynomial could come arbitrarily close). I would state that a neural network is a large, complicated, differentiable function, and the beauty of deep learning is that it turns out that by doing optimization that's derived from basic calculus, you can optimize this complicated function to do surprisingly useful things.