3 ms·

> says nothing about efficiency or any practical considerations Sure it does, it's practical for pretty much any problem we want to solve, in fact i can't thin

by seydor 4y ago

> says nothing about efficiency or any practical considerations Sure it does, it's practical for pretty much any problem we want to solve, in fact i can't think of something it has not been shown to be practical about. Gradient descent is pretty straightforward and the most efficient generic method we have, even though it's not proven to be the fastest. Forward-forward is proposed as an alternative that is biologically plausible, not as an efficient alternative (it isnt)

snovv_crash 4y ago
If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used for NN because we initialize all the parameters to random values, so the assumption of the JtJ approximating the Hessian (required by LM) doesn't hold, since residuals are too high. I suspect there might also be something about the learning rate effectively being too high when using quadratic descent. Even for large problems there are techniques like LBFGS which converge a lot faster than gradient descent, that don't require O(N^2) memory on the number of parameters.
slabity 4y ago
Why are you responding to only parts of comments and ignoring the context around it? It just makes it look like strawmen arguments. > Can learn arbitrary functions says nothing about efficiency or any practical considerations. This is absolutely true. A turing machine can perform any computation, but they are not really practical for anything outside of an academic exercise. > A vanilla neural network can't learn a sine function or anything periodic, for inputs outside the range of the training data. This is also true. A vanilla neural network isn't that useful outside of very basic tasks with a lot of training data. It's only when you start changing their shapes and using specialized techniques to fit the problem that they become practical. Even a simple task like classifying MNIST digits is done far more efficiently using a basic CNN than a vanilla NN.

by seydor 4y ago

snovv_crash 4y ago

If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used for NN because we initialize all the parameters to random values, so the assumption of the JtJ approximating the Hessian (required by LM) doesn't hold, since residuals are too high. I suspect there might also be something about the learning rate effectively being too high when using quadratic descent. Even for large problems there are techniques like LBFGS which converge a lot faster than gradient descent, that don't require O(N^2) memory on the number of parameters.

slabity 4y ago

Why are you responding to only parts of comments and ignoring the context around it? It just makes it look like strawmen arguments. > Can learn arbitrary functions says nothing about efficiency or any practical considerations. This is absolutely true. A turing machine can perform any computation, but they are not really practical for anything outside of an academic exercise. > A vanilla neural network can't learn a sine function or anything periodic, for inputs outside the range of the training data. This is also true. A vanilla neural network isn't that useful outside of very basic tasks with a lot of training data. It's only when you start changing their shapes and using specialized techniques to fit the problem that they become practical. Even a simple task like classifying MNIST digits is done far more efficiently using a basic CNN than a vanilla NN.