3 ms·

Can learn arbitrary functions says nothing about efficiency or any practical considerations. A vanilla neural network can't learn a sine function or anything pe

Can learn arbitrary functions says nothing about efficiency or any practical considerations. A vanilla neural network can't learn a sine function or anything periodic, for inputs outside the range of the training data. Re the question of other methods, Jeffrey Hinton proposed what I think is called a "forward-forward" alternative to back propagation somewhat recently. I don't think it's been proven to be better in any way, but it shows people are searching for new methods.

seydor 4y ago
> says nothing about efficiency or any practical considerations Sure it does, it's practical for pretty much any problem we want to solve, in fact i can't think of something it has not been shown to be practical about. Gradient descent is pretty straightforward and the most efficient generic method we have, even though it's not proven to be the fastest. Forward-forward is proposed as an alternative that is biologically plausible, not as an efficient alternative (it isnt)
- snovv_crash 4y ago
  If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used for NN because we initialize all the parameters to random values, so the assumption of the JtJ approximating the Hessian (required by LM) doesn't hold, since residuals are too high. I suspect there might also be something about the learning rate effectively being too high when using quadratic descent. Even for large problems there are techniques like LBFGS which converge a lot faster than gradient descent, that don't require O(N^2) memory on the number of parameters.
- slabity 4y ago
  Why are you responding to only parts of comments and ignoring the context around it? It just makes it look like strawmen arguments. > Can learn arbitrary functions says nothing about efficiency or any practical considerations. This is absolutely true. A turing machine can perform any computation, but they are not really practical for anything outside of an academic exercise. > A vanilla neural network can't learn a sine function or anything periodic, for inputs outside the range of the training data. This is also true. A vanilla neural network isn't that useful outside of very basic tasks with a lot of training data. It's only when you start changing their shapes and using specialized techniques to fit the problem that they become practical. Even a simple task like classifying MNIST digits is done far more efficiently using a basic CNN than a vanilla NN.
agnosticmantis 4y ago
It’s funny that the masters (Hinton, LeCun, et al) are constantly looking beyond the orthodoxy while the disciples worship the masters’ creations and fight any and all criticism. Wondering where I saw that pattern before…
moffkalast 4y ago
Efficiency is a function of the underlying hardware. Most graphics shaders would be comically inefficient if not for graphical accelerators. With this explosion likely ending up with an inference accelerator in every device, it should make it practical and efficient even if it's not the best approach.

seydor 4y ago

> says nothing about efficiency or any practical considerations Sure it does, it's practical for pretty much any problem we want to solve, in fact i can't think of something it has not been shown to be practical about. Gradient descent is pretty straightforward and the most efficient generic method we have, even though it's not proven to be the fastest. Forward-forward is proposed as an alternative that is biologically plausible, not as an efficient alternative (it isnt)

snovv_crash 4y ago
If you have a smaller number of parameters, then quadratic descent (eg. Levenberg Marquardt) is a lot faster for most problems. Gradient descent is only used for NN because we initialize all the parameters to random values, so the assumption of the JtJ approximating the Hessian (required by LM) doesn't hold, since residuals are too high. I suspect there might also be something about the learning rate effectively being too high when using quadratic descent. Even for large problems there are techniques like LBFGS which converge a lot faster than gradient descent, that don't require O(N^2) memory on the number of parameters.
slabity 4y ago
Why are you responding to only parts of comments and ignoring the context around it? It just makes it look like strawmen arguments. > Can learn arbitrary functions says nothing about efficiency or any practical considerations. This is absolutely true. A turing machine can perform any computation, but they are not really practical for anything outside of an academic exercise. > A vanilla neural network can't learn a sine function or anything periodic, for inputs outside the range of the training data. This is also true. A vanilla neural network isn't that useful outside of very basic tasks with a lot of training data. It's only when you start changing their shapes and using specialized techniques to fit the problem that they become practical. Even a simple task like classifying MNIST digits is done far more efficiently using a basic CNN than a vanilla NN.

agnosticmantis 4y ago

It’s funny that the masters (Hinton, LeCun, et al) are constantly looking beyond the orthodoxy while the disciples worship the masters’ creations and fight any and all criticism. Wondering where I saw that pattern before…

moffkalast 4y ago

Efficiency is a function of the underlying hardware. Most graphics shaders would be comically inefficient if not for graphical accelerators. With this explosion likely ending up with an inference accelerator in every device, it should make it practical and efficient even if it's not the best approach.