4 ms·

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones), would you care to elaborate?

p1esk 6y ago
Same here, I’m not sure what he’s trying to say.
wnoise 6y ago
While they can be thought of as stacked regression, it's only logistic regression with one particular non-linearity. And for many non-linearities you'll have a hard time usefully interpreting them as a regression.
- unishark 6y ago
  I think the common ones have statistical interpretations (that predate deep learning by a lot). Perhaps the one for the rectified linear unit is pretty obscure. But as I understand it, the statistics concept is called the "Tobit" model. It's meaning is not so obscure though, just a prediction that can be non-negative only, which is pretty common like a mass or energy.
mhh__ 6y ago
To get the neural network to do something useful you need to formalize some way of training it too?
woko 6y ago
I think this post means the same as this tweet: > Transformers are a special case of Graph Neural Networks. This may be obvious to some. https://twitter.com/OriolVinyalsML/status/1233783593626951681 https://twitter.com/OriolVinyalsML/status/123378359362695168...
iflp 6y ago
It depends on what kind of understanding you want to achieve. It can be helpful to think of DNN as approximating the corresponding infinitely-wide versions. Depending on how you deal with certain scaling, they then act like a linear filter of the error signal in function space, or for single-hidden-layer networks at least, an interacting particle system. In both cases you can understand the convergence of gradient descent training using these analogies, although gaps from real-world practice exist.

4 ms·

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones

by cjauvin 6y ago

I'm genuinely a bit surprised by that, that was always my high-level understanding of what the essence of neural networks was (at least feedforward vanilla ones), would you care to elaborate?

p1esk 6y ago
Same here, I’m not sure what he’s trying to say.
wnoise 6y ago
While they can be thought of as stacked regression, it's only logistic regression with one particular non-linearity. And for many non-linearities you'll have a hard time usefully interpreting them as a regression.
- unishark 6y ago
  I think the common ones have statistical interpretations (that predate deep learning by a lot). Perhaps the one for the rectified linear unit is pretty obscure. But as I understand it, the statistics concept is called the "Tobit" model. It's meaning is not so obscure though, just a prediction that can be non-negative only, which is pretty common like a mass or energy.
mhh__ 6y ago
To get the neural network to do something useful you need to formalize some way of training it too?
woko 6y ago
I think this post means the same as this tweet: > Transformers are a special case of Graph Neural Networks. This may be obvious to some. https://twitter.com/OriolVinyalsML/status/1233783593626951681 https://twitter.com/OriolVinyalsML/status/123378359362695168...
iflp 6y ago
It depends on what kind of understanding you want to achieve. It can be helpful to think of DNN as approximating the corresponding infinitely-wide versions. Depending on how you deal with certain scaling, they then act like a linear filter of the error signal in function space, or for single-hidden-layer networks at least, an interacting particle system. In both cases you can understand the convergence of gradient descent training using these analogies, although gaps from real-world practice exist.