4 ms·

> deep learning-based solutions lack mathematical elegance and offer very little interpretability of the found solution or understanding of the underlying pheno

> deep learning-based solutions lack mathematical elegance and offer very little interpretability of the found solution or understanding of the underlying phenomena I disagree. Elegance and interpretability comes with one big flaw: assumption, lots of them. While deep learning based methods assume few or not. If they are so elegant why would them underperform? Probably it is because the problem we are trying to solve doesn't follow the assumptions, like convexity, etc. In that sense, one can say deep learning methods are even more elegant, because they can work end-2-end. In that sense, NN can be a blessing, because now we got the answer, we don't need to assume anything, but just to DECODE it.

backpropaganda 9y ago
> because they can work end-2-end Agree. Not just end-2-end, but the same model can be applied to many different tasks. DeepMind's recent papers make progress in image generation, audio generation, language modeling & machine translation using the same basic construct of a masked dilated shifted convolutions.
hatmatrix 9y ago
Well here's an argument for interpretability: how do you know when your model will fail? One tradeoff might be that with deep learning, it's unlikely to fail, but you won't know when it will. With conventional methods, you may not get as good of a performance in comparison, but have a better sense of the domain of applicability because you have a better understanding of the feature space (for instance, you can determine similarity of new data in this feature space when it arrives to determine whether it is similar to past data your model has been trained on).
- throw_away_777 9y ago
  You can do the same with deep learning, your model will fail when the train data and test data are systematically different. This isn't about understanding the model, it is about understanding the data. Understanding the data is much more important than interpreting the model. Systematic uncertainty is a problem for all models.
  - hatmatrix 9y ago
    Yes - but how do you know when your test data and training data are different in the way that the model cares? With conventional methods, you can assess similarity by distances in the feature space of the model, or you have a physical understanding of why the model works than you have a better intuition about the differences in data that are important.
js8 9y ago
There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assumptions. No doubt that DNNs are subject to the same theorem, the only question is, how does their hypothesis space looks like? Does anyone have idea about this? I suspect we don't really know what the DNNs assumptions are.
- yorwba 9y ago
  The DNNs assumptions are "The true model can be fit by this network architecture". If you use convolutions, then an additional assumption is translation invariance.
  - js8 9y ago
    > The DNNs assumptions are "The true model can be fit by this network architecture" Yeah, but that's tautological. What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? Or more importantly, what functions it cannot learn? Or at least, how many bits are assumed and how many bits have to be learned? I think that is what bothers people about DNNs. I personally think they are sometimes even inefficient - we are learning them much more parameters (bits) than the actual hypothesis space requires.
    - yorwba 9y ago
      > What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? For a two-layer architecture with ReLU activations and n units in the hidden layer, this is the set of piecewise linear continuous functions with n kinks.
      - Klockan 9y ago
        How many of those do you need to tell the difference between cats and dogs in random internet pictures?
- 9y ago

backpropaganda 9y ago

> because they can work end-2-end Agree. Not just end-2-end, but the same model can be applied to many different tasks. DeepMind's recent papers make progress in image generation, audio generation, language modeling & machine translation using the same basic construct of a masked dilated shifted convolutions.

hatmatrix 9y ago

Well here's an argument for interpretability: how do you know when your model will fail? One tradeoff might be that with deep learning, it's unlikely to fail, but you won't know when it will. With conventional methods, you may not get as good of a performance in comparison, but have a better sense of the domain of applicability because you have a better understanding of the feature space (for instance, you can determine similarity of new data in this feature space when it arrives to determine whether it is similar to past data your model has been trained on).

throw_away_777 9y ago
You can do the same with deep learning, your model will fail when the train data and test data are systematically different. This isn't about understanding the model, it is about understanding the data. Understanding the data is much more important than interpreting the model. Systematic uncertainty is a problem for all models.
- hatmatrix 9y ago
  Yes - but how do you know when your test data and training data are different in the way that the model cares? With conventional methods, you can assess similarity by distances in the feature space of the model, or you have a physical understanding of why the model works than you have a better intuition about the differences in data that are important.

js8 9y ago

There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assumptions. No doubt that DNNs are subject to the same theorem, the only question is, how does their hypothesis space looks like? Does anyone have idea about this? I suspect we don't really know what the DNNs assumptions are.

yorwba 9y ago
The DNNs assumptions are "The true model can be fit by this network architecture". If you use convolutions, then an additional assumption is translation invariance.
- js8 9y ago
  > The DNNs assumptions are "The true model can be fit by this network architecture" Yeah, but that's tautological. What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? Or more importantly, what functions it cannot learn? Or at least, how many bits are assumed and how many bits have to be learned? I think that is what bothers people about DNNs. I personally think they are sometimes even inefficient - we are learning them much more parameters (bits) than the actual hypothesis space requires.
  - yorwba 9y ago
    > What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? For a two-layer architecture with ReLU activations and n units in the hidden layer, this is the set of piecewise linear continuous functions with n kinks.
    - Klockan 9y ago
      How many of those do you need to tell the difference between cats and dogs in random internet pictures?
9y ago