5 ms·

It seems like this can leave the reader with the wrong impression. Calculus really is "the mathematics of Newtonian physics". This is just "some mathematics tha

It seems like this can leave the reader with the wrong impression. Calculus really is "the mathematics of Newtonian physics". This is just "some mathematics that might help a bit in your intuitions of deep learning". IE, Deep learning is fundamentally just about getting the mathematically simple but complex and multi-layerd "neural networks" to do stuff. Training them, testing them and deploying them. There are many intuitions about these things but there's no complete theory - some intuitions involve mathematical analogies and simplifications while other involve "folk knowledge" or large scale experiments. And that's not saying folks giving math about deep learning aren't proving real things. It's just they characterizing the whole or even a substantial part of such systems. It's not surprising that a complex like a many-layered Relu network can't fully characterized or solved mathematically. You'd expect that of any arbitrarily complex algorithmic construct. Differential equations of many variables and arbitrary functions also can't have their solutions fully characterized.

conformist 5y ago
It seems like it aims at giving somebody who would like to get started doing theoretical research in the field some pointers and basic insights. I don't think it does a particularly bad job at this, in particular given that it will be a book chapter? The target audience are probably people who have had some exposure to Functional Analysis and the likes before.
jhrmnn 5y ago
There are a few works that try to put deep learning on some theoretical basis, I like this one, for example: https://arxiv.org/abs/1703.00810 https://arxiv.org/abs/1703.00810 This goes beyond mere intuition, but it is also still very far from a “complete theory”. I find it disappointing that so few people in deep learning work on the theoretical foundations.
- quibono 5y ago
  What are some subfields of mathematics that you would say are crucial for gaining a proper understanding of all the things related to deep learning (e.g. let's say the paper you linked)? Even though the theory isn't complete, I'm sure a grounding in certain fields of mathematics will be helpful.
  - iNic 5y ago
    This is always difficult to answer, and it will probably be a mixture of many, however I am currently following categorical approaches to machine learning. Category Theory is the area of mathematics that studies composable structures, i.e. like layers in a deep network. It is very abstract and was invented to solve problem in algebraic geometry, but has been fruitful in other areas as well.
    - convexity123 5y ago
      Could you give some favourite references, some use of category theory in ML which gives good results compared to standard approaches? Is there a group doing this in Zurich?
    - joe_the_user 5y ago
      That you this illustrates that the situation today is "take whatever math-stuff you have, throw it against neural networks and see what you get". IE, I'm pretty sure not much progress has been made with category theory and neural networks - but you might be the first. I've seen differential equations, Markov chains, differential geometry and other stuff. We might be in heady days before the "big breakthrough" is made. But these constructs might be inherently pathological (even then, non-pathological variants might be possible).
      - aborsy 5y ago
        It’s good for paper publication, and for public sector. Take an obscure area of math and throw it against latest trend. It’s a question if it’s useful.
        SubiculumCode 5y ago
        It can be useful for innovation in the aggregate.
  - 317070 5y ago
    Dynamical systems and chaos theory (especially for neural networks), information theory (especially for the paper linked), probability theory (especially the more foundational and axiomatic work)
- 0-_-0 5y ago
  Of the many "understanding neural networks" papers this is one of the few valuable ones.
- keithalewis 5y ago
  Agreed. Until we get to the point where there are theorems of the form, for example, "Given a problem satisfying conditions X, the optimal number of layers to minimize expected training time for data satisfying Y is Z", it is just stamp collecting.
fogof 5y ago
As a PhD student who sort of burned out on this type of research, I agree that the complexity of Neural Networks as a mathematical construct makes them very difficult to analyze. This might also have to do with Deep learning theory being a subset of learning theory which is subject to "No Free Lunch" [1], which means that you always have to be very careful not to try to prove something that turns out to be impossible. That being said, research on the Kernel regime is one of the very cool ideas, in my opinion, to gain traction in this field in the past few years. To summarize: "If you make a neural network wide enough, it gains the power to control its output on each individual input separately, and will begin to fit its training data perfectly". Of course, the real pleasure is in understanding all the mathematical details of this statement! [1] : https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem
- joe_the_user 5y ago
  I got my master's years ago so now I'm a strict amateur. That said, I don't think the "No free lunch theorem" is very "interesting". It's nearly tautological that no approximation method works for "any" function. The set of predictable/interesting/useful/"real-world" functions is going to have measure 0 compared to white noise so "any function" will basically look like white noise and can't be predicted. Approximating functions/sequences with vanishingly low Kolmogorov complexity is more interesting, impossible in general by Godel's theorem but what's the case "on average"? (depends on the choice process and so ill-defined but defining might be interesting). The kernel regime stuff looks interesting but I don't know it's relation to wide networks. Neural networks "tend to generalize well in the real world". That's a pretty fuzzy statement imo since "real world" is hardly defined but it's still what people experience and it's more useful to provide a more precise model where this works rather than a model where this doesn't work. Also, there's good theory on deep networks as universal well as theories of wide/shallow networks [1]. [1]: https://arxiv.org/abs/1901.02220 https://arxiv.org/abs/1901.02220
  - roenxi 5y ago
    > Neural networks "tend to generalize well in the real world". I've always interpreted that as "we've found an algorithm that could, given a foreseeable amount of computing power and maybe some tweaks, simulate human decision making". It isn't so much that neural networks can approximate the real world as they can approximate human perception of the real world.

conformist 5y ago

It seems like it aims at giving somebody who would like to get started doing theoretical research in the field some pointers and basic insights. I don't think it does a particularly bad job at this, in particular given that it will be a book chapter? The target audience are probably people who have had some exposure to Functional Analysis and the likes before.

jhrmnn 5y ago

There are a few works that try to put deep learning on some theoretical basis, I like this one, for example: https://arxiv.org/abs/1703.00810 https://arxiv.org/abs/1703.00810 This goes beyond mere intuition, but it is also still very far from a “complete theory”. I find it disappointing that so few people in deep learning work on the theoretical foundations.

quibono 5y ago
What are some subfields of mathematics that you would say are crucial for gaining a proper understanding of all the things related to deep learning (e.g. let's say the paper you linked)? Even though the theory isn't complete, I'm sure a grounding in certain fields of mathematics will be helpful.
- iNic 5y ago
  This is always difficult to answer, and it will probably be a mixture of many, however I am currently following categorical approaches to machine learning. Category Theory is the area of mathematics that studies composable structures, i.e. like layers in a deep network. It is very abstract and was invented to solve problem in algebraic geometry, but has been fruitful in other areas as well.
  - convexity123 5y ago
    Could you give some favourite references, some use of category theory in ML which gives good results compared to standard approaches? Is there a group doing this in Zurich?
  - joe_the_user 5y ago
    That you this illustrates that the situation today is "take whatever math-stuff you have, throw it against neural networks and see what you get". IE, I'm pretty sure not much progress has been made with category theory and neural networks - but you might be the first. I've seen differential equations, Markov chains, differential geometry and other stuff. We might be in heady days before the "big breakthrough" is made. But these constructs might be inherently pathological (even then, non-pathological variants might be possible).
    - aborsy 5y ago
      It’s good for paper publication, and for public sector. Take an obscure area of math and throw it against latest trend. It’s a question if it’s useful.
      - SubiculumCode 5y ago
        It can be useful for innovation in the aggregate.
- 317070 5y ago
  Dynamical systems and chaos theory (especially for neural networks), information theory (especially for the paper linked), probability theory (especially the more foundational and axiomatic work)
0-_-0 5y ago
Of the many "understanding neural networks" papers this is one of the few valuable ones.
keithalewis 5y ago
Agreed. Until we get to the point where there are theorems of the form, for example, "Given a problem satisfying conditions X, the optimal number of layers to minimize expected training time for data satisfying Y is Z", it is just stamp collecting.

fogof 5y ago

As a PhD student who sort of burned out on this type of research, I agree that the complexity of Neural Networks as a mathematical construct makes them very difficult to analyze. This might also have to do with Deep learning theory being a subset of learning theory which is subject to "No Free Lunch" [1], which means that you always have to be very careful not to try to prove something that turns out to be impossible. That being said, research on the Kernel regime is one of the very cool ideas, in my opinion, to gain traction in this field in the past few years. To summarize: "If you make a neural network wide enough, it gains the power to control its output on each individual input separately, and will begin to fit its training data perfectly". Of course, the real pleasure is in understanding all the mathematical details of this statement! [1] : https://en.wikipedia.org/wiki/No_free_lunch_theorem https://en.wikipedia.org/wiki/No_free_lunch_theorem

joe_the_user 5y ago
I got my master's years ago so now I'm a strict amateur. That said, I don't think the "No free lunch theorem" is very "interesting". It's nearly tautological that no approximation method works for "any" function. The set of predictable/interesting/useful/"real-world" functions is going to have measure 0 compared to white noise so "any function" will basically look like white noise and can't be predicted. Approximating functions/sequences with vanishingly low Kolmogorov complexity is more interesting, impossible in general by Godel's theorem but what's the case "on average"? (depends on the choice process and so ill-defined but defining might be interesting). The kernel regime stuff looks interesting but I don't know it's relation to wide networks. Neural networks "tend to generalize well in the real world". That's a pretty fuzzy statement imo since "real world" is hardly defined but it's still what people experience and it's more useful to provide a more precise model where this works rather than a model where this doesn't work. Also, there's good theory on deep networks as universal well as theories of wide/shallow networks [1]. [1]: https://arxiv.org/abs/1901.02220 https://arxiv.org/abs/1901.02220
- roenxi 5y ago
  > Neural networks "tend to generalize well in the real world". I've always interpreted that as "we've found an algorithm that could, given a foreseeable amount of computing power and maybe some tweaks, simulate human decision making". It isn't so much that neural networks can approximate the real world as they can approximate human perception of the real world.