6 ms·

I've only recently started reading about deep neural networks, and the thing that strikes me the most about the literature is the lack of mathematics. Open a N

I've only recently started reading about deep neural networks, and the thing that strikes me the most about the literature is the lack of mathematics. Open a NIPS paper from 2010 or so, and you'll see extremely dense mathematics: nonparametrics, variational approximation, sampling theory, riemannian geometry. But from my (admittedly small) sampling of the convnet / RNN literature there really doesn't seem to be much maths there. The typical paper seems to run along the lines of "We tried this, and it worked". I'm not sure whether there's anything to learn from this observation, but I think it's striking all the same.

iaw 10y ago
I think once the underpinnings were created via math it became questions about what network geometries to use. We don't have much math I'm aware of that can describe the capabilities of different network configurations from first principles. Even though we constructed the network it feels like we're back to the beginnings of science with this one. Let's change this and see what happens.
- soVeryTired 10y ago
  The thing is, the underpinnings have been there for decades. The convnet was invented in the '90s. The only thing that's changed is the availability of data and processing power.
  - ebalit 10y ago
    ConvNet was indeed invented a long time ago. But a lot of things have changed since then other than data and processing power. There have been new development in optimizers, non linearities, architectures, loss functions and theoretical understanding.
deleted 10y ago
[deleted]
habitue 10y ago
So what we're seeing is these fields pre-deep learning were mathematical disciplines making steady progress on well-understood foundations. Then deep learning came in, was exceptionally effective at problems that had been difficult to crack, and people shifted focus because it seems weird to be diddling around with incremental gains on techniques that are significantly less effective. What this created though, was a clean break from a field with a theoretical foundation like what we expect in CS usually, to an empirical foundation, like what we see in psychology, sociology, medicine etc. The theory will eventually catch up, but the fact that "it works" really trumps having a full understanding of why it works (at least for now).
- reader5000 10y ago
  Also, deep net problems are different than traditional CS because we can mathematically define when say a list of numbers is sorted but not when "this image looks photorealistic". Effectively, deep nets are good at tricking human visual cortex, so I'm not sure theres a deeper mathematical reason they work well, in the same way that the reason mp3 compression works so well is that they tested it against human hearing. But yes, all deep learning is based on gradient descent which is a greedy heuristic algorithm and it's not a priori apparent it should do anything interesting. But it does.
  - mistercow 10y ago
    > in the same way that the reason mp3 compression works so well But there are deeper reasons that mp3 works well, besides "tested against humans". It's just that those reasons are about psychoacoustics. The most obvious example is that humans are bad at discerning amplitude and phase differences at high frequencies. I expect that as we get deeper into the theory around why particular algorithms work, we'll find similar answers. We'll be able to say things like "when an image has the following statistical properties, we're likely to interpret it as a cat". Of course, right now we can trivially say that if by "the following statistical properties" we mean "activates this specifically trained network in a particular way", but the point is that we can probably boil that down into deeper (and simpler) statements about how human image recognition works. I can't find a link about it now, but I remember reading that for their Translate app, Google created a way to lossily compress their character recognition neural net. That kind of research seems likely to me to lead to a better understanding of how to boil down the insights that these learning algorithms are gleaning. This Two Minute Papers video has some thinking along the same lines: https://www.youtube.com/watch?v=ZBWTD2aNb_o https://www.youtube.com/watch?v=ZBWTD2aNb_o But I'd also point out that the human visual cortex is itself a bunch of models about the real world. It's gotten that way through a combination of evolution and learning. When engineers train neural networks to do tricks like TFA shows, those networks are effectively modeling the real world in two different ways: directly, and by proxy in modeling human visual systems. So the insights we can find by studying them are likely to tell us not only about our own vision, but about the world itself. Finally, it's worth noting that graphics algorithms have always been about human visual systems and the real physical world, as opposed to deep mathematical truths. Look at this paper for example: http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf The oversimplified plain English insight those researchers had is "For a photo that doesn't contain haze, you usually can find a fully saturated pixel within some reasonably small neighborhood of any point in the image." There's no deep mathematical insight there, but there is an important insight into the statistical nature of natural images – a fact about the physical world. And they use that insight to great effect: they can remove haze from photographs, and even use the haze as a way of estimating a depth map. So sure, while you're right that there probably aren't many deep mathematical insights to find from the results of deep learning algorithms (which is generally true of graphics algorithms anyway), I wouldn't gloss over the deep practical insights that they encode.
deleted 10y ago
[deleted]
ganfortran 10y ago
Too bad that Math heavy approaches just cannot compete.
- bluetwo 10y ago
  We have a winner.
  - ganfortran 10y ago
    Yes we do. In terms of spending time mathurbating what is fundamentally limited and incapable, I suggest we spend more time play with the shiny new toys that do fullfill its very own promises.
deleted 10y ago
[deleted]
empath75 10y ago
Wait until we have neural networks designing neural networks.
- deepnotderp 10y ago
  Been there, done that. Beats humans :)
- ganfortran 10y ago
  Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 Too bad, it is already there :P
- shreyassaxena 10y ago
  https://arxiv.org/abs/1606.02492 https://arxiv.org/abs/1606.02492 ^^ Disclaimer: I am the author
WhiteNoiz3 10y ago
Neural networks are very much math based (more so than I imagined when I first started reading about them), it's just the underlying math hasn't changed much - only the architectures and techniques.
visarga 10y ago
> I'm not sure whether there's anything to learn from this observation I think it is our duty to explore all the possibilities of neural networks. They are still not fully understood yet. Theory will catch up once we practically understand the beast. It might not be pure math, but it is a necessary exploration nonetheless.
kevinalexbrown 10y ago
A friend of mine from academia was considering going into industry, so went to some data science meetups. Someone was giving a presentation about convolutional networks, yet did not know what a convolution was. At first I was startled, but in the long run machine learning applications will be decided just as much on user experience and design features as on algorithmic choices. I'm not a web programmer, but I imagine few developers could remember the mathematics[0] of the sorting algorithms that are fundamental underpinnings of their work (if they ever learned them at all). Yet I'm not sure it matters, even to great developers. The same thing will probably ultimately be true of machine learning. Honestly, you need not know what a convolution is to build a perfectly usable convnet. (And ultimately you may not need to even build your own if you can use a nifty amazon API.) Whether NIPS should care or not is a separate story. It seems a little sad - I took all this hard pure math as an undergrad, and it doesn't seem to be important if all I'm doing is changing a few layer parameters (even if the change is ingenious). [0] mathematics as in proof of sorting, proof of bounds on space/time complexity, etc.
- ganfortran 10y ago
  Math needs to fulfill a purpose in order to succeed in INDUSTRY. A image recognition is first and foremost about recognizing an image, the math behind it is mystical, but the accuracy is measurable. If you want your knowledge to be hold as useful, then you need find a usage case for it. Simple as that. This is not only true for math, but for all the other techniques as well. Otherwise so-called knowledge is yet another self-indulgent toy, disconnected even further from being useful. Contrary to what OP states here, recent development of WGAN and LSGAN pretty much math driven, and it leads to very useful realworld extension to the original model, that improves it quite a bit.
pjreddie 10y ago
NIPS is more math/theory heavy as a conference while ICCV/CVPR/other vision conferences will tolerate less math and more pictures. You have probably been reading mostly vision papers but if you read more papers targeted at NIPS you will find more math. This looks to be an ICCV submission, whereas if you read the original GAN paper (a NIPS paper) there are theorems and proofs. Your observation is correct, you are just comparing different subsets of the field.
mrmaximus 10y ago
I think this is because at it's core, the math behind deep learning is pretty simple and there's not much of it. Linear algebra, some simple activation functions and gradient descent. Implementing a net from scratch in python is pretty concise. The simplicity-to-effectiveness ratio is what I find so interesting about deep learning.
dahart 10y ago
> The typical paper seems to run along the lines of "We tried this, and it worked". This sounds like science to me, is "we tried this, and it worked" something to shy away from for some reason? Of course, it's super important to publish "we tried this, and it didn't work." But that's another topic altogether... If less math really is a trend, do you see it as a bad thing? You didn't state it strongly, but you've hinted at that, and it seems like the replies assume that's what you're suggesting. I love math and mathy papers, and I would still welcome a trend toward less math in papers in return for more effort spent on making simple ideas plain and easily understood. But academics don't always operate that way. Math in papers is often used to obscure simple ideas, sometimes on purpose, and sometimes it's an indicator that the author doesn't understand the domain clearly enough but still wants to sound smart. Sometimes a paper really requires dense math, but not very often. Dense math almost always makes a paper more difficult to reproduce. Either way it is harder for even experts to evaluate the quality of dense math than of expository writing that strives for simplicity and clarity. Neural networks are really simple math under the hood, well understood algorithms and simple linear algebra, why not write great papers that work and don't re-hash the math but instead focus on clarity, reproducibility and results?
- soVeryTired 10y ago
  I agree with everything you've said. But my suspicion is that there's less maths in the papers because no-one knows how to use it to analyse neural networks properly. That means current research is operating via trial and error. One man's trial and error is another's blind search. Without maths to point researchers in the right direction, neural net research could easily hit a wall once the low-hanging fruit has been picked.

iaw 10y ago

I think once the underpinnings were created via math it became questions about what network geometries to use. We don't have much math I'm aware of that can describe the capabilities of different network configurations from first principles. Even though we constructed the network it feels like we're back to the beginnings of science with this one. Let's change this and see what happens.

soVeryTired 10y ago
The thing is, the underpinnings have been there for decades. The convnet was invented in the '90s. The only thing that's changed is the availability of data and processing power.
- ebalit 10y ago
  ConvNet was indeed invented a long time ago. But a lot of things have changed since then other than data and processing power. There have been new development in optimizers, non linearities, architectures, loss functions and theoretical understanding.

deleted 10y ago

[deleted]

habitue 10y ago

So what we're seeing is these fields pre-deep learning were mathematical disciplines making steady progress on well-understood foundations. Then deep learning came in, was exceptionally effective at problems that had been difficult to crack, and people shifted focus because it seems weird to be diddling around with incremental gains on techniques that are significantly less effective. What this created though, was a clean break from a field with a theoretical foundation like what we expect in CS usually, to an empirical foundation, like what we see in psychology, sociology, medicine etc. The theory will eventually catch up, but the fact that "it works" really trumps having a full understanding of why it works (at least for now).

reader5000 10y ago
Also, deep net problems are different than traditional CS because we can mathematically define when say a list of numbers is sorted but not when "this image looks photorealistic". Effectively, deep nets are good at tricking human visual cortex, so I'm not sure theres a deeper mathematical reason they work well, in the same way that the reason mp3 compression works so well is that they tested it against human hearing. But yes, all deep learning is based on gradient descent which is a greedy heuristic algorithm and it's not a priori apparent it should do anything interesting. But it does.
- mistercow 10y ago
  > in the same way that the reason mp3 compression works so well But there are deeper reasons that mp3 works well, besides "tested against humans". It's just that those reasons are about psychoacoustics. The most obvious example is that humans are bad at discerning amplitude and phase differences at high frequencies. I expect that as we get deeper into the theory around why particular algorithms work, we'll find similar answers. We'll be able to say things like "when an image has the following statistical properties, we're likely to interpret it as a cat". Of course, right now we can trivially say that if by "the following statistical properties" we mean "activates this specifically trained network in a particular way", but the point is that we can probably boil that down into deeper (and simpler) statements about how human image recognition works. I can't find a link about it now, but I remember reading that for their Translate app, Google created a way to lossily compress their character recognition neural net. That kind of research seems likely to me to lead to a better understanding of how to boil down the insights that these learning algorithms are gleaning. This Two Minute Papers video has some thinking along the same lines: https://www.youtube.com/watch?v=ZBWTD2aNb_o https://www.youtube.com/watch?v=ZBWTD2aNb_o But I'd also point out that the human visual cortex is itself a bunch of models about the real world. It's gotten that way through a combination of evolution and learning. When engineers train neural networks to do tricks like TFA shows, those networks are effectively modeling the real world in two different ways: directly, and by proxy in modeling human visual systems. So the insights we can find by studying them are likely to tell us not only about our own vision, but about the world itself. Finally, it's worth noting that graphics algorithms have always been about human visual systems and the real physical world, as opposed to deep mathematical truths. Look at this paper for example: http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf http://www.jiansun.org/papers/Dehaze_CVPR2009.pdf The oversimplified plain English insight those researchers had is "For a photo that doesn't contain haze, you usually can find a fully saturated pixel within some reasonably small neighborhood of any point in the image." There's no deep mathematical insight there, but there is an important insight into the statistical nature of natural images – a fact about the physical world. And they use that insight to great effect: they can remove haze from photographs, and even use the haze as a way of estimating a depth map. So sure, while you're right that there probably aren't many deep mathematical insights to find from the results of deep learning algorithms (which is generally true of graphics algorithms anyway), I wouldn't gloss over the deep practical insights that they encode.

deleted 10y ago

[deleted]

ganfortran 10y ago

Too bad that Math heavy approaches just cannot compete.

bluetwo 10y ago
We have a winner.
- ganfortran 10y ago
  Yes we do. In terms of spending time mathurbating what is fundamentally limited and incapable, I suggest we spend more time play with the shiny new toys that do fullfill its very own promises.

deleted 10y ago

[deleted]

empath75 10y ago

Wait until we have neural networks designing neural networks.

deepnotderp 10y ago
Been there, done that. Beats humans :)
ganfortran 10y ago
Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578 https://arxiv.org/abs/1611.01578 Too bad, it is already there :P
shreyassaxena 10y ago
https://arxiv.org/abs/1606.02492 https://arxiv.org/abs/1606.02492 ^^ Disclaimer: I am the author

WhiteNoiz3 10y ago

Neural networks are very much math based (more so than I imagined when I first started reading about them), it's just the underlying math hasn't changed much - only the architectures and techniques.

visarga 10y ago

> I'm not sure whether there's anything to learn from this observation I think it is our duty to explore all the possibilities of neural networks. They are still not fully understood yet. Theory will catch up once we practically understand the beast. It might not be pure math, but it is a necessary exploration nonetheless.

kevinalexbrown 10y ago

A friend of mine from academia was considering going into industry, so went to some data science meetups. Someone was giving a presentation about convolutional networks, yet did not know what a convolution was. At first I was startled, but in the long run machine learning applications will be decided just as much on user experience and design features as on algorithmic choices. I'm not a web programmer, but I imagine few developers could remember the mathematics[0] of the sorting algorithms that are fundamental underpinnings of their work (if they ever learned them at all). Yet I'm not sure it matters, even to great developers. The same thing will probably ultimately be true of machine learning. Honestly, you need not know what a convolution is to build a perfectly usable convnet. (And ultimately you may not need to even build your own if you can use a nifty amazon API.) Whether NIPS should care or not is a separate story. It seems a little sad - I took all this hard pure math as an undergrad, and it doesn't seem to be important if all I'm doing is changing a few layer parameters (even if the change is ingenious). [0] mathematics as in proof of sorting, proof of bounds on space/time complexity, etc.

ganfortran 10y ago
Math needs to fulfill a purpose in order to succeed in INDUSTRY. A image recognition is first and foremost about recognizing an image, the math behind it is mystical, but the accuracy is measurable. If you want your knowledge to be hold as useful, then you need find a usage case for it. Simple as that. This is not only true for math, but for all the other techniques as well. Otherwise so-called knowledge is yet another self-indulgent toy, disconnected even further from being useful. Contrary to what OP states here, recent development of WGAN and LSGAN pretty much math driven, and it leads to very useful realworld extension to the original model, that improves it quite a bit.

pjreddie 10y ago

NIPS is more math/theory heavy as a conference while ICCV/CVPR/other vision conferences will tolerate less math and more pictures. You have probably been reading mostly vision papers but if you read more papers targeted at NIPS you will find more math. This looks to be an ICCV submission, whereas if you read the original GAN paper (a NIPS paper) there are theorems and proofs. Your observation is correct, you are just comparing different subsets of the field.

mrmaximus 10y ago

I think this is because at it's core, the math behind deep learning is pretty simple and there's not much of it. Linear algebra, some simple activation functions and gradient descent. Implementing a net from scratch in python is pretty concise. The simplicity-to-effectiveness ratio is what I find so interesting about deep learning.

dahart 10y ago

> The typical paper seems to run along the lines of "We tried this, and it worked". This sounds like science to me, is "we tried this, and it worked" something to shy away from for some reason? Of course, it's super important to publish "we tried this, and it didn't work." But that's another topic altogether... If less math really is a trend, do you see it as a bad thing? You didn't state it strongly, but you've hinted at that, and it seems like the replies assume that's what you're suggesting. I love math and mathy papers, and I would still welcome a trend toward less math in papers in return for more effort spent on making simple ideas plain and easily understood. But academics don't always operate that way. Math in papers is often used to obscure simple ideas, sometimes on purpose, and sometimes it's an indicator that the author doesn't understand the domain clearly enough but still wants to sound smart. Sometimes a paper really requires dense math, but not very often. Dense math almost always makes a paper more difficult to reproduce. Either way it is harder for even experts to evaluate the quality of dense math than of expository writing that strives for simplicity and clarity. Neural networks are really simple math under the hood, well understood algorithms and simple linear algebra, why not write great papers that work and don't re-hash the math but instead focus on clarity, reproducibility and results?

soVeryTired 10y ago
I agree with everything you've said. But my suspicion is that there's less maths in the papers because no-one knows how to use it to analyse neural networks properly. That means current research is operating via trial and error. One man's trial and error is another's blind search. Without maths to point researchers in the right direction, neural net research could easily hit a wall once the low-hanging fruit has been picked.