5 ms·

A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we ad

by fg6hr 7y ago

A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of GPUs that costs $10k/hour, feed it a dataset with 1 billion images and see what happens; but nobody can predict the behavior of training or convergence or accuracy based on the ML graph and data alone.

psv1 7y ago
Just want to point out that you're asking about "ML" but your questions are about neural networks/deep learning which is only a subset of machine learning. > is there any math behind ML at all? Yes. There is a lot of research in this area (some people argue it's excessive at the moment). You have the correct intuition that the answers aren't black and white all the time. For example, there are solid reasons to choose relu activations over tanh. Or to build certain types of network architectures for certain tasks. That doesn't mean that you can immediately calculate what would happen if you switch from one activation to another without running your network.
nightcracker 7y ago
There is a subfield that does serious mathematics, but their results are usually far removed from the state of the art stuff. Their results usually look like "a neural network with 1 hidden layer is a universal approximator" or "exponential expected convergence speed on linear relations for <X>".
- SpaceManNabs 7y ago
  Not necessarily true. The polynomial time for the escape of saddle points by using stochastic noise is pretty practical.
  - nightcracker 7y ago
    Sorry, my statement was too strong. I edited it to "usually".
  - p1esk 7y ago
    How in the world is this "pretty practical"? It would be if people used this theorem to come up with an idea of SGD, but that's not what happened. SGD appeared as a way to overcome the practical constraint of computing full GD. Not to mention that "polynomial time" is meaningless to any practitioner.
    - SpaceManNabs 7y ago
      it did come with a new online update for orthogonal tensor decomposition using higher order moments and with comments on NP-hardness for 4th order and higher. In addition, it came with tricks with how much noise to inject in certain situations. "How much noise do you need is enough to escape?" which is pretty practical
SpaceManNabs 7y ago
Yes. https://joanbruna.github.io/MathsDL-spring18/ https://joanbruna.github.io/MathsDL-spring18/ On Tanh and ReLus, the paper on SeLUs is pretty intense. ReLUs help with vanishing gradients to some degree on a larger degree of loss and activation functions, btw. edit: On your question about a billion data points and predicting behavior, we are getting there. http://www.offconvex.org/2017/03/30/GANs2/ http://www.offconvex.org/2017/03/30/GANs2/
publicrootkey 7y ago
If I understand your question correctly, this is exactly what Bayes' Theorem deals with. Namely that B must be true within some probability given that B is true represented as P(A | B).
- fg6hr 7y ago
  Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on? [Edit] Whether this model works is question. Sure, it can't recognize dogs or cats, but what if the dataset is 0-9 digits? Now it suddenly works, right? And works really well. But what's changed? Can we describe in mathematical terms what makes the 0-9 dataset so special? It'll probably work with A-J letters too, but what about hieroglyphs? The math I'm looking for would tell that E(T) is the error and on such model and such dataset, E(T)=exp(-T^2)+O(exp(-T^3)), according to such and such theorem; and according to another theorem, if the dataset is isomorphic to that manifold, E(T) can be improved to O(exp(-3T^2)).
  - improbable22 7y ago
    I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemistry (what shape is a sugar molecule? A DNA molecule?) vs trying out drugs (if you eat the bark of this tree, you don't get malaria! Let's refine that stuff). Both can be useful, but they are very far from a unified theory of how your body works.
    - fg6hr 7y ago
      ML is more like alchemy, I'd say: mixing components using intuition and experience, but without understanding what these components really are and why they work. In this analogy, AI is the recipe to make gold and the ML alchemists haven't invented nuclear physics yet.
jeeceebees 7y ago
Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The reason some losses allow GANs to converge in certain situations while others don't isn't a complete mystery, there is theory that supports this. Researchers designing new models are considering weak points in old approaches, identifying why they aren't working correctly, and proposing something new that solves a part of the problem. All of this is done by looking at the math behind all the operations in the network (or at least the parts relevant to a certain question). That nobody really knows how AI works is one of those myths told by the media. Just because the model weights aren't interprable doesn't mean we don't know why that model works well. It just takes quite a bit of maths knowledge to really understand state of the art models. All that knowledge is also easily packaged into modern frameworks that make it easy to use without a deep knowledge of why it works. All of this contributes to the feeling that nobody really knows what's going on, while in reality it's onky the majority of people that don't know what's going on ;)
- fg6hr 7y ago
  Actually, I read the batch norm paper and maybe I forgot important details, but it roughly went like this: "here we add a term `b` to make sure the mean values of Ax+b are zero and that will help us with convergence; ah, and here is a covariance matrix!", but no quantitative proofs about how much that convergence was helped. No, I intuitively agree that shifting the mean value to zero should help, but math taught me that there is a huge difference between a seemingly correct statement and its proof. The ML papers seem to just state these seemingly correct ideas without real, proof-backed understanding, why this works. In other words, ML is entirely about empirical results, peppered with math-like terminology. But don't take my blunt writing style personally. Let's take the simplest example: recognizing the grayscale 30x80 pictures with 0-9 digits. IIRC, this is called the MNIST example and can be done by my cat in 1 hour without prior knowledge. Let's choose the probably simplest model: 2400 inputs are fully connected with a 1024 vector that's fully connected with a 10 vector. And let's use relu at both steps. We know that this kinda works and converges quickly. In particular, after T steps we get error E(T) and E(1e6) < 0.03 (a random guess). Can you tell me how T and E will change if we add another layer: 2400->1024->1024->10, using the same relu? Same question, but now we replace relu with tanh: 2400->1024->10.
BiasRegularizer 7y ago
I'd like to chime in that Variational Autoencoder's theoretical framework is also quite useful in unsupervised learning tasks. It bridges variational inference methods (a powerful classical ML technique) with neural networks, thus allow very efficient representation, generalization and disentanglement.
m0zg 7y ago
There's plenty of math still. In order to not waste GPU-hours, you gotta have some intution as to what _not_ to do, because otherwise you'll be trying for a very long time and relying entirely on luck, which is combinatorially unfavorable and frustrating. That intuition is often grounded in math, mostly differential calculus and statistics. That said, there are a good number of techniques which nobody is sure why they work. I.e. batchnorm, which is used all the time. People have some theories and intuitions as to why it works, but nobody is sure why, nor is there any rigorous way to pick hyperparameters for it.
playing_colours 7y ago
There are maths and proves behind Neural Networks, but Machine Learning is an umbrella term. There are areas within it, maybe not that mainstream, that are also based on solid maths foundations. One of the most interesting and elegant examples , Topological Data Analysis that is based on Topology and utilises tools like persistent homology. They can be applied to image processing, classification, etc.
tel 7y ago
First up, what you seem to be talking about is Deep Learning, not Machine Learning in general. In more general ML there are many theorems, some also apply to DL. Also, the step of "shuffle around the ML graph using some intuition" involves gathering that intuition, which usually arises from a great deal of mathematical competence. A 3x3 conv kernel versus a 2x2 one can, for instance, be discussed in terms of Fourier theory and mathematical image processing, but areas with huge built-in theory. Things like replacing the activation function were initially studied anecdotally. People realized that in some settings one activation function or another would lead. Eventually, there was also theory showing that in large nets of stable configurations, there was serious interaction between the initialization method and the activation function and problems like poor backprop signal propagation were tackled theoretically and practically. Generally, the mystery comes from the vast parameterization of these DL models. They operate in a space that's very hard to generalize—large, finite spaces. Small finite spaces get treated exhaustively. Infinite spaces get treated asymptotically. Large finite spaces get bounded on either side by those methods. So yes, there might feel like there's a dearth of theory in DL when it comes to the large scale behavior of a general network. That can be super frustrating. At the same time, people are trying to push through and create more theory every day.

by fg6hr 7y ago

psv1 7y ago

Just want to point out that you're asking about "ML" but your questions are about neural networks/deep learning which is only a subset of machine learning. > is there any math behind ML at all? Yes. There is a lot of research in this area (some people argue it's excessive at the moment). You have the correct intuition that the answers aren't black and white all the time. For example, there are solid reasons to choose relu activations over tanh. Or to build certain types of network architectures for certain tasks. That doesn't mean that you can immediately calculate what would happen if you switch from one activation to another without running your network.

nightcracker 7y ago

There is a subfield that does serious mathematics, but their results are usually far removed from the state of the art stuff. Their results usually look like "a neural network with 1 hidden layer is a universal approximator" or "exponential expected convergence speed on linear relations for <X>".

SpaceManNabs 7y ago
Not necessarily true. The polynomial time for the escape of saddle points by using stochastic noise is pretty practical.
- nightcracker 7y ago
  Sorry, my statement was too strong. I edited it to "usually".
- p1esk 7y ago
  How in the world is this "pretty practical"? It would be if people used this theorem to come up with an idea of SGD, but that's not what happened. SGD appeared as a way to overcome the practical constraint of computing full GD. Not to mention that "polynomial time" is meaningless to any practitioner.
  - SpaceManNabs 7y ago
    it did come with a new online update for orthogonal tensor decomposition using higher order moments and with comments on NP-hardness for 4th order and higher. In addition, it came with tricks with how much noise to inject in certain situations. "How much noise do you need is enough to escape?" which is pretty practical

SpaceManNabs 7y ago

Yes. https://joanbruna.github.io/MathsDL-spring18/ https://joanbruna.github.io/MathsDL-spring18/ On Tanh and ReLus, the paper on SeLUs is pretty intense. ReLUs help with vanishing gradients to some degree on a larger degree of loss and activation functions, btw. edit: On your question about a billion data points and predicting behavior, we are getting there. http://www.offconvex.org/2017/03/30/GANs2/ http://www.offconvex.org/2017/03/30/GANs2/

publicrootkey 7y ago

If I understand your question correctly, this is exactly what Bayes' Theorem deals with. Namely that B must be true within some probability given that B is true represented as P(A | B).

fg6hr 7y ago
Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on? [Edit] Whether this model works is question. Sure, it can't recognize dogs or cats, but what if the dataset is 0-9 digits? Now it suddenly works, right? And works really well. But what's changed? Can we describe in mathematical terms what makes the 0-9 dataset so special? It'll probably work with A-J letters too, but what about hieroglyphs? The math I'm looking for would tell that E(T) is the error and on such model and such dataset, E(T)=exp(-T^2)+O(exp(-T^3)), according to such and such theorem; and according to another theorem, if the dataset is isomorphic to that manifold, E(T) can be improved to O(exp(-3T^2)).
- improbable22 7y ago
  I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemistry (what shape is a sugar molecule? A DNA molecule?) vs trying out drugs (if you eat the bark of this tree, you don't get malaria! Let's refine that stuff). Both can be useful, but they are very far from a unified theory of how your body works.
  - fg6hr 7y ago
    ML is more like alchemy, I'd say: mixing components using intuition and experience, but without understanding what these components really are and why they work. In this analogy, AI is the recipe to make gold and the ML alchemists haven't invented nuclear physics yet.

jeeceebees 7y ago

Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The reason some losses allow GANs to converge in certain situations while others don't isn't a complete mystery, there is theory that supports this. Researchers designing new models are considering weak points in old approaches, identifying why they aren't working correctly, and proposing something new that solves a part of the problem. All of this is done by looking at the math behind all the operations in the network (or at least the parts relevant to a certain question). That nobody really knows how AI works is one of those myths told by the media. Just because the model weights aren't interprable doesn't mean we don't know why that model works well. It just takes quite a bit of maths knowledge to really understand state of the art models. All that knowledge is also easily packaged into modern frameworks that make it easy to use without a deep knowledge of why it works. All of this contributes to the feeling that nobody really knows what's going on, while in reality it's onky the majority of people that don't know what's going on ;)

fg6hr 7y ago
Actually, I read the batch norm paper and maybe I forgot important details, but it roughly went like this: "here we add a term `b` to make sure the mean values of Ax+b are zero and that will help us with convergence; ah, and here is a covariance matrix!", but no quantitative proofs about how much that convergence was helped. No, I intuitively agree that shifting the mean value to zero should help, but math taught me that there is a huge difference between a seemingly correct statement and its proof. The ML papers seem to just state these seemingly correct ideas without real, proof-backed understanding, why this works. In other words, ML is entirely about empirical results, peppered with math-like terminology. But don't take my blunt writing style personally. Let's take the simplest example: recognizing the grayscale 30x80 pictures with 0-9 digits. IIRC, this is called the MNIST example and can be done by my cat in 1 hour without prior knowledge. Let's choose the probably simplest model: 2400 inputs are fully connected with a 1024 vector that's fully connected with a 10 vector. And let's use relu at both steps. We know that this kinda works and converges quickly. In particular, after T steps we get error E(T) and E(1e6) < 0.03 (a random guess). Can you tell me how T and E will change if we add another layer: 2400->1024->1024->10, using the same relu? Same question, but now we replace relu with tanh: 2400->1024->10.

BiasRegularizer 7y ago

I'd like to chime in that Variational Autoencoder's theoretical framework is also quite useful in unsupervised learning tasks. It bridges variational inference methods (a powerful classical ML technique) with neural networks, thus allow very efficient representation, generalization and disentanglement.

m0zg 7y ago

There's plenty of math still. In order to not waste GPU-hours, you gotta have some intution as to what _not_ to do, because otherwise you'll be trying for a very long time and relying entirely on luck, which is combinatorially unfavorable and frustrating. That intuition is often grounded in math, mostly differential calculus and statistics. That said, there are a good number of techniques which nobody is sure why they work. I.e. batchnorm, which is used all the time. People have some theories and intuitions as to why it works, but nobody is sure why, nor is there any rigorous way to pick hyperparameters for it.

playing_colours 7y ago

There are maths and proves behind Neural Networks, but Machine Learning is an umbrella term. There are areas within it, maybe not that mainstream, that are also based on solid maths foundations. One of the most interesting and elegant examples , Topological Data Analysis that is based on Topology and utilises tools like persistent homology. They can be applied to image processing, classification, etc.

tel 7y ago

First up, what you seem to be talking about is Deep Learning, not Machine Learning in general. In more general ML there are many theorems, some also apply to DL. Also, the step of "shuffle around the ML graph using some intuition" involves gathering that intuition, which usually arises from a great deal of mathematical competence. A 3x3 conv kernel versus a 2x2 one can, for instance, be discussed in terms of Fourier theory and mathematical image processing, but areas with huge built-in theory. Things like replacing the activation function were initially studied anecdotally. People realized that in some settings one activation function or another would lead. Eventually, there was also theory showing that in large nets of stable configurations, there was serious interaction between the initialization method and the activation function and problems like poor backprop signal propagation were tackled theoretically and practically. Generally, the mystery comes from the vast parameterization of these DL models. They operate in a space that's very hard to generalize—large, finite spaces. Small finite spaces get treated exhaustively. Infinite spaces get treated asymptotically. Large finite spaces get bounded on either side by those methods. So yes, there might feel like there's a dearth of theory in DL when it comes to the large scale behavior of a general network. That can be super frustrating. At the same time, people are trying to push through and create more theory every day.