4 ms·

> For one - a neural network is a universal function approximator. Pretty much any way you turn you'll find classes of universal function approximators. The al

> For one - a neural network is a universal function approximator. Pretty much any way you turn you'll find classes of universal function approximators. The almost 200-year old Weierstrass-Stone theorem says that polynomials have this property. More prosaically, locally constant functions have this property (that's how the Riemann integral is defined). Trees and random forests exploit just this fact. Trigonometric polynomials have this property (via Fourier series). Wavelets. Now, here's a mind-blowing fact: did you know that addition and 1-d functions form such a class of universal approximators. It's called Kolmogorov-Arnold theorem [1], and it's the origin of the generalized additive models (GAM) in statistics. Back to neural networks: the universal approximation theorem is valid with a single layer. But nobody uses them with a single layer. Which means, people don't use NN's because of this (quite useless) property. Separately, this theorem is just an existence theorem, and it's not concerned with the complexity if the approximation. The approximation eventually relies in some form of optimization (see all the flavors of stochastic descent). But optimization, even convex optimization, is known to be NP-hard. So, these are all good reason to discard the universal approximation theorem when talking about neural networks. [1] https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...

mdp2021 4y ago
I think there is an ambiguity in the use of the expression. One is the mathematical property of a function to almost repeat the results of another. An somehow different one is conveying the idea that there are systems to "almost repeat the intended results" of a function unavailable to us - "you could build a well-defined function for that or approximate its results building a function approximator". It is more of a suggestive introductory idea and core concept.
- YeGoblynQueenne 4y ago
  (Not the OP) Yes, that's right, but the problem with trying to approximate a function you don't know is that you don't know how well you've approximated it. You're left having to, well, approximate, the accuracy of the approximation. That's why everyone in machine learning trains their models and then tests them on a held-out test set. In theory, the error on the test set should predict the error on the true distribution (and that's under PAC-Learning assumptions). Unfortunately the way this is done in practice is sloppy and uninformative. For example, researchers usually have access to the test set during training, they split the training/test set to 80/20% so that the error calculated on the test set does not generalise well to truly unseen data, etc. Bottom line: all this approximation stuff is great in theory, but not so good in practice. Or, rather, in theory machine learning systems have almost perfect accuracies, in practice we have no idea how good they are.
  - sgt101 4y ago
    >For example, researchers usually have access to the test set during training What can happen is that people train various models, then run them on the test set and decide that one or two of them are promising, and rinse and repeat. This creates an inefficient optimization (mediated by the researcher) on the test set. BTW this is absolutely unacceptable practice.
    - YeGoblynQueenne 4y ago
      Yes! And even without direct access to the test set one can optimise on the test set. And it is as absolutely unacceptable practice as it is widespread, which is to say, almost universally so. And the reason for that is the poor generalisation ability of neural nets, which basically forces the hand of researchers: splitting training/test sets by 20/80% would give a much better approximation to true error, but that would leave too litle data for actual training. So people just train with as much data as they can and hope, or don't care, about the true error. This kind of thing is so widespread that not only is it standard practice, and accepted by the majority of the field, but, I suspect, a majority of people in the field don't even realise how little sense it makes. It's just how people learn to do this stuff and they never question it because the graphs look good and the papers get cited. We're stuck in a very broken paradigm today in machine learning and that's all because we optimise for self-selected metrics of success. I find it extremely concerning to see the same paradigm applied to science. We risk destroying our ability to understand the world irreparably, or at least for such a long time that we will never recover from it.
      - mdp2021 4y ago
        > too little data > very broken paradigm Well, that first mention makes it evident that the context is "Big Data driven Machine Learning", and this paradigm linked to BDDML is not the "intended ML" - it is just a passage in our toying. Because it is also very clear to us (well¹) that some proper learning has "one shot" properties, so at least until single information instances are determinant in model building (as opposed to slightly retouching weights) like in symbolic communication and more, we will know that something is amiss, that we are still crossing the river's fords. But the preparation of the environment to enable e.g. "one shot learning" is the current problem in such direction. (¹even though some seem as if so inebriated with the current toys to claim "that's it, done, it rolls, TheBall 2.0, it's animated, it lives, we have created the Golem ... Look, a gramophone, it speaks, ask him for advice and serve him food") This given, it is not that the "toying" has not produced noteworthy aids in real life application. > I find it extremely concerning That concern is that we are in need of enhanced human learning - we have more and more complexity to manage -, yet the effect and promise from these areas swing from boost to complacency. Some ask, "where is the problem in the "transparency problem", in blackbox responses": in that we do require more understanding, the human part is that which should primarily seek refinement.
- credit_guy 4y ago
  It doesn't change what I said. Every class of functions that has the universal approximation property can be used to "learn" from data. This is what statistical learning is about. You can use trees, random forests, piecewise linear functions, multidimensional adaptive regression splines (MARS), multidimensional fourier series or wavelets, radial basis functions, the list goes on and on. And people have been using all these methods with success. The idea that the universal approximation theorem is something special about neural networks is completely wrong. Deep learning is valuable because it represents a class of functions closed under composition. That allows it to discover features at different scales. You can in principle add composability to other classes of functions. You would end up with deep splines, for example. Or deep radial basis functions, or deep wavelets. And people have tried those, but it just didn't take off. It was probably the advantage of the first mover. Deep NN's turned out to be useful, people built a large ecosystem for them, and then other deep things (like deep splines or wavelets) just did not show a significant advantage. Yet.
  - mdp2021 4y ago
    > It was probably the advantage of the first mover It was probably because of the "inspiring" stretched simile between nervous system and network of non-linear operators. Also, there is a matter of the efficiency of learning algorithms: would the other classes of functions that satisfy the universal approximation theorem also use gradient descent and similar and subtricks, or would they allow different tricks for their adaptation to the desired results? > It doesn't change what I said Well, you stated that «people don't use NN's because of this (quite useless) property». I replied that the general idea of automatic function development may drive towards this promised Eldorado that would not only spare us from the burden of solution definition, but also (with the right tricks implemented) possibly confide such solutions to us in a "spoiler".

mdp2021 4y ago

I think there is an ambiguity in the use of the expression. One is the mathematical property of a function to almost repeat the results of another. An somehow different one is conveying the idea that there are systems to "almost repeat the intended results" of a function unavailable to us - "you could build a well-defined function for that or approximate its results building a function approximator". It is more of a suggestive introductory idea and core concept.

YeGoblynQueenne 4y ago
(Not the OP) Yes, that's right, but the problem with trying to approximate a function you don't know is that you don't know how well you've approximated it. You're left having to, well, approximate, the accuracy of the approximation. That's why everyone in machine learning trains their models and then tests them on a held-out test set. In theory, the error on the test set should predict the error on the true distribution (and that's under PAC-Learning assumptions). Unfortunately the way this is done in practice is sloppy and uninformative. For example, researchers usually have access to the test set during training, they split the training/test set to 80/20% so that the error calculated on the test set does not generalise well to truly unseen data, etc. Bottom line: all this approximation stuff is great in theory, but not so good in practice. Or, rather, in theory machine learning systems have almost perfect accuracies, in practice we have no idea how good they are.
- sgt101 4y ago
  >For example, researchers usually have access to the test set during training What can happen is that people train various models, then run them on the test set and decide that one or two of them are promising, and rinse and repeat. This creates an inefficient optimization (mediated by the researcher) on the test set. BTW this is absolutely unacceptable practice.
  - YeGoblynQueenne 4y ago
    Yes! And even without direct access to the test set one can optimise on the test set. And it is as absolutely unacceptable practice as it is widespread, which is to say, almost universally so. And the reason for that is the poor generalisation ability of neural nets, which basically forces the hand of researchers: splitting training/test sets by 20/80% would give a much better approximation to true error, but that would leave too litle data for actual training. So people just train with as much data as they can and hope, or don't care, about the true error. This kind of thing is so widespread that not only is it standard practice, and accepted by the majority of the field, but, I suspect, a majority of people in the field don't even realise how little sense it makes. It's just how people learn to do this stuff and they never question it because the graphs look good and the papers get cited. We're stuck in a very broken paradigm today in machine learning and that's all because we optimise for self-selected metrics of success. I find it extremely concerning to see the same paradigm applied to science. We risk destroying our ability to understand the world irreparably, or at least for such a long time that we will never recover from it.
    - mdp2021 4y ago
      > too little data > very broken paradigm Well, that first mention makes it evident that the context is "Big Data driven Machine Learning", and this paradigm linked to BDDML is not the "intended ML" - it is just a passage in our toying. Because it is also very clear to us (well¹) that some proper learning has "one shot" properties, so at least until single information instances are determinant in model building (as opposed to slightly retouching weights) like in symbolic communication and more, we will know that something is amiss, that we are still crossing the river's fords. But the preparation of the environment to enable e.g. "one shot learning" is the current problem in such direction. (¹even though some seem as if so inebriated with the current toys to claim "that's it, done, it rolls, TheBall 2.0, it's animated, it lives, we have created the Golem ... Look, a gramophone, it speaks, ask him for advice and serve him food") This given, it is not that the "toying" has not produced noteworthy aids in real life application. > I find it extremely concerning That concern is that we are in need of enhanced human learning - we have more and more complexity to manage -, yet the effect and promise from these areas swing from boost to complacency. Some ask, "where is the problem in the "transparency problem", in blackbox responses": in that we do require more understanding, the human part is that which should primarily seek refinement.
credit_guy 4y ago
It doesn't change what I said. Every class of functions that has the universal approximation property can be used to "learn" from data. This is what statistical learning is about. You can use trees, random forests, piecewise linear functions, multidimensional adaptive regression splines (MARS), multidimensional fourier series or wavelets, radial basis functions, the list goes on and on. And people have been using all these methods with success. The idea that the universal approximation theorem is something special about neural networks is completely wrong. Deep learning is valuable because it represents a class of functions closed under composition. That allows it to discover features at different scales. You can in principle add composability to other classes of functions. You would end up with deep splines, for example. Or deep radial basis functions, or deep wavelets. And people have tried those, but it just didn't take off. It was probably the advantage of the first mover. Deep NN's turned out to be useful, people built a large ecosystem for them, and then other deep things (like deep splines or wavelets) just did not show a significant advantage. Yet.
- mdp2021 4y ago
  > It was probably the advantage of the first mover It was probably because of the "inspiring" stretched simile between nervous system and network of non-linear operators. Also, there is a matter of the efficiency of learning algorithms: would the other classes of functions that satisfy the universal approximation theorem also use gradient descent and similar and subtricks, or would they allow different tricks for their adaptation to the desired results? > It doesn't change what I said Well, you stated that «people don't use NN's because of this (quite useless) property». I replied that the general idea of automatic function development may drive towards this promised Eldorado that would not only spare us from the burden of solution definition, but also (with the right tricks implemented) possibly confide such solutions to us in a "spoiler".