7 ms·

> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function ht

> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?

gizmo686 11y ago
As the article says: " If you're a mathematician the argument is not difficult to follow, but it's not so easy for most people. That's a pity, since the underlying reasons for universality are simple and beautiful." Indeed, as a mathematician, the universality of neural networks is obvious to me from their definition. However, this article is explicitly not aimed at mathematicians, and (as far as I can tell) does a good job of presenting the argument without requiring unnecessary math knowledge, which is something that we tend to be bad at doing. Furthermore, the universality does not directly follow from the quote you provide, as you would still need to show that the hidden layer neurons form a basis. Infact, the article was about constructing such a basis in the neurons. The only thing that bringing in your quote would serve to do is make the article more confusing by unnecessarily introducing the concept of basis functions.
- dicroce 11y ago
  As a non mathematician, this is non obvious to me. Thinking about it a bit (I haven't finished reading the article yet)... Since the size of the hidden layer isn't specified, I suppose you could have a hidden layer node for every possible input... So, of course any function is computable with a neural network. Really the magical thing here is finding the smallest set of nodes that computes the function...
  - 49531 11y ago
    I've been toying with a NN trying to get it to play 2048 based on game data I recorded. I still have about 60% error rate, but I found that with 16 inputs (the tiles on the game), and 4 outputs (directions to move), it works best like a funnel. I currently have 2 hidden layers, of 12 and 8 and it's the best I've gotten so far.
  - gizmo686 11y ago
    That is essentially correct. The subtlety is that the hidden layer still has a finite (but arbitrarily large) number of nodes, while there are an infinite number of inputs. The solution to this is that you can keep adding nodes until the space between the nodes is close enough to the function you are modelling.
knappador 11y ago
Right. I had a moment of severe mistrust develop around all the times I've heard the phrase "neural network" mouthed in a pitched voice after I studied the math and realized it was barely cooler than Taylor series approximation. For a moment stacked Boltzmann machines were going to stay in obscurity, but then they became "deep learning" while also solving real problems. If there is one thing I could contribute to the popular programming lexicon, it would be making sure that the phrase "isomorphic" gets used for something other than javascript.
DougMerritt 11y ago
> This is basic and obvious math. Yes. But only once you prove that these are in fact space-spanning basis functions. Does slapping the world 'basis' magically make results obvious and interesting? Why? Because the word 'basis' carries some of that abstract-mathematics cachet?

gizmo686 11y ago

As the article says: " If you're a mathematician the argument is not difficult to follow, but it's not so easy for most people. That's a pity, since the underlying reasons for universality are simple and beautiful." Indeed, as a mathematician, the universality of neural networks is obvious to me from their definition. However, this article is explicitly not aimed at mathematicians, and (as far as I can tell) does a good job of presenting the argument without requiring unnecessary math knowledge, which is something that we tend to be bad at doing. Furthermore, the universality does not directly follow from the quote you provide, as you would still need to show that the hidden layer neurons form a basis. Infact, the article was about constructing such a basis in the neurons. The only thing that bringing in your quote would serve to do is make the article more confusing by unnecessarily introducing the concept of basis functions.

dicroce 11y ago
As a non mathematician, this is non obvious to me. Thinking about it a bit (I haven't finished reading the article yet)... Since the size of the hidden layer isn't specified, I suppose you could have a hidden layer node for every possible input... So, of course any function is computable with a neural network. Really the magical thing here is finding the smallest set of nodes that computes the function...
- 49531 11y ago
  I've been toying with a NN trying to get it to play 2048 based on game data I recorded. I still have about 60% error rate, but I found that with 16 inputs (the tiles on the game), and 4 outputs (directions to move), it works best like a funnel. I currently have 2 hidden layers, of 12 and 8 and it's the best I've gotten so far.
- gizmo686 11y ago
  That is essentially correct. The subtlety is that the hidden layer still has a finite (but arbitrarily large) number of nodes, while there are an infinite number of inputs. The solution to this is that you can keep adding nodes until the space between the nodes is close enough to the function you are modelling.

knappador 11y ago

Right. I had a moment of severe mistrust develop around all the times I've heard the phrase "neural network" mouthed in a pitched voice after I studied the math and realized it was barely cooler than Taylor series approximation. For a moment stacked Boltzmann machines were going to stay in obscurity, but then they became "deep learning" while also solving real problems. If there is one thing I could contribute to the popular programming lexicon, it would be making sure that the phrase "isomorphic" gets used for something other than javascript.

DougMerritt 11y ago

> This is basic and obvious math. Yes. But only once you prove that these are in fact space-spanning basis functions. Does slapping the world 'basis' magically make results obvious and interesting? Why? Because the word 'basis' carries some of that abstract-mathematics cachet?