7 ms·

Foundations Built for a General Theory of Neural Networks

79 points by LogicRiver 8y ago

superpermutat0r 8y ago
I love the fact that I'm seeing how stuff works. Before theory, people built huge buildings, cars, airplanes, magical electrical circuits etc. Neural networks are the same thing. Bunch of heuristics that will later be proven in theory. Yet still work incredibly well. Like cathedrals did without any mathematics. Says a lot about how tinkering with heuristics is powerful.
- throwawaymath 8y ago
  But there was a stunning amount of mathematics used for all the feats of engineering you're talking about, even early on.
  - superpermutat0r 8y ago
    Well, I'm pretty sure there was not even Euclid's geometry thousand years after he invented it in any architecture recipe book, yet they built brutally large structures. We have 13th century books from architects that built cathedrals filled with recipes with 0 mathematics, all heuristics, rules.
    - throwawaymath 8y ago
      Are you sure? For example, the Florence Cathedral (Il Duomo) required nontrivial, novel mathematics. Brunelleschi didn't succeed in that effort simply through iterative engineering. I think in general it would be more appropriate to say that fundamental theory tends to predate its applications, and then the theory and applications evolve as one informs the other in a positive feedback cycle. Likewise a lot of improvements to rocket science are due to iterative engineering, but we couldn't get off the ground (literally) without calculus.
      - zazen 8y ago
        Brunelleschi was working at the height of the Renaissance, in the early 1400s. They weren't short of book learning in renaissance Florence, but that's why it's called the renaissance. I imagine medieval cathedral building, e.g. Notre Dame starting more than 200 years previously, might have looked very different. They couldn't build a dome like Brunelleschi back then.
  - imh 8y ago
    Similarly, there's a ton of math in the applied side of neural network research too. It's not math vs no math. It's a question of foundational "General Theory of X" math.
ForWhatPurpose 8y ago
It's. Literally. Just. Regression.
zazen 8y ago
Here are the two papers linked towards the end of the slightly rambling article: https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1810.00393 https://arxiv.org/abs/1810.00393
- stochastic_monk 8y ago
  I’m reminded of this paper [0] on the relations between depth and width and expressivity and optimization. I think that ResNext’s concept of cardinality likely plays a meaningful role. [0] https://openreview.net/forum?id=BJjquybCW https://openreview.net/forum?id=BJjquybCW
- andbberger 8y ago
  See also: https://arxiv.org/abs/1606.05340 https://arxiv.org/abs/1606.05340
the_grue 8y ago
I think a important advance is being able to build neural networks in a modular fashion by constructing a toolkit of components and techniques one can reasonably expect, when combined, to produce certain result. As the field advances, this toolbox will become richer and better defined and delineated, forming basic building blocks. Just like functions, loops and virtual methods came out of the chaos of assembly.
geuis 8y ago
The current state of neural networks is somewhat analogous to how cathedrals were built in the medieval era. Master architects had a good understanding of then-ancient geometry and applied that knowledge plus knowledge of arches from Roman times to create the great cathedrals found throughout Europe. The architects didn't have a deep understanding of material science (stone) and other forces that are taken into account in the modern era, but they had enough knowledge from experience and experimentation combined with relatively simple geometry to create strong structures that have stood the test of centuries. The analogy between that and modern practical AI/neural network implementations is that experts in the field have good basic knowledge of how to construct neural networks but there isn't yet an advanced theoretical knowledge of the science that allows for precise predictions and modeling. i.e. Experts are still at the level of experimentation and noting the results and getting useful results without a super-deep understanding of how neural networks produce useful results. Hypotheses have been/are being proposed and tested right now. Expect that as a deeper level of experimental results are achieved we'll soon have a sound theoretical model that will lead to profound advances in the field.
- bra-ket 8y ago
  I think the current state of the art in AI is more comparable to the first mud huts, when they just started throwing mud on a wall to see if it sticks
  - geuis 8y ago
    I dunno. I think we’re a bit further along than that. Baked bricks at least.
nora4 8y ago
Deep Learning is currently an empirical science guided by intuition of practitioners. A main principle in experimental sciences is that a theory without predictive power is not considered a full-fledged theory. As such, unless they are interesting predictions coming from their theory (rather than only barely justifying existing empirically observed phenomena), this is just speculative theory that I would not use the phrase "Foundations Built" for. As an example of this general litmus test for a theory see e.g. Eddington's confirmation of GR: https://en.wikipedia.org/wiki/Tests_of_general_relativity#Deflection_of_light_by_the_Sun https://en.wikipedia.org/wiki/Tests_of_general_relativity#De... . If there are hitherto unknown phenomena in DL predicted by this theory then I'd stand corrected and concede that there may be something to these theories.
- fizx 8y ago
  Read this as "some foundations built". Even that is much-needed progress.
  - hyperpallium 8y ago
    Yeah, it's not foundations like elementary or axiomatic. An actual theory of ANN and then of NNN will be of more momentous import than of any previous phenomena, will usher in a fundamentally transformative age where we understand ourselves, and will surely require a fundamental breakthrough in pure mathematics... probably a great many.
- matt4077 8y ago
  The article seems pretty upfront about the fact that we are nowhere near a solid theory. One of many quotes to that effect: “ […] beginning to build the rudiments of a theory of neural networks.” The main point seems to be why such a theory is highly desirable. To that end, it does a far better job than the somehat tired Feynman aphorism about good theories “giving us more than we give them”. Namely that the current state of the art of “intuition” and vast trial-and-error runs being somewhat embarrassing for a field so closely related to math and statistics.
lotaezenwa 8y ago
Proof of the universal approximation theorem [0] for those interested [0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441...
rdlecler1 8y ago
I've beaten this drum before but another area to look at is network connectivity. When visualizing a neural network we typically see a fully connected network from one layer to the next, but if you set the value of many weights to zero you'll find that they have absolutely no effect on the function of the network. So basically we're looking at a lot of spurious interactions and this just clouds our thinking. In fact the network topology is what's driving network function here but we're not doing a good job exposing that intuitively. Computational research on Gene Regulatory Networks--which are also modeled with the same NN math--have shown how network topology is really the key driver of function. For anyone who is interested, I published a paper during my PhD that has about 195 citations. It shows that under an evolutionary process that Artificial Gene Regulatory Networks select out spurious network interactions, leaving you with what's functionally necessary (Also see the supplementary). I came from AI before starting my PhD and believe that this could be important for building a foundation for NN as well. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/

79 points by LogicRiver 8y ago

I love the fact that I'm seeing how stuff works. Before theory, people built huge buildings, cars, airplanes, magical electrical circuits etc. Neural networks are the same thing. Bunch of heuristics that will later be proven in theory. Yet still work incredibly well. Like cathedrals did without any mathematics. Says a lot about how tinkering with heuristics is powerful.

throwawaymath 8y ago
But there was a stunning amount of mathematics used for all the feats of engineering you're talking about, even early on.
- superpermutat0r 8y ago
  Well, I'm pretty sure there was not even Euclid's geometry thousand years after he invented it in any architecture recipe book, yet they built brutally large structures. We have 13th century books from architects that built cathedrals filled with recipes with 0 mathematics, all heuristics, rules.
  - throwawaymath 8y ago
    Are you sure? For example, the Florence Cathedral (Il Duomo) required nontrivial, novel mathematics. Brunelleschi didn't succeed in that effort simply through iterative engineering. I think in general it would be more appropriate to say that fundamental theory tends to predate its applications, and then the theory and applications evolve as one informs the other in a positive feedback cycle. Likewise a lot of improvements to rocket science are due to iterative engineering, but we couldn't get off the ground (literally) without calculus.
    - zazen 8y ago
      Brunelleschi was working at the height of the Renaissance, in the early 1400s. They weren't short of book learning in renaissance Florence, but that's why it's called the renaissance. I imagine medieval cathedral building, e.g. Notre Dame starting more than 200 years previously, might have looked very different. They couldn't build a dome like Brunelleschi back then.
- imh 8y ago
  Similarly, there's a ton of math in the applied side of neural network research too. It's not math vs no math. It's a question of foundational "General Theory of X" math.

ForWhatPurpose 8y ago

It's. Literally. Just. Regression.

zazen 8y ago

Here are the two papers linked towards the end of the slightly rambling article: https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1810.00393 https://arxiv.org/abs/1810.00393

stochastic_monk 8y ago
I’m reminded of this paper [0] on the relations between depth and width and expressivity and optimization. I think that ResNext’s concept of cardinality likely plays a meaningful role. [0] https://openreview.net/forum?id=BJjquybCW https://openreview.net/forum?id=BJjquybCW
andbberger 8y ago
See also: https://arxiv.org/abs/1606.05340 https://arxiv.org/abs/1606.05340

the_grue 8y ago

I think a important advance is being able to build neural networks in a modular fashion by constructing a toolkit of components and techniques one can reasonably expect, when combined, to produce certain result. As the field advances, this toolbox will become richer and better defined and delineated, forming basic building blocks. Just like functions, loops and virtual methods came out of the chaos of assembly.

geuis 8y ago

The current state of neural networks is somewhat analogous to how cathedrals were built in the medieval era. Master architects had a good understanding of then-ancient geometry and applied that knowledge plus knowledge of arches from Roman times to create the great cathedrals found throughout Europe. The architects didn't have a deep understanding of material science (stone) and other forces that are taken into account in the modern era, but they had enough knowledge from experience and experimentation combined with relatively simple geometry to create strong structures that have stood the test of centuries. The analogy between that and modern practical AI/neural network implementations is that experts in the field have good basic knowledge of how to construct neural networks but there isn't yet an advanced theoretical knowledge of the science that allows for precise predictions and modeling. i.e. Experts are still at the level of experimentation and noting the results and getting useful results without a super-deep understanding of how neural networks produce useful results. Hypotheses have been/are being proposed and tested right now. Expect that as a deeper level of experimental results are achieved we'll soon have a sound theoretical model that will lead to profound advances in the field.

bra-ket 8y ago
I think the current state of the art in AI is more comparable to the first mud huts, when they just started throwing mud on a wall to see if it sticks
- geuis 8y ago
  I dunno. I think we’re a bit further along than that. Baked bricks at least.

nora4 8y ago

Deep Learning is currently an empirical science guided by intuition of practitioners. A main principle in experimental sciences is that a theory without predictive power is not considered a full-fledged theory. As such, unless they are interesting predictions coming from their theory (rather than only barely justifying existing empirically observed phenomena), this is just speculative theory that I would not use the phrase "Foundations Built" for. As an example of this general litmus test for a theory see e.g. Eddington's confirmation of GR: https://en.wikipedia.org/wiki/Tests_of_general_relativity#Deflection_of_light_by_the_Sun https://en.wikipedia.org/wiki/Tests_of_general_relativity#De... . If there are hitherto unknown phenomena in DL predicted by this theory then I'd stand corrected and concede that there may be something to these theories.

fizx 8y ago
Read this as "some foundations built". Even that is much-needed progress.
- hyperpallium 8y ago
  Yeah, it's not foundations like elementary or axiomatic. An actual theory of ANN and then of NNN will be of more momentous import than of any previous phenomena, will usher in a fundamentally transformative age where we understand ourselves, and will surely require a fundamental breakthrough in pure mathematics... probably a great many.
matt4077 8y ago
The article seems pretty upfront about the fact that we are nowhere near a solid theory. One of many quotes to that effect: “ […] beginning to build the rudiments of a theory of neural networks.” The main point seems to be why such a theory is highly desirable. To that end, it does a far better job than the somehat tired Feynman aphorism about good theories “giving us more than we give them”. Namely that the current state of the art of “intuition” and vast trial-and-error runs being somewhat embarrassing for a field so closely related to math and statistics.

lotaezenwa 8y ago

Proof of the universal approximation theorem [0] for those interested [0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441...

rdlecler1 8y ago

I've beaten this drum before but another area to look at is network connectivity. When visualizing a neural network we typically see a fully connected network from one layer to the next, but if you set the value of many weights to zero you'll find that they have absolutely no effect on the function of the network. So basically we're looking at a lot of spurious interactions and this just clouds our thinking. In fact the network topology is what's driving network function here but we're not doing a good job exposing that intuitively. Computational research on Gene Regulatory Networks--which are also modeled with the same NN math--have shown how network topology is really the key driver of function. For anyone who is interested, I published a paper during my PhD that has about 195 citations. It shows that under an evolutionary process that Artificial Gene Regulatory Networks select out spurious network interactions, leaving you with what's functionally necessary (Also see the supplementary). I came from AI before starting my PhD and believe that this could be important for building a foundation for NN as well. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/