8 ms·

>> This push towards elegance is unscientific I think you are right on that one. The Standard Model for example is absoluteley not elegant, it is a huge convol

>> This push towards elegance is unscientific I think you are right on that one. The Standard Model for example is absoluteley not elegant, it is a huge convoluted mess of parameters and constants (just google for "Standard Model Lagrangian"). Yet, it is the best we came up with, explaining the dynamics of fields and particles[0], that surround us. Correct answers don't have to be elegant per se, its nice when they are, but it's not a prerequisite. But, consider this: You can train a Neural Network to predict the distance of an object, thrown with velocity v_0, an angle of α, under the influcence of gravitational acceleration g. After a few hunderd rounds of training, a suited NN can reasonably predict the outcome of said experiment, with the input: v_0, α and g. And, as you pointed out, a researcher can explain to you why this is the case based on activation, feedback-loops, learning algorithm and other parameters. But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics. A Neural Network can give you answers and predictions, yes, but you are not able to incorporate them into a wider theory, since the output is always numerical in nature. It is also always tied to one specific fact you are interested in and cannot give you a generalization. [0]: In the model of QFT particles are also fields of a different kind.

manquer 9y ago
A Neural network will give you a relation which is approximation of that relation not just a numeric output . Also you should remember that while it looks elegant the Newtonian model is also an approximation and fails noticeably in cases where relativistic effects are significant
candiodari 9y ago
That may have something to do with the fact that even the biggest neural networks have a "cortex size" one or two orders of magnitude smaller than a cat and "exist" (number of impulses processed * amount of time that would take in an animal) for a few weeks. That means one should expect them to perform tasks about as well as a mouse brain can do them, between a third and halfway to adulthood. And yes, that level of cortical processing does not seem to support coming up with symbolic systems and attempting to use them to describe the world around them. Absolutely. Mice don't do that either.
- sgt101 9y ago
  Am I out of date? I thought that current big deep networks had 100's of millions of parameters, where as mammalian brains were of the order of 10's of billions of neurons with perhaps 1000 times more connections so giving 10's of trillions of parameters (if you imagine a mammalian neuron to work like that).
  - candiodari 9y ago
    One parameter maps to one synapse, not to one neuron.
DrNuke 9y ago
> But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics. Come on, you can just fit the NN numerical outputs with Excel and come up with tentative explanations for both the underlying elegant rule and the science. The point with NN-based science being you are not left guessing much, instead retrofit the NN numerical output and build a coherent narrative with other accepted results.
- kosmet 9y ago
  Without that equation, you cannot build the rocket that goes to the moon, however.
  - DrNuke 9y ago
    Feeding the retrofitted "blurred" equation to mathematicians and let them come up with tentative exact models, while being almost there already?
    - pshc 9y ago
      Anything that fits within the error bars is viable if you ask me. Whatever works works.
      - sgt101 9y ago
        A lot of investors agree with you. Often they get very rich, especially if they head for the beach before the fit breaks down. On the other hand...
- deleted 9y ago
  [deleted]
chestervonwinch 9y ago
It seems to me that you could treat a sufficiently accurate model (be it a deep net or otherwise) as the solution to an unknown, governing physical law, and attempt to identify the law from the model, post-hoc. I'm not sure if there's been work done in the domain of pattern recognition, but here's an overview where in an example, Navier-Stokes is identified from data: https://sinews.siam.org/Details-Page/data-driven-discovery-of-governing-physical-laws https://sinews.siam.org/Details-Page/data-driven-discovery-o...
criddell 9y ago
> cannot give you a generalization Maybe a second NN could find that for you. You might end up with factors like "speed of an unladen swallow", but perhaps all that is needed is some nudge away from that local maxima to keep searching the space.
Houshalter 9y ago
But it will be much more accurate. The neural network will learn to take into account air resistance and wind, for instance. Depending on how much data you give it and the resolution, it might even learn to model air turbulence, spinning and wobbling of thrown objects, etc. It will be a lot more accurate then the simplistic physics model. And sure, it doesn't output a nice simple equation. But in most real problems, there aren't nice simple equations to output. Physics is sort of a fluke, where the fundamental laws of nature happen to be so simple we can do that. In something like biology, you are never going to find a simple equation that explains a biological system. They are complicated messes of millions of interacting pieces. Even if we knew how everything worked, we don't have the computing resources to accurately simulate it anyway. And we are actually seeing neural networks be useful in these areas. They are able to predict what chemicals are more likely to be useful drugs. If you restricted yourself to using only simple mathematical models, you would never get these results. This very article is about the very difficult domain of image processing. Why should we expect the statistical distribution of photographs to be simple and explainable by simple math? Finding an equation that accurately models the shape of a human face is pointless and misguided. Before NNs came along, they were doing crazy stuff like handcoding complicated heuristics that detected dark spots that might be eye sockets. There's nothing elegant about that. Physicists are really blessed to have a domain where they can ignore stuff like air resistance. Where the data really does fit simple equations. And as the parent comment says, you can interpret neural networks. You can observe the activation behaviors, and discover what features it's learned to detect eye sockets, or chemicals with certain properties. You can see how the output changes as the inputs change, and fit a simpler model to that.
- nightski 9y ago
  But that's precisely the problem isn't it? It's hard to know when it's picking up additional details or when it's overfitting to noise.
  - jacquesm 9y ago
    Why is that hard? Expose it to data it hasn't seen yet and see if the results check out.
omalleyt 9y ago
A beautiful example and well-illustrated by the fact that humans could play catch before the development of trigonometry
jacquesm 9y ago
That's the same as my own example of explaining to people that dogs brains can do some pretty complex math. I throw a stick that's turning as it flies and the dog catches it every time, but it can't explain nor communicate the formula that went into that. But this is really an optimization problem: representing your formula as a bunch of weights that are then used to drive a cascade of multiply-add rules to derive some output is an imprecise but roughly accurate way to model a problem. That there is a much more direct and analytical way of modeling that problem requires intelligence of a different order. But a program like mathematica gets awfully close to that, so that's not a problem that is a good fit for being solved with a neural network even if it would work. I'd prefer to use a neural network for problems that have a less well defined solution.
sapphireblue 9y ago
>But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics. It is not impossible. There is a combination of a special neural network architecture and strong sparsity-inducing regularization that makes it possible to learn equations from dynamics dataset: https://openreview.net/pdf?id=BkgRp0FYe https://openreview.net/pdf?id=BkgRp0FYe
- arbie 9y ago
  This is a very interesting paper. Is there an implementation example in any major framework?
zardo 9y ago
>Correct answers don't have to be elegant per se, its nice when they are, but it's not a prerequisite If a powerful idea is not elegant, we have the option of creating new mathematics in which it _can_ be elegantly expressed.

manquer 9y ago

A Neural network will give you a relation which is approximation of that relation not just a numeric output . Also you should remember that while it looks elegant the Newtonian model is also an approximation and fails noticeably in cases where relativistic effects are significant

candiodari 9y ago

That may have something to do with the fact that even the biggest neural networks have a "cortex size" one or two orders of magnitude smaller than a cat and "exist" (number of impulses processed * amount of time that would take in an animal) for a few weeks. That means one should expect them to perform tasks about as well as a mouse brain can do them, between a third and halfway to adulthood. And yes, that level of cortical processing does not seem to support coming up with symbolic systems and attempting to use them to describe the world around them. Absolutely. Mice don't do that either.

sgt101 9y ago
Am I out of date? I thought that current big deep networks had 100's of millions of parameters, where as mammalian brains were of the order of 10's of billions of neurons with perhaps 1000 times more connections so giving 10's of trillions of parameters (if you imagine a mammalian neuron to work like that).
- candiodari 9y ago
  One parameter maps to one synapse, not to one neuron.

DrNuke 9y ago

> But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics. Come on, you can just fit the NN numerical outputs with Excel and come up with tentative explanations for both the underlying elegant rule and the science. The point with NN-based science being you are not left guessing much, instead retrofit the NN numerical output and build a coherent narrative with other accepted results.

kosmet 9y ago
Without that equation, you cannot build the rocket that goes to the moon, however.
- DrNuke 9y ago
  Feeding the retrofitted "blurred" equation to mathematicians and let them come up with tentative exact models, while being almost there already?
  - pshc 9y ago
    Anything that fits within the error bars is viable if you ask me. Whatever works works.
    - sgt101 9y ago
      A lot of investors agree with you. Often they get very rich, especially if they head for the beach before the fit breaks down. On the other hand...
deleted 9y ago
[deleted]

chestervonwinch 9y ago

It seems to me that you could treat a sufficiently accurate model (be it a deep net or otherwise) as the solution to an unknown, governing physical law, and attempt to identify the law from the model, post-hoc. I'm not sure if there's been work done in the domain of pattern recognition, but here's an overview where in an example, Navier-Stokes is identified from data: https://sinews.siam.org/Details-Page/data-driven-discovery-of-governing-physical-laws https://sinews.siam.org/Details-Page/data-driven-discovery-o...

criddell 9y ago

> cannot give you a generalization Maybe a second NN could find that for you. You might end up with factors like "speed of an unladen swallow", but perhaps all that is needed is some nudge away from that local maxima to keep searching the space.

Houshalter 9y ago

But it will be much more accurate. The neural network will learn to take into account air resistance and wind, for instance. Depending on how much data you give it and the resolution, it might even learn to model air turbulence, spinning and wobbling of thrown objects, etc. It will be a lot more accurate then the simplistic physics model. And sure, it doesn't output a nice simple equation. But in most real problems, there aren't nice simple equations to output. Physics is sort of a fluke, where the fundamental laws of nature happen to be so simple we can do that. In something like biology, you are never going to find a simple equation that explains a biological system. They are complicated messes of millions of interacting pieces. Even if we knew how everything worked, we don't have the computing resources to accurately simulate it anyway. And we are actually seeing neural networks be useful in these areas. They are able to predict what chemicals are more likely to be useful drugs. If you restricted yourself to using only simple mathematical models, you would never get these results. This very article is about the very difficult domain of image processing. Why should we expect the statistical distribution of photographs to be simple and explainable by simple math? Finding an equation that accurately models the shape of a human face is pointless and misguided. Before NNs came along, they were doing crazy stuff like handcoding complicated heuristics that detected dark spots that might be eye sockets. There's nothing elegant about that. Physicists are really blessed to have a domain where they can ignore stuff like air resistance. Where the data really does fit simple equations. And as the parent comment says, you can interpret neural networks. You can observe the activation behaviors, and discover what features it's learned to detect eye sockets, or chemicals with certain properties. You can see how the output changes as the inputs change, and fit a simpler model to that.

nightski 9y ago
But that's precisely the problem isn't it? It's hard to know when it's picking up additional details or when it's overfitting to noise.
- jacquesm 9y ago
  Why is that hard? Expose it to data it hasn't seen yet and see if the results check out.

omalleyt 9y ago

A beautiful example and well-illustrated by the fact that humans could play catch before the development of trigonometry

jacquesm 9y ago

That's the same as my own example of explaining to people that dogs brains can do some pretty complex math. I throw a stick that's turning as it flies and the dog catches it every time, but it can't explain nor communicate the formula that went into that. But this is really an optimization problem: representing your formula as a bunch of weights that are then used to drive a cascade of multiply-add rules to derive some output is an imprecise but roughly accurate way to model a problem. That there is a much more direct and analytical way of modeling that problem requires intelligence of a different order. But a program like mathematica gets awfully close to that, so that's not a problem that is a good fit for being solved with a neural network even if it would work. I'd prefer to use a neural network for problems that have a less well defined solution.

sapphireblue 9y ago

>But neither the NN nor the researcher will be able to give you a rule like: "F=(G m_1 m_2)/(r^2)" to explain the underlying reasons for the object-trajectory dynamics. It is not impossible. There is a combination of a special neural network architecture and strong sparsity-inducing regularization that makes it possible to learn equations from dynamics dataset: https://openreview.net/pdf?id=BkgRp0FYe https://openreview.net/pdf?id=BkgRp0FYe

arbie 9y ago
This is a very interesting paper. Is there an implementation example in any major framework?

zardo 9y ago

>Correct answers don't have to be elegant per se, its nice when they are, but it's not a prerequisite If a powerful idea is not elegant, we have the option of creating new mathematics in which it _can_ be elegantly expressed.