7 ms·

Some applications in computational physics involve solving a "variational" problem, where you have some parameterized function and try to numerically find the p

Some applications in computational physics involve solving a "variational" problem, where you have some parameterized function and try to numerically find the parameters that minimize energy or error. This does not necessarily involve supervised learning from outside data as in this article -- it can be purely an optimization problem. But neural networks are very good parametric function approximators, generally better than what traditionally gets used in physics (b-splines or whatever). So people have started to design neural networks that are well-suited as function approximators for specific physical systems. It's fairly straightforward -- it's not an "AI" that has "knowledge" of "physics" -- just using modern techniques and hardware to solve a numerical minimization problem. I think this will probably become pretty widespread. It won't be flashy or exciting though -- it will be boring to anyone but specialists, as the rest of machine learning ought to be.

wenc 6y ago
So the idea of surrogate models (for parameter estimation) has been around for some time, where f(x, θ) is some (computationally) simplified model of a complex model/simulation (x = factors, θ = parameters). f can be any arbitrary choice that works. Not sure if the choice of f being a NN is necessarily related to AI, where some cognitive function is being replicated. It is a good function approximator though.
mumbisChungo 6y ago
Why ought machine learning be boring to anyone but specialists? Does this imply that specialists ought to be born, rather than become specialists out of interest?
- catalogia 6y ago
  I don't think they mean ML in general is boring, just that this particular application of it isn't particularly flashy.
  - mumbisChungo 6y ago
    Maybe I misinterpreted this: >it will be boring to anyone but specialists, as the rest of machine learning ought to be.
- KineticLensman 6y ago
  I interpreted this to mean something like 'how phones / car engines / etc work' is not of interest to most of their users as long as they get the job done. If they get 'interesting' it can means that something isn't working right. Where 'interesting' = 'suddenly noticeable'.
- currymj 6y ago
  There are lots of things that I think are comparable to machine learning in the sense that they combine applied math and heavy computation and are very practically important, like simulating chemical reactions, solving operations research problems, or computational fluid dynamics. You cannot talk about these things at cocktail parties, though, because people will slowly shuffle away from you -- whereas you can talk about deep learning, which is odd. Basically, I think if somebody wants to work in machine learning then they should be encouraged, and I think it's great that barriers to entry are lower than most fields, but the average person should not feel like they need to care about it, and if they do it might be because they have an inaccurate narrative.
  - mumbisChungo 6y ago
    >You cannot talk about these things at cocktail parties, though, because people will slowly shuffle away from you -- whereas you can talk about deep learning, which is odd. It's really not odd at all. The average person has some familiarity with ML/AI, so you don't have to expend the energy to introduce them to the topic in a way that is understandable and also engaging to them. They already have a baseline, and are likely already aware of some interesting use cases. By contrast, they might not even know what "operations research" is, so you have to be both willing and able to expend the energy to explain the field in a way that is comprehensible and interesting. I'm sure it's possible, but the cross-section of people with the knowledge, the interest, and the social graces to do it is probably small. To me it seems that a large swath of the science community dislikes buzzwordification and pop science more than they like proliferation of knowledge, based on how negative responses seem to be to things like normie interest in AI here. I would be fascinated to read any peer reviewed studies on the negative impacts of pop science on long term scientific advancement, so that I could understand this bias (and debunk my own bias that more interest in science is better in the long term).
ChrisRackauckas 6y ago
Yes, I think this is a great use for neural networks since they are effectively high dimensional function approximators, and something like Schrondinger's equation is a PDE where the number of dimensions is the number of observables so it can get very high dimensional very fast. Classical methods don't necessarily scale that well in high dimensions (curse of dimensionality: cost is exponential in dimensions), but using neural networks does very well. This gives rise to the physics-informed neural network and deep backwards stochastic differential equation approaches which will likely be driving a lot of future HPC applications in a way that blends physical equations with neural network approaches. We recently released a library, NeuralPDE [1], which utilizes a lot of these approaches to solve what were traditionally difficult equations in an automated form. I think the future is bright for scientific machine learning! [1] https://neuralpde.sciml.ai/dev/ https://neuralpde.sciml.ai/dev/
- wenc 6y ago
  This is fascinating. ELI5: how does this work? (I'm couldn't find references on the linked site) Let's say I supply a high-dimensional DAE, f(x', x, z) = 0, x(0) = x₀, where classical methods like quadrature are unwieldy. Does the algorithm generate n samples in the solution space by integrating n times and then fitting an NN? With different initial conditions? Or does it perform quadrature with NNs instead of polynomial basis functions?
  - ChrisRackauckas 6y ago
    A lot of these methods here utilize the universal differential equation framework described here: https://arxiv.org/abs/2001.04385 https://arxiv.org/abs/2001.04385 . Specifically, the last example in this preprint describes how high dimensional parabolic PDEs can be solving using neural networks inside of a specific SDE (derivation in the supplemental). Discrete physics-informed neural networks also are a subset of this methodology. The other subset of methods, continuous physics-informed neural networks, are described in https://www.sciencedirect.com/science/article/pii/S0021999118307125 https://www.sciencedirect.com/science/article/pii/S002199911... . For a very basic introduction, I wrote some lecture notes on how this is done for a simple ODE with code examples: https://mitmath.github.io/18S096SciML/lecture2/ml https://mitmath.github.io/18S096SciML/lecture2/ml
amelius 6y ago
Is this AI taking the place of preconditioners as used in iterative solvers? Would an AI be able to learn how to apply multigrid methods?
- ChrisRackauckas 6y ago
  We have prototypes in Julia for this. The answer is yes, there are tricks you can use to do this effectively.
  - astroH 6y ago
    Could you elaborate a bit more on this? I have thought about employing NNs in this way for quite a while but the thing I never wrapped my head around was ensuring how it generalizes to different problems.
whinvik 6y ago
Almost every time a PDE is solved on a computer, it is a variational problem. Maybe neural networks are indeed good at this but I haven't seen any literature that shows that it is provably better. A reference would be good, especially to this point "But neural networks are very good parametric function approximators, generally better than what traditionally gets used in physics (b-splines or whatever)."
- currymj 6y ago
  https://arxiv.org/abs/1909.08423 https://arxiv.org/abs/1909.08423 and https://arxiv.org/abs/1909.02487 https://arxiv.org/abs/1909.02487 are some examples I've been looking at recently.
  - whinvik 6y ago
    Thanks, not familiar with QM at all, but it seems to me from glancing through one of the papers that the neural network is used to replace a popular way of representing the wave function which itself is an Ansatz. Not very convincing, but of course, as I said not familiar with the background and so I may be overlooking something.
    - currymj 6y ago
      that's exactly it -- they take an existing form for the ansatz (or the general idea of it, at least), and make it more flexible by replacing pieces with neural networks that have many more parameters, while maintaining constraints required by physics. I think this will become very common in the future.
      - whinvik 6y ago
        That maybe true but what I was looking for is a more convincing way of showing that a neural network approximates a function better than other functions, such as say a b-spline. For example, you say that the neural network with many more parameters works better, but what if we had a b-spline with many more nodes.
noobermin 6y ago
Thank you. More importantly, this is not new. Sheesh, how much of AI is just hype. I am in favor in not using the term AI for such things.
- baq 6y ago
  > John McCarthy, the Father of AI, famously said: "As soon as it works, no one calls it AI any more." Leading researcher Rodney Brooks says "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation. '" https://cacm.acm.org/blogs/blog-cacm/138907-john-mccarthy/fulltext https://cacm.acm.org/blogs/blog-cacm/138907-john-mccarthy/fu...
  - Barrin92 6y ago
    this is an extremely overused fortune-cookie like quote. There's a legitimate distinction to be made between intelligence on one hand, and simply computation or calculation on the other. If we start calling every numerical method AI we're rendering the term meaningless.
    - glial 6y ago
      What’s the distinction?
      - Barrin92 6y ago
        In the most basic sense intelligence involves the aqcuisition of knowledge, which is representation or generalisation at some higher level of abstraction and the ability to make decisions. The mere ability to perform computational work is something virtually even the tiniest piece of hardware entails, or even an abacus for that matter.
      - LoSboccacc 6y ago
        intelligence is inductive finding correlations and using a human to filter the interesting ones from the flukes doesn't make the correlations engine an ai, the intelligence is still in the human
bencw 6y ago
This is something I've wondered about (along with potential applications of autograd outside of deep learning). Do you have a recommended starting point for someone who wants to learn more about this?
exabyte 6y ago
The power that I see in machine learning is the techniques being developed to handle the unavoidable noise in empirical data. I think that poses a large obstacle for traditional techniques although I am not familiar enough to compare.
- willis936 6y ago
  To me the value is in matching relationships (equations) of curated parameters from empirical data and using simulated recreations of the experiment as the objective. As soon as you can recreate experimental results in a simulation then you’ve made a successful model for that domain. This is an incredibly important and difficult task for fluid dynamics and plasma physics.
  - stainforth 6y ago
    Fusion is right around the corner!
deleted 6y ago
[deleted]
mtgx 6y ago
Wouldn't a D-Wave quantum annealing computer be even better for this sort of thing?

wenc 6y ago

So the idea of surrogate models (for parameter estimation) has been around for some time, where f(x, θ) is some (computationally) simplified model of a complex model/simulation (x = factors, θ = parameters). f can be any arbitrary choice that works. Not sure if the choice of f being a NN is necessarily related to AI, where some cognitive function is being replicated. It is a good function approximator though.

mumbisChungo 6y ago

Why ought machine learning be boring to anyone but specialists? Does this imply that specialists ought to be born, rather than become specialists out of interest?

catalogia 6y ago
I don't think they mean ML in general is boring, just that this particular application of it isn't particularly flashy.
- mumbisChungo 6y ago
  Maybe I misinterpreted this: >it will be boring to anyone but specialists, as the rest of machine learning ought to be.
KineticLensman 6y ago
I interpreted this to mean something like 'how phones / car engines / etc work' is not of interest to most of their users as long as they get the job done. If they get 'interesting' it can means that something isn't working right. Where 'interesting' = 'suddenly noticeable'.
currymj 6y ago
There are lots of things that I think are comparable to machine learning in the sense that they combine applied math and heavy computation and are very practically important, like simulating chemical reactions, solving operations research problems, or computational fluid dynamics. You cannot talk about these things at cocktail parties, though, because people will slowly shuffle away from you -- whereas you can talk about deep learning, which is odd. Basically, I think if somebody wants to work in machine learning then they should be encouraged, and I think it's great that barriers to entry are lower than most fields, but the average person should not feel like they need to care about it, and if they do it might be because they have an inaccurate narrative.
- mumbisChungo 6y ago
  >You cannot talk about these things at cocktail parties, though, because people will slowly shuffle away from you -- whereas you can talk about deep learning, which is odd. It's really not odd at all. The average person has some familiarity with ML/AI, so you don't have to expend the energy to introduce them to the topic in a way that is understandable and also engaging to them. They already have a baseline, and are likely already aware of some interesting use cases. By contrast, they might not even know what "operations research" is, so you have to be both willing and able to expend the energy to explain the field in a way that is comprehensible and interesting. I'm sure it's possible, but the cross-section of people with the knowledge, the interest, and the social graces to do it is probably small. To me it seems that a large swath of the science community dislikes buzzwordification and pop science more than they like proliferation of knowledge, based on how negative responses seem to be to things like normie interest in AI here. I would be fascinated to read any peer reviewed studies on the negative impacts of pop science on long term scientific advancement, so that I could understand this bias (and debunk my own bias that more interest in science is better in the long term).

ChrisRackauckas 6y ago

Yes, I think this is a great use for neural networks since they are effectively high dimensional function approximators, and something like Schrondinger's equation is a PDE where the number of dimensions is the number of observables so it can get very high dimensional very fast. Classical methods don't necessarily scale that well in high dimensions (curse of dimensionality: cost is exponential in dimensions), but using neural networks does very well. This gives rise to the physics-informed neural network and deep backwards stochastic differential equation approaches which will likely be driving a lot of future HPC applications in a way that blends physical equations with neural network approaches. We recently released a library, NeuralPDE [1], which utilizes a lot of these approaches to solve what were traditionally difficult equations in an automated form. I think the future is bright for scientific machine learning! [1] https://neuralpde.sciml.ai/dev/ https://neuralpde.sciml.ai/dev/

wenc 6y ago
This is fascinating. ELI5: how does this work? (I'm couldn't find references on the linked site) Let's say I supply a high-dimensional DAE, f(x', x, z) = 0, x(0) = x₀, where classical methods like quadrature are unwieldy. Does the algorithm generate n samples in the solution space by integrating n times and then fitting an NN? With different initial conditions? Or does it perform quadrature with NNs instead of polynomial basis functions?
- ChrisRackauckas 6y ago
  A lot of these methods here utilize the universal differential equation framework described here: https://arxiv.org/abs/2001.04385 https://arxiv.org/abs/2001.04385 . Specifically, the last example in this preprint describes how high dimensional parabolic PDEs can be solving using neural networks inside of a specific SDE (derivation in the supplemental). Discrete physics-informed neural networks also are a subset of this methodology. The other subset of methods, continuous physics-informed neural networks, are described in https://www.sciencedirect.com/science/article/pii/S0021999118307125 https://www.sciencedirect.com/science/article/pii/S002199911... . For a very basic introduction, I wrote some lecture notes on how this is done for a simple ODE with code examples: https://mitmath.github.io/18S096SciML/lecture2/ml https://mitmath.github.io/18S096SciML/lecture2/ml

amelius 6y ago

Is this AI taking the place of preconditioners as used in iterative solvers? Would an AI be able to learn how to apply multigrid methods?

ChrisRackauckas 6y ago
We have prototypes in Julia for this. The answer is yes, there are tricks you can use to do this effectively.
- astroH 6y ago
  Could you elaborate a bit more on this? I have thought about employing NNs in this way for quite a while but the thing I never wrapped my head around was ensuring how it generalizes to different problems.

whinvik 6y ago

Almost every time a PDE is solved on a computer, it is a variational problem. Maybe neural networks are indeed good at this but I haven't seen any literature that shows that it is provably better. A reference would be good, especially to this point "But neural networks are very good parametric function approximators, generally better than what traditionally gets used in physics (b-splines or whatever)."

currymj 6y ago
https://arxiv.org/abs/1909.08423 https://arxiv.org/abs/1909.08423 and https://arxiv.org/abs/1909.02487 https://arxiv.org/abs/1909.02487 are some examples I've been looking at recently.
- whinvik 6y ago
  Thanks, not familiar with QM at all, but it seems to me from glancing through one of the papers that the neural network is used to replace a popular way of representing the wave function which itself is an Ansatz. Not very convincing, but of course, as I said not familiar with the background and so I may be overlooking something.
  - currymj 6y ago
    that's exactly it -- they take an existing form for the ansatz (or the general idea of it, at least), and make it more flexible by replacing pieces with neural networks that have many more parameters, while maintaining constraints required by physics. I think this will become very common in the future.
    - whinvik 6y ago
      That maybe true but what I was looking for is a more convincing way of showing that a neural network approximates a function better than other functions, such as say a b-spline. For example, you say that the neural network with many more parameters works better, but what if we had a b-spline with many more nodes.

noobermin 6y ago

Thank you. More importantly, this is not new. Sheesh, how much of AI is just hype. I am in favor in not using the term AI for such things.

baq 6y ago
> John McCarthy, the Father of AI, famously said: "As soon as it works, no one calls it AI any more." Leading researcher Rodney Brooks says "Every time we figure out a piece of it, it stops being magical; we say, 'Oh, that's just a computation. '" https://cacm.acm.org/blogs/blog-cacm/138907-john-mccarthy/fulltext https://cacm.acm.org/blogs/blog-cacm/138907-john-mccarthy/fu...
- Barrin92 6y ago
  this is an extremely overused fortune-cookie like quote. There's a legitimate distinction to be made between intelligence on one hand, and simply computation or calculation on the other. If we start calling every numerical method AI we're rendering the term meaningless.
  - glial 6y ago
    What’s the distinction?
    - Barrin92 6y ago
      In the most basic sense intelligence involves the aqcuisition of knowledge, which is representation or generalisation at some higher level of abstraction and the ability to make decisions. The mere ability to perform computational work is something virtually even the tiniest piece of hardware entails, or even an abacus for that matter.
    - LoSboccacc 6y ago
      intelligence is inductive finding correlations and using a human to filter the interesting ones from the flukes doesn't make the correlations engine an ai, the intelligence is still in the human

bencw 6y ago

This is something I've wondered about (along with potential applications of autograd outside of deep learning). Do you have a recommended starting point for someone who wants to learn more about this?

exabyte 6y ago

The power that I see in machine learning is the techniques being developed to handle the unavoidable noise in empirical data. I think that poses a large obstacle for traditional techniques although I am not familiar enough to compare.

willis936 6y ago
To me the value is in matching relationships (equations) of curated parameters from empirical data and using simulated recreations of the experiment as the objective. As soon as you can recreate experimental results in a simulation then you’ve made a successful model for that domain. This is an incredibly important and difficult task for fluid dynamics and plasma physics.
- stainforth 6y ago
  Fusion is right around the corner!

deleted 6y ago

[deleted]

mtgx 6y ago

Wouldn't a D-Wave quantum annealing computer be even better for this sort of thing?