3 ms·

Very well explained to a lay person. Are PINNs the current state of the art in ML methods for solving PDEs? What are their limitations?

nchagnet 2y ago
Thank you! As far as I can tell, PINNs are promising and an active research area, but they are also young and far from being as widely adopted as finite element methods (at least that's my experience academic environments). I do see great improvements are being made both on the performance level but also on the applications. One aspect I didn't discuss in the post is the use for inverse solution search, where you fit experimental data to your equation, and where your parameters and your initial conditions can also be trainable parameters. This has great potential to improve the methodology of experimental results analysis.
cherryteastain 2y ago
> Are PINNs the current state of the art in ML methods for solving PDEs? What are their limitations? I guess in a way they are. They aren't new, they have been around since the 90s [1]. The problem with them is, you typically need to train them on a specific problem (boundary conditions, domain, equation, PDE coefficients etc). Compared to a traditional solver, the training is much slower, and on top of that the results are typically much less accurate. The PDE + NN community has a bit of a problem dealing with this in general [2], there are tons of papers that make NNs look much better at solving PDEs than they are compared to traditional solvers. [1] https://www.cs.uoi.gr/~lagaris/papers/TNN-LLF.pdf https://www.cs.uoi.gr/~lagaris/papers/TNN-LLF.pdf [2] https://www.nature.com/articles/s42256-024-00897-5 https://www.nature.com/articles/s42256-024-00897-5
hansvm 2y ago
It depends on the PDE and what you want to do with it. A PINN requires: 1. Some example data or other way to add boundary conditions 2. Autograd over PDE constraints 3. A training loop incorporating both of those And it produces a. An approximate, differentiable, mesh-free solution PINNs are most applicable when (1) is expensive (since that expense will apply more to traditional solvers, especially with fine meshes) and when the error in (a) is acceptable. Regarding the error, PINNs are still extremely useful in generating an initial state to pass to a traditional solver even when the error is not tolerable, so that's not _really_ a concern. The main consideration is how expensive a particular problem is to solve classically. If it's too cheap, the PINN will never beat it. You have a secondary consideration with (2) and (3). The training loop is a fixed cost which you can amortize over many executions, but you have to use the network enough times for that to actually pay off. The last point I want to bring up is that you can sometimes get value from the extra features in (a). Perhaps you want to use the PINN to figure out where your mesh should be finer, or you have a derived field you want to inspect. Neural net gradients in general tend to poorly approximate real gradients if you only train on the function itself, but PINNs have the gradients you're likely to care about baked into their definition (and can thus approximate them well), and they'll model those much more cheaply than traditional solvers will. We used them for a few things at my last job, and they were definitely worth it. We erred toward smaller (faster) nets with higher errors just to accelerate convergence with a classical solver.
- quanto 2y ago
  I get that PINN is a less expensive approximate solution method. If so, how does it perform superior to many approximate, coarse numerical methods?
  - hansvm 2y ago
    1. Those methods are coarse. The interpolation they provide is worse than what a PINN provides, meaning that equivalently performing PINNs (compard to coarse numerical methods) can easily and cheaply serve as better initializations for your finer numerical methods. 2. Go back to (1) from my previous message. For some intuition, fiddly solutions take a long time to optimize. Your only options (aside from spending more time and money) are tailoring the initial conditions and the algorithm for your particular problem. You see that a lot in, e.g., 1-3 atom quantum chemistry, where a good choice of basis functions is worth several papers. A neural network allows you to automagically bake everything that's hard about your problem into the training step and amortize those hard calculations across many experiments. It's not superior to enough man-centuries of human intuition, but it's dead simple to deploy, and for those sorts of hard problems it definitely beats a single human century of effort. Once you have a neural network output, the problem is well conditioned and suitable for refinement by a classical solver. For a somewhat concrete example, imagine a problem where the space is largely uninteresting but there are a few tight swirls here and there. Coarse numerical methods can't really do anything with those. Adaptive-precision numerical methods can, but they're slow, and you have to re-run an intensive solving step for every new input. The PINN solution bakes everything that's hard about that into the neural net structure, and it solution will have approximately the right swirls in approximately the right places. If you want to refine them further, the fact that your solver doesn't have to dynamically handle resolution anymore and doesn't have to deal with any major phase shifts makes it much easier to iterate on via the normal classical methods.
    - quanto 2y ago
      Many thanks for your detailed input. For your concrete example, existing solutions employ FEM. My understanding of your point is that a NN abstracts away the meshing rules and learns the correct resolution in the areas of interest? I could see how this could be beneficial for quick solutions before a full blown solver. If my above understanding is correct, than the following question is, why not use a NN to generate meshes directly? Let the classical solvers do what they do best: solve. Let NN do what they do best: take care of messy reality of geometry. This approach would actually give provable error bounds on the solution. I understand there are existing works on NN mesh generation, but I do not know any work that proves error bounds or has been incorporated into mainstream engineering software. Any hints? (Thanks for this fascinating discussion.)

nchagnet 2y ago

Thank you! As far as I can tell, PINNs are promising and an active research area, but they are also young and far from being as widely adopted as finite element methods (at least that's my experience academic environments). I do see great improvements are being made both on the performance level but also on the applications. One aspect I didn't discuss in the post is the use for inverse solution search, where you fit experimental data to your equation, and where your parameters and your initial conditions can also be trainable parameters. This has great potential to improve the methodology of experimental results analysis.

cherryteastain 2y ago

> Are PINNs the current state of the art in ML methods for solving PDEs? What are their limitations? I guess in a way they are. They aren't new, they have been around since the 90s [1]. The problem with them is, you typically need to train them on a specific problem (boundary conditions, domain, equation, PDE coefficients etc). Compared to a traditional solver, the training is much slower, and on top of that the results are typically much less accurate. The PDE + NN community has a bit of a problem dealing with this in general [2], there are tons of papers that make NNs look much better at solving PDEs than they are compared to traditional solvers. [1] https://www.cs.uoi.gr/~lagaris/papers/TNN-LLF.pdf https://www.cs.uoi.gr/~lagaris/papers/TNN-LLF.pdf [2] https://www.nature.com/articles/s42256-024-00897-5 https://www.nature.com/articles/s42256-024-00897-5

hansvm 2y ago

It depends on the PDE and what you want to do with it. A PINN requires: 1. Some example data or other way to add boundary conditions 2. Autograd over PDE constraints 3. A training loop incorporating both of those And it produces a. An approximate, differentiable, mesh-free solution PINNs are most applicable when (1) is expensive (since that expense will apply more to traditional solvers, especially with fine meshes) and when the error in (a) is acceptable. Regarding the error, PINNs are still extremely useful in generating an initial state to pass to a traditional solver even when the error is not tolerable, so that's not _really_ a concern. The main consideration is how expensive a particular problem is to solve classically. If it's too cheap, the PINN will never beat it. You have a secondary consideration with (2) and (3). The training loop is a fixed cost which you can amortize over many executions, but you have to use the network enough times for that to actually pay off. The last point I want to bring up is that you can sometimes get value from the extra features in (a). Perhaps you want to use the PINN to figure out where your mesh should be finer, or you have a derived field you want to inspect. Neural net gradients in general tend to poorly approximate real gradients if you only train on the function itself, but PINNs have the gradients you're likely to care about baked into their definition (and can thus approximate them well), and they'll model those much more cheaply than traditional solvers will. We used them for a few things at my last job, and they were definitely worth it. We erred toward smaller (faster) nets with higher errors just to accelerate convergence with a classical solver.

quanto 2y ago
I get that PINN is a less expensive approximate solution method. If so, how does it perform superior to many approximate, coarse numerical methods?
- hansvm 2y ago
  1. Those methods are coarse. The interpolation they provide is worse than what a PINN provides, meaning that equivalently performing PINNs (compard to coarse numerical methods) can easily and cheaply serve as better initializations for your finer numerical methods. 2. Go back to (1) from my previous message. For some intuition, fiddly solutions take a long time to optimize. Your only options (aside from spending more time and money) are tailoring the initial conditions and the algorithm for your particular problem. You see that a lot in, e.g., 1-3 atom quantum chemistry, where a good choice of basis functions is worth several papers. A neural network allows you to automagically bake everything that's hard about your problem into the training step and amortize those hard calculations across many experiments. It's not superior to enough man-centuries of human intuition, but it's dead simple to deploy, and for those sorts of hard problems it definitely beats a single human century of effort. Once you have a neural network output, the problem is well conditioned and suitable for refinement by a classical solver. For a somewhat concrete example, imagine a problem where the space is largely uninteresting but there are a few tight swirls here and there. Coarse numerical methods can't really do anything with those. Adaptive-precision numerical methods can, but they're slow, and you have to re-run an intensive solving step for every new input. The PINN solution bakes everything that's hard about that into the neural net structure, and it solution will have approximately the right swirls in approximately the right places. If you want to refine them further, the fact that your solver doesn't have to dynamically handle resolution anymore and doesn't have to deal with any major phase shifts makes it much easier to iterate on via the normal classical methods.
  - quanto 2y ago
    Many thanks for your detailed input. For your concrete example, existing solutions employ FEM. My understanding of your point is that a NN abstracts away the meshing rules and learns the correct resolution in the areas of interest? I could see how this could be beneficial for quick solutions before a full blown solver. If my above understanding is correct, than the following question is, why not use a NN to generate meshes directly? Let the classical solvers do what they do best: solve. Let NN do what they do best: take care of messy reality of geometry. This approach would actually give provable error bounds on the solution. I understand there are existing works on NN mesh generation, but I do not know any work that proves error bounds or has been incorporated into mainstream engineering software. Any hints? (Thanks for this fascinating discussion.)