4 ms·

Thanks for this submission. I hadn't seen it. A somewhat-related recent study is this National Academies report on simulation-guided discovery systems: https:/

Thanks for this submission. I hadn't seen it. A somewhat-related recent study is this National Academies report on simulation-guided discovery systems: https://www.nationalacademies.org/our-work/realizing-opportunities-for-advanced-and-automated-workflows-in-scientific-research https://www.nationalacademies.org/our-work/realizing-opportu... OP is a long read and suffers from being something of a catalog of promising ideas -- so it's hard to digest. But from all the examples given, you do get the feeling there's something really powerful and new coming into being that combines the large datasets you can obtain from simulation with machine learning/stats tools to build models, find posteriors, etc. In science applications of remote sensing, my group at NASA/JPL has obtained huge speedups from replacing computationally-expensive physics-based forward models with emulators based on ANNs or GPs. In terms of OP, this is "Motif 2" (surrogates/emulators) combined with fitting based on "Motif 7" (differentiable programming). You build a training set using selected runs of the physics-based forward model, and train an emulator that links (say) at-sensor radiances with ground or atmosphere conditions. Roughly a 400-variable to 400-variable function emulator. Then you use the emulator for (say) each 30m x 30m pixel of a global satellite dataset, instead of running the forward model for each such pixel. Replacing the expensive forward model can reduce workload for a NASA science mission by factors of 10 or more. This capability really didn't exist 10 years ago, even though a lot of the algorithms were almost there, and the computational capacity was almost there. It's some kind of capacity-building thing where you have to have competence across several kinds of "data science" in order to make a system that's effective.

JBorrow 4y ago
Similar things are happening in galaxy formation too; numerical models here cost tens of thousands - tens of millions of cpu hours to run, and have maybe 8-10 free parameters. We use emulators based upon a training set of maybe 100-200 real simulations to calibrate out those free parameters to scaling relations (e.g. sizes of galaxies v.s. their mass). Odd that they did not mention these when discussing sub-grid modelling, as afaik these Astronomy simulations (along with planetary impacts) are probably some of the first to have used them…
- dreamcompiler 4y ago
  Galaxy formation is inherently chaotic i.e. it exhibits sensitive dependence to initial conditions. My first thought upon reading your comment is that an ANN approach must be wrong because of chaos. But of course a purely numerical gravitational simulation will also be wrong because of chaos. So how do you tell which method is ... less wrong?
  - JBorrow 4y ago
    Ah, we use the ML methods to understand what happens to a large collection of galaxies (maybe 100k - 1 million), all co-evolving together. Trying to use these methods v.s. direct simulation on individual galaxies is a no-go, too many nonlinearities exist. Your comment about a numerical gravitational simulation being 'wrong' because of chaos may be true on an individual resolution element level, yes, but our simulations are made up of many billions of individual particles. Like traditional CFD simulations we can also demonstrate some level of convergence.
    - dreamcompiler 4y ago
      Thanks. I suspected the answer might be "we're only concerned with characterizing extremely macroscopic behavior" and you confirmed that.
whatever1 4y ago
I do not understand what is new. Surrogate modeling for complex physics is a thing for the past 100 years? Why use overfitting black box ANN when we know how to it with well behaved explicit functions?
- danielmarkbruce 4y ago
  Way faster
- mturmon 4y ago
  You raise several points which I’ll try to address. We did develop a surrogate model for this problem, see the narrative and references below eq. 1 of [1]. The surrogate is perhaps 10x faster than the full physics model. It was developed by a postdoc working with the science team and took a few months. (We needed a lot of science team expertise because they advised on what computations could be simplified and what parameters could be lumped together.) We use the surrogate for various Monte Carlo analyses, parameter sweeps, etc. And for MCMC as described in [1]. All tasks that the full physics model is too slow for. But the emulator, developed more recently, is much, much faster than the surrogate. It did not require as much science team expertise to develop, it was more of a data science problem than an integrated physics/data science task. So it was faster to put together. The increased speed of the emulator means that all those parameter sweeps and MCMC inversions can be much broader in scope. Furthermore, the emulator is somewhat generic. We have used the same approach in other spectroscopic contexts. E.g., [2], for imaging spectroscopy, and there is a still-unpublished infrared water-vapor/cloud sounder example. It seems like generic emulation for a broad class of far IR to UV spectroscopy radiative transfer is possible, without needing problem-specific physical modeling expertise to find model simplifications. (We have direct access to this expertise, it’s just that those folks have their own research to do.) So: faster than a surrogate, broad in scope, faster turnaround. About error control. We’re using a Gaussian process with a learned kernel, so we do get error estimates out of the approach, if we want to use them. See under “kernel flows” at [3]. [1] https://www.mdpi.com/2072-4292/11/17/2061 https://www.mdpi.com/2072-4292/11/17/2061 [2] https://ui.adsabs.harvard.edu/abs/2021AGUFMNG25A0506S/abstract https://ui.adsabs.harvard.edu/abs/2021AGUFMNG25A0506S/abstra... [3] http://users.cms.caltech.edu/~owhadi/research.htm http://users.cms.caltech.edu/~owhadi/research.htm

JBorrow 4y ago

Similar things are happening in galaxy formation too; numerical models here cost tens of thousands - tens of millions of cpu hours to run, and have maybe 8-10 free parameters. We use emulators based upon a training set of maybe 100-200 real simulations to calibrate out those free parameters to scaling relations (e.g. sizes of galaxies v.s. their mass). Odd that they did not mention these when discussing sub-grid modelling, as afaik these Astronomy simulations (along with planetary impacts) are probably some of the first to have used them…

dreamcompiler 4y ago
Galaxy formation is inherently chaotic i.e. it exhibits sensitive dependence to initial conditions. My first thought upon reading your comment is that an ANN approach must be wrong because of chaos. But of course a purely numerical gravitational simulation will also be wrong because of chaos. So how do you tell which method is ... less wrong?
- JBorrow 4y ago
  Ah, we use the ML methods to understand what happens to a large collection of galaxies (maybe 100k - 1 million), all co-evolving together. Trying to use these methods v.s. direct simulation on individual galaxies is a no-go, too many nonlinearities exist. Your comment about a numerical gravitational simulation being 'wrong' because of chaos may be true on an individual resolution element level, yes, but our simulations are made up of many billions of individual particles. Like traditional CFD simulations we can also demonstrate some level of convergence.
  - dreamcompiler 4y ago
    Thanks. I suspected the answer might be "we're only concerned with characterizing extremely macroscopic behavior" and you confirmed that.

whatever1 4y ago

I do not understand what is new. Surrogate modeling for complex physics is a thing for the past 100 years? Why use overfitting black box ANN when we know how to it with well behaved explicit functions?

danielmarkbruce 4y ago
Way faster
mturmon 4y ago
You raise several points which I’ll try to address. We did develop a surrogate model for this problem, see the narrative and references below eq. 1 of [1]. The surrogate is perhaps 10x faster than the full physics model. It was developed by a postdoc working with the science team and took a few months. (We needed a lot of science team expertise because they advised on what computations could be simplified and what parameters could be lumped together.) We use the surrogate for various Monte Carlo analyses, parameter sweeps, etc. And for MCMC as described in [1]. All tasks that the full physics model is too slow for. But the emulator, developed more recently, is much, much faster than the surrogate. It did not require as much science team expertise to develop, it was more of a data science problem than an integrated physics/data science task. So it was faster to put together. The increased speed of the emulator means that all those parameter sweeps and MCMC inversions can be much broader in scope. Furthermore, the emulator is somewhat generic. We have used the same approach in other spectroscopic contexts. E.g., [2], for imaging spectroscopy, and there is a still-unpublished infrared water-vapor/cloud sounder example. It seems like generic emulation for a broad class of far IR to UV spectroscopy radiative transfer is possible, without needing problem-specific physical modeling expertise to find model simplifications. (We have direct access to this expertise, it’s just that those folks have their own research to do.) So: faster than a surrogate, broad in scope, faster turnaround. About error control. We’re using a Gaussian process with a learned kernel, so we do get error estimates out of the approach, if we want to use them. See under “kernel flows” at [3]. [1] https://www.mdpi.com/2072-4292/11/17/2061 https://www.mdpi.com/2072-4292/11/17/2061 [2] https://ui.adsabs.harvard.edu/abs/2021AGUFMNG25A0506S/abstract https://ui.adsabs.harvard.edu/abs/2021AGUFMNG25A0506S/abstra... [3] http://users.cms.caltech.edu/~owhadi/research.htm http://users.cms.caltech.edu/~owhadi/research.htm