5 ms·

There's also data in another source: domain knowledge and physical laws. This is the core tenant of scientific machine learning techniques (SciML). I have a tal

There's also data in another source: domain knowledge and physical laws. This is the core tenant of scientific machine learning techniques (SciML). I have a talk that walks through how you can go from something without domain knowledge (like a neural ODE) and how as you add more prior model knowledge to the system the extrapolation accuracy improves even for small amounts of data (https://www.youtube.com/watch?v=FihLyzdjN_8 https://www.youtube.com/watch?v=FihLyzdjN_8). People have used these techniques in all kinds of places, like extrapolating black hole trajectories and building earthquake-safe buildings, off of like 20-40 data points (https://www.youtube.com/watch?v=eSeY4K4bITI https://www.youtube.com/watch?v=eSeY4K4bITI). While there is a lot of work to be done in the domain still, there's a lot of empirical evidence that this approach does indeed decrease the data requirements (but is then no longer "pure" machine learning in some sense)

joshgev 4y ago
Damn, that was a really cool talk. Nice work and thanks for sharing.
- residual 4y ago
  I see two links. Which one did you think was cool?
dekhn 4y ago
why would you build an earthquake-safe building using only 20-40 data points (sorry, I skimmed the video and I dont think it answered that question). That sounds irresponsible since you could collect far more data points, and unless you could demonstrate that collecting tiny amounts of data was "safe", you're always going to collect high quality data and lots of it. The advantage of physical systems is that it's straightforward to collect data from natural systems.
- ChrisRackauckas 4y ago
  > The advantage of physical systems is that it's straightforward to collect data from natural systems. For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.
  - dekhn 4y ago
    yes, but it's still unclear that for data-poor physics problems (the joke in grad school is that you could graduate when you collected your second data point, since that was enough to draw a straight line, and we all know any interesting physical phenomenon is linear in the engineering regime, right?) that NNs really provide anything more than a bayesian prior that physics folks already knew how to incorporate. High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.
    - uoaei 4y ago
      The general principle can be referred to as "inductive bias": out of the space of all functions that can be represented by computational means, how does one restrict the search space to only those functions which are relevant for the data/system of study? Developing physical theories is nothing more than the process of whittling down what space of functions can represent the data most accurately, using analytical and scientific techniques to get there. Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.
- crabbygrabby 4y ago
  When you can model a system using physical or known principles the data points are mostly there to prove the model is valid. Also they usually aren't randomly chosen data points like in ML they are chosen via statistical theory. The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD. I've made models of physical systems that involve kind of life or death stuff with less than 30 samples. It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!
  - dekhn 4y ago
    can you share more about models of life and death stuff with less than 30 samples? (I work tangential to a regulated health industry)
    - crabbygrabby 4y ago
      I'd like to stay anonymous on the matter, but I've worked in various regulated industries. Now days I do none of this .
      - dekhn 4y ago
        Uh, OK. because if you were doing life and death stuff with 30 samples, people probably died due to false negatives or positives in your model, in a way that would have been better handled with more samples. Sorry to hear it- it's one of the reasons health care is so shitty.
        crabbygrabby 4y ago
        Not healthcare and no one died. My models have been deployed for two years without incident.
        dekhn 4y ago
idealmedtech 4y ago
Are we coming full circle back to Knowledge-Based Systems of the 1980s, with a sprinkling of NN?
- ianandrich 4y ago
  I think thats a strong possibility.

joshgev 4y ago

Damn, that was a really cool talk. Nice work and thanks for sharing.

residual 4y ago
I see two links. Which one did you think was cool?

dekhn 4y ago

why would you build an earthquake-safe building using only 20-40 data points (sorry, I skimmed the video and I dont think it answered that question). That sounds irresponsible since you could collect far more data points, and unless you could demonstrate that collecting tiny amounts of data was "safe", you're always going to collect high quality data and lots of it. The advantage of physical systems is that it's straightforward to collect data from natural systems.

ChrisRackauckas 4y ago
> The advantage of physical systems is that it's straightforward to collect data from natural systems. For a lot of physical systems it might be "straightforward" but expensive. Testing new building designs for energy efficiency is just a matter of building a new test building, why not just build 1000? Testing new car designs to see if they reach the safety qualifications is just a matter of crashing a few of your prototypes, why not crash 1000? It's just a matter of getting more telescope time. It's just a matter of getting more robots to try more materials or proteins. Yes you can do theoretically do it, it's just cost prohibitive to get Google-level datasets for many scientific and engineering problems. Of course that's not true for all domains (high-throughput sequencing for bioinformatics is the prime example of something that became cheaply data-rich for standard machine learning), but there are many domains where getting another data point is always possible but just would cost another million.
- dekhn 4y ago
  yes, but it's still unclear that for data-poor physics problems (the joke in grad school is that you could graduate when you collected your second data point, since that was enough to draw a straight line, and we all know any interesting physical phenomenon is linear in the engineering regime, right?) that NNs really provide anything more than a bayesian prior that physics folks already knew how to incorporate. High throughput sequeuncing didn't solve any problems for bioinformatics- in fact, the problem is that we have so much data we don't know how to process it to extract the actual value from the data.
  - uoaei 4y ago
    The general principle can be referred to as "inductive bias": out of the space of all functions that can be represented by computational means, how does one restrict the search space to only those functions which are relevant for the data/system of study? Developing physical theories is nothing more than the process of whittling down what space of functions can represent the data most accurately, using analytical and scientific techniques to get there. Bioinformaticists, with sufficiently accurate and detailed models of how genes translate to $BEHAVIOR_OR_OUTCOME, would need far fewer data to extract meaningful insights. So of course throughput hasn't solved the problem. That wasn't the point of the comment above.
crabbygrabby 4y ago
When you can model a system using physical or known principles the data points are mostly there to prove the model is valid. Also they usually aren't randomly chosen data points like in ML they are chosen via statistical theory. The disadvantage of physical systems is the data is usually expensive. In my old field I know someone who had to make a decision on a data set of 5 samples and each sample costed more than a million USD. I've made models of physical systems that involve kind of life or death stuff with less than 30 samples. It's not for the faint of heart and if someone doesn't know what they are doing, the system they are modelling, or how to share their results correctly... Yikes!
- dekhn 4y ago
  can you share more about models of life and death stuff with less than 30 samples? (I work tangential to a regulated health industry)
  - crabbygrabby 4y ago
    I'd like to stay anonymous on the matter, but I've worked in various regulated industries. Now days I do none of this .
    - dekhn 4y ago
      Uh, OK. because if you were doing life and death stuff with 30 samples, people probably died due to false negatives or positives in your model, in a way that would have been better handled with more samples. Sorry to hear it- it's one of the reasons health care is so shitty.
      - crabbygrabby 4y ago
        Not healthcare and no one died. My models have been deployed for two years without incident.
        dekhn 4y ago

idealmedtech 4y ago

Are we coming full circle back to Knowledge-Based Systems of the 1980s, with a sprinkling of NN?

ianandrich 4y ago
I think thats a strong possibility.