3 ms·

My feeling is that this reproducibility crisis points to deeper stuff just waiting to be collectively understood about how to use math in experimental sciences.

My feeling is that this reproducibility crisis points to deeper stuff just waiting to be collectively understood about how to use math in experimental sciences. So many researchers rely on assumptions of linear dynamics. So many experiments and studies are designed without consideration of observer effects. Is it any wonder that a model might fit one day but not the next? There is so much to be learned from applied topology, dynamical systems theory and (quantum) information theory, but methods from these disciplines are only just barely starting to become more widely accessible.

skybrian 7y ago
Making the model more complicated doesn't add external validity. It seems like the problem often comes from trying to make do with collecting a minimum amount of data, collected in the most convenient way. (For example, doing experiments on local college students.) It seems like a practice of at least collecting the data at two different colleges might help? But this would require more coordination.
- JorgeGT 7y ago
  The whole, complex scientific field of metrology exists in order to make physical measurements anywhere on Earth comparable in a meaningful way. Trying to derive conclusions about the psychological traits of humans by observing n=200 of your local college students certainly seems... rather dubious.
- macawfish 7y ago
  Just because alternative methods aren't widespread, it doesn't necessarily mean they are "more complicated", just that rely on different assumptions. I'm really excited, for example, about "model-free" time series methods emerging from non-linear topological data analysis. Takens' embedding theorem shows that low dimensional "shadows" of high dimensional attractors can be constructed from a time series alone, and that they can reveal deep, useful facts about system a time series is part of, even if that system defies geometric modeling. These kinds of methods are radically different from conventional statistical methods. see here: https://www.youtube.com/watch?v=NrFdIz-D2yM https://www.youtube.com/watch?v=NrFdIz-D2yM Previously, researchers found themselves flailing around with geometry, trying to come up with models that accurately described the dynamical nature of the systems they were studying (e.g. fish populations). A model might work well for a few years, then stop working. These topological methods cut through and get straight to the underlying relationships that drive various observables in a complex system, without relying on fickle geometric assumptions about how exactly those relationships will be expressed.

skybrian 7y ago

Making the model more complicated doesn't add external validity. It seems like the problem often comes from trying to make do with collecting a minimum amount of data, collected in the most convenient way. (For example, doing experiments on local college students.) It seems like a practice of at least collecting the data at two different colleges might help? But this would require more coordination.

JorgeGT 7y ago
The whole, complex scientific field of metrology exists in order to make physical measurements anywhere on Earth comparable in a meaningful way. Trying to derive conclusions about the psychological traits of humans by observing n=200 of your local college students certainly seems... rather dubious.
macawfish 7y ago
Just because alternative methods aren't widespread, it doesn't necessarily mean they are "more complicated", just that rely on different assumptions. I'm really excited, for example, about "model-free" time series methods emerging from non-linear topological data analysis. Takens' embedding theorem shows that low dimensional "shadows" of high dimensional attractors can be constructed from a time series alone, and that they can reveal deep, useful facts about system a time series is part of, even if that system defies geometric modeling. These kinds of methods are radically different from conventional statistical methods. see here: https://www.youtube.com/watch?v=NrFdIz-D2yM https://www.youtube.com/watch?v=NrFdIz-D2yM Previously, researchers found themselves flailing around with geometry, trying to come up with models that accurately described the dynamical nature of the systems they were studying (e.g. fish populations). A model might work well for a few years, then stop working. These topological methods cut through and get straight to the underlying relationships that drive various observables in a complex system, without relying on fickle geometric assumptions about how exactly those relationships will be expressed.