3 ms·

Non-scientist here. Isn't this just stating that "experiments", even those involving non-laboratory measurements, need to begin with a falsifiable hypothesis? F

Non-scientist here. Isn't this just stating that "experiments", even those involving non-laboratory measurements, need to begin with a falsifiable hypothesis? From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand.

Frost1x 3y ago
>From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand. Oh my sweet summer child, a lot of lab work and data collection is expensive and in the game of research, you spend a lot of time gaming the system and meeting expectations relative to doing actual fundamental research. So much work wants to take the hard low return work like doing tests and collecting data and releveraging it with increasingly complex statistical approaches. I've worked in so many research environments that you find more often the case is that there's a selection towards research that could be pursued and falsified with existing data vs the other way around. Here's this set of data and how it was collected, what arbitrarily new novel thing can we say about it? It may not be something interesting but it may be statistically or theoretically valid. The result is you get a paper/publication out of it without doing the footwork. This is part of the reason researchers often hold their data tightly. You'd think scientists would want to share data but it's a highly competitive environment and if you took the risk to invest time and money in some costly data collection process, you want to do everything you can to say everything you can about it before someone else does it without any of the underlying cost. Sure, you may get a reference or footnote for your data but that's not going to help that much in the big scheme of things, not as much as a fresh publication. Also, if you're only being referenced for the data collection portion of your work... it doesn't speak alot about the work you did around that data collection.
- jewayne 3y ago
  But doesn't that explain the reproducibility crisis, in a nutshell? If you work backwards from a dataset and look for correlations, doesn't that effectively set the p-value to 1?
  - Frost1x 3y ago
    I'm not disagreeing with you, just pointing out how reality diverges from how science is often taught to be practiced as this innocent discovery process. The structure behind research and funding in research, which leads towards all sorts of system gaming, is definitely a core contributor to reproducibility but it's not the the only issue. A lot of issues are in sheer variability of things being studied and how context sensitive some things can be. Combine that with limited time, funding, and expectation to produce positive results in some weird form and you yet a lot of the reproducibility issues out there. Another huge factor beyond just analytic approach is analytic follow through. There's a lot of highly questionable computational code out there that researchers earnestly believe is doing something that it's actually not.
  - readthenotes1 3y ago
    Explaining the reproducibility crisis: Pride, greed, envy, sloth. (Alt) Explaining the reproducibility crisis: acquisitiveness, rivalry, vanity, power-seeking. (Alt) Explaining the reproducibility crisis: being human.

Frost1x 3y ago

>From my understanding, you can't draw any real conclusions from the data unless you predicted that it would look that way beforehand. Oh my sweet summer child, a lot of lab work and data collection is expensive and in the game of research, you spend a lot of time gaming the system and meeting expectations relative to doing actual fundamental research. So much work wants to take the hard low return work like doing tests and collecting data and releveraging it with increasingly complex statistical approaches. I've worked in so many research environments that you find more often the case is that there's a selection towards research that could be pursued and falsified with existing data vs the other way around. Here's this set of data and how it was collected, what arbitrarily new novel thing can we say about it? It may not be something interesting but it may be statistically or theoretically valid. The result is you get a paper/publication out of it without doing the footwork. This is part of the reason researchers often hold their data tightly. You'd think scientists would want to share data but it's a highly competitive environment and if you took the risk to invest time and money in some costly data collection process, you want to do everything you can to say everything you can about it before someone else does it without any of the underlying cost. Sure, you may get a reference or footnote for your data but that's not going to help that much in the big scheme of things, not as much as a fresh publication. Also, if you're only being referenced for the data collection portion of your work... it doesn't speak alot about the work you did around that data collection.

jewayne 3y ago
But doesn't that explain the reproducibility crisis, in a nutshell? If you work backwards from a dataset and look for correlations, doesn't that effectively set the p-value to 1?
- Frost1x 3y ago
  I'm not disagreeing with you, just pointing out how reality diverges from how science is often taught to be practiced as this innocent discovery process. The structure behind research and funding in research, which leads towards all sorts of system gaming, is definitely a core contributor to reproducibility but it's not the the only issue. A lot of issues are in sheer variability of things being studied and how context sensitive some things can be. Combine that with limited time, funding, and expectation to produce positive results in some weird form and you yet a lot of the reproducibility issues out there. Another huge factor beyond just analytic approach is analytic follow through. There's a lot of highly questionable computational code out there that researchers earnestly believe is doing something that it's actually not.
- readthenotes1 3y ago
  Explaining the reproducibility crisis: Pride, greed, envy, sloth. (Alt) Explaining the reproducibility crisis: acquisitiveness, rivalry, vanity, power-seeking. (Alt) Explaining the reproducibility crisis: being human.