3 ms·

Aren't we forced to require replication for a measure of truth in empirical studies? If that's universally accepted, shouldn't we demand that researchers come u

Aren't we forced to require replication for a measure of truth in empirical studies? If that's universally accepted, shouldn't we demand that researchers come up with predictions of the form "do X and see result Y"? And if we accept that, then should we not discuss who is going do actually "do X" before research gets published? From the outside perspective, it appears that a lot of people do "open-ended" experiments, write up some findings and pretty much no one tries to confirm the results with the notable exception of actual industrial applications (e.g., pharmacy or aircraft design), where the validation happens out of necessity.

pjc50 6y ago
It's important to notice that the funding, which determines what research gets done, is conditional on "impact factor" and publication rates. The classic case of letting a single metric drive a business function without regard for what it actually means. This has been incredibly bad for research, which now requires very large amounts of gaming the system to get actual science done. Reproducibility and other forms of validity are not currently factored in very well.
- asplake 6y ago
  A recipe for dysfunction not just in science but in any field
ketzu 6y ago
Preregistering studies [1] are common in some fields and a nice idea. Unfortunately my personal experience is that reproductibility, or even providing the sources for your experiments, is not something many reviewers care about in comp-sci fields. At least no conference review I received ever mentioned reproductibility no matter if it was available or not (and I am not sure anyone ever mentioned source code availability either). [1] https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science)
- rscho 6y ago
  Pre-registration is very common in medicine. I don't see it change anything, because there are no real consequences for not doing what you registered for.
  - analog31 6y ago
    Unless registration is an employment contract, it will be as binding and useful as project plans in business.
BeetleB 6y ago
> Aren't we forced to require replication for a measure of truth in empirical studies? Almost no scientific discipline requires it, sadly. There's usually no career prospects if you do replication studies.
seesawtron 6y ago
Substantially solid research takes years to complete. Each new finding opens possibilties of 10 new research directions that one "should" investigate. The timelines necessary for this follow up research studies blatantly exceed the scope of a Phd or Postdoc researcher and just one scientific paper. It is not true that no one tries to confirm the results. The labs do further studies internally to follow up. Also those working closely in the field in other monitor these "findings" and try to align them in their own research whether directly comparing them or indirectly using those ideas to support or reject their own observation. It is never one study but years of past research that lays the foundation for a scientific discovery that finally creates a paradigm shift in our understanding of the field.

pjc50 6y ago

It's important to notice that the funding, which determines what research gets done, is conditional on "impact factor" and publication rates. The classic case of letting a single metric drive a business function without regard for what it actually means. This has been incredibly bad for research, which now requires very large amounts of gaming the system to get actual science done. Reproducibility and other forms of validity are not currently factored in very well.

asplake 6y ago
A recipe for dysfunction not just in science but in any field

ketzu 6y ago

Preregistering studies [1] are common in some fields and a nice idea. Unfortunately my personal experience is that reproductibility, or even providing the sources for your experiments, is not something many reviewers care about in comp-sci fields. At least no conference review I received ever mentioned reproductibility no matter if it was available or not (and I am not sure anyone ever mentioned source code availability either). [1] https://en.wikipedia.org/wiki/Preregistration_(science) https://en.wikipedia.org/wiki/Preregistration_(science)

rscho 6y ago
Pre-registration is very common in medicine. I don't see it change anything, because there are no real consequences for not doing what you registered for.
- analog31 6y ago
  Unless registration is an employment contract, it will be as binding and useful as project plans in business.

BeetleB 6y ago

> Aren't we forced to require replication for a measure of truth in empirical studies? Almost no scientific discipline requires it, sadly. There's usually no career prospects if you do replication studies.

seesawtron 6y ago

Substantially solid research takes years to complete. Each new finding opens possibilties of 10 new research directions that one "should" investigate. The timelines necessary for this follow up research studies blatantly exceed the scope of a Phd or Postdoc researcher and just one scientific paper. It is not true that no one tries to confirm the results. The labs do further studies internally to follow up. Also those working closely in the field in other monitor these "findings" and try to align them in their own research whether directly comparing them or indirectly using those ideas to support or reject their own observation. It is never one study but years of past research that lays the foundation for a scientific discovery that finally creates a paradigm shift in our understanding of the field.