3 ms·

> But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participant

> But a much stronger and more relevant form of reproducibility for actually advancing science is running the same study e.g. on different groups of participants (or in computer science / applied math/stats / etc., with different codebases, with different model variants/hyperparameters, on different datasets) and the overall conclusions hold > Plenty of good science got done before modern devops came to be This isn't as strong of an argument as you think. This is more-or-less the underlying foundation behind the social sciences, which argues that no social sampling can ever be entirely reproduced since no two people are alike, and even the same person cannot be reliably sampled twice as people change with time. Has there been "good science" done in the social sciences? Sure. I don't think that you're going to find anybody arguing that the state of the social sciences today is about the same as it was in the Dark Ages. With that said, one of the reasons why so many laypeople look at the social sciences as a kind of joke is because so many contradictory studies come out of these peer-reviewed journals that their trustworthiness is quite low. One of the reasons why there's so much confusion surrounding what constitutes a healthy diet and how people should best attempt to lose weight is precisely because diet-and-exercise studies are more-or-less impossible to reproduce. > If you can achieve that, great -- it's certainly a useful property to have for debugging If you can achieve that, for the area of study in which you conduct your experiment, it should be required. Deciding to forego formal reproducibility should be justified with a clear explanation as to why reproducibility is infeasible for your experiment, and peer-review should reject studies that could have be reproducible but weren't in practice.

jbay808 6y ago
Plenty of good physics got done before modern devops came to be, too! Maybe the pace of advancement was slower when the best practice was to publish a cryptographic hash of your discoveries in the form of a poetic latin anagram rather than just straight-up saying it, but it's not like Hooke's law is considered unreproducible today because you can't deterministically re-instantiate his experimental setup with a centuries-old piece of brass and get the same result to n significant figures.
- mnl 6y ago
  And physicists have been writing code for a while simply because the number of software engineers with a working knowledge of physics (as in ready for research), have been trained in numerical analysis (as in being able to read applied mathematics) and then are willing to help you with your paper for peanuts is about zero. I don't understand why it is so hard to see that you need either a pretty big collaboration where somebody else has isolated the specifications so you don't need to know anything about the problem your code solves really, or becoming a physics graduate student yourself for this line of work.
a1369209993 6y ago
I wouldn't argue for it, but I would be extremely reluctant to argue against the assertion that the state of the social 'sciences' today is about the same as it was in the Dark Ages. To the extent that any "good science" gets done in the social 'sciences', it is entirely in spite of the entire (social-'science'-specific) underlying foundation thereof. If your results aren't reproducible with (the overwhelming majority of[0]) other samples collected based on the same criteria, your results aren't. 0: specifically, for results claiming a 95% confidence level (p<0.05), if nineteen replications are attempted, you should encounter a replication failure exactly once. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).

jbay808 6y ago

Plenty of good physics got done before modern devops came to be, too! Maybe the pace of advancement was slower when the best practice was to publish a cryptographic hash of your discoveries in the form of a poetic latin anagram rather than just straight-up saying it, but it's not like Hooke's law is considered unreproducible today because you can't deterministically re-instantiate his experimental setup with a centuries-old piece of brass and get the same result to n significant figures.

mnl 6y ago
And physicists have been writing code for a while simply because the number of software engineers with a working knowledge of physics (as in ready for research), have been trained in numerical analysis (as in being able to read applied mathematics) and then are willing to help you with your paper for peanuts is about zero. I don't understand why it is so hard to see that you need either a pretty big collaboration where somebody else has isolated the specifications so you don't need to know anything about the problem your code solves really, or becoming a physics graduate student yourself for this line of work.

a1369209993 6y ago

I wouldn't argue for it, but I would be extremely reluctant to argue against the assertion that the state of the social 'sciences' today is about the same as it was in the Dark Ages. To the extent that any "good science" gets done in the social 'sciences', it is entirely in spite of the entire (social-'science'-specific) underlying foundation thereof. If your results aren't reproducible with (the overwhelming majority of[0]) other samples collected based on the same criteria, your results aren't. 0: specifically, for results claiming a 95% confidence level (p<0.05), if nineteen replications are attempted, you should encounter a replication failure exactly once. I would accept perhaps four or five out of nineteen (or one out of two or three) under the reasoning that the law of large numbers hasn't kicked in yet, but anything with zero successful replications is not science, it's evidence (in this, against the entire field of study).