3 ms·
I see, thank you for clarifying. I misread your comment, but if it is a general comment about the concern for version history of code that supports a scientific
by p4wnc6 11y ago
I see, thank you for clarifying. I misread your comment, but if it is a general comment about the concern for version history of code that supports a scientific conclusion, then I absolutely withdraw my comments about it being insulting or personal.
Without the history to prove exactly what happened and when, there is no reproducibility.
I would even go further and say that each and every model fitting attempt should be recorded in some type of version history. This way, researchers can't engage in multiple comparison errors, whereby they test lots of things and fish for a specific threshold of significance that may be met solely by chance based on how many things they tried, or researcher degrees of freedom problems where seeing the output of intermediate model fits causes you to change the experimental design mid-stream (like collecting more data in the context of a model where that would change the required significance threshold).
These kinds of pitfalls are extremely common and go unreported most of the time. If someone found a significant relationship after testing hundreds of things and fine tuning the data collection, even if that result is statistically significant we should deeply discount it. But what if there is no record anywhere that all those previous attempts took place and that they motivated the experimenter to tweak the experimental design?