5 ms·

>Science can. But can we though?

guscost 6y ago
Only after you have co-authored several papers with the very people you are trying to question, and earned a credential, it seems.
- AstralStorm 6y ago
  Not really. First you have to understand the paper in question, so you need at least partial background. Then you would have to find the authors. Many sensational paper authors or citation hunters don't like to answer critical questions. Nor is often raw data/algorithms/notes published so you can check if there is a problem.
  - guscost 6y ago
    That practice of “not publishing all code and data” really sets off the BS detector.
    - roel_v 6y ago
      Are you a scientist? Because it really doesn't. For example, in most papers that I think harder than normal about, I don't care so much about 'the data' as I do about a description of what this 'data' is actually meant to represent, and how it was collected and processed. (I'm talking here about things that are a bit more complicated than e.g. railfall measurements or any other such lab-like, STEM topics) E.g., I do quite a bit of population modeling. You would think that 'population' is relatively easy to quantify, but it really isn't, and I can talk for hours about how cavalier people throw their 'population data' numbers into models and make all sorts of conclusions based on objectively wrong interpretations what this 'data' is. If the vast majority of papers can't even get that right, I don't care as much about the HN idea of what 'reproducibility' is - i.e. check if 'git clone <xyz> && run_model.sh && run_tests.sh' says 'All OK!' at the end.
      - 6gvONxR4sf7o 6y ago
        I agree in part here, but think it doesn’t apply to all papers. Those wrong interpretations you mention are generally because statistical inference is hard, and it’s an unfortunate reality that a lot of scientists are bad statisticians. There’s an article on the front page of HN about a nature retraction that came from someone asking authors of a paper for their data. That tells me that the data can be useful for fixing suspicions results. But also that simply asking for it can be sufficient. I wonder how often authors say no to data requests. My guess is only in the fishiest cases and in old cases. Publishing code and data isn’t high up the priority list, but in a lot of studies, it’s simple enough to be good ROI.
pdonis 6y ago
Sure: just look at whether the claimed "science" has an actual track record of correct predictions to back up its claims. The nice thing about this method is that it doesn't require you to understand anything about how the scientific model makes its predictions; it could be reading tea leaves or using a magic 8-ball for all you know. The model itself is just a black box that outputs predictions, and all you need to check is how well the predictions matched reality. The usual objection to this is that it's hard to assemble the data to make these evaluations. But imagine if society's standard response to this was to simply ignore any scientific claims not backed up by publicly available data sufficient to evaluate the predictions on which the claims are based? Note that physicists, for example, have no trouble making reams of data freely available online, or writing periodic review articles that summarize the current state of scientific models in a field, with copious references to the original research that verified the predictions.
- andi999 6y ago
  I think society should ignore any result not independently reproduced. Especially (although statistically significant) the actual difference was tiny. (What I mean something like this (making this up): Eating a bar of chocalates increase sleep duration by 3,5 minutes on average)
  - pdonis 6y ago
    Independent reproduction of a successful prediction by a proposed scientific model would be a first step towards a predictive track record, yes. But only a first step. A solid predictive track record for a model has to be built up over a substantial period of time and a substantial number of accurate predictions, not just one trial and one replication. Note, also, that in many cases trials that produce statistically significant results, even when replicated, are not testing any actual model. They are just testing a hypothesis about a correlation. Correlation is not causation, and most such trials are not even capable of testing an actual causal model--they can test for the presence of a correlation, but not why it is present. Figuring out the actual causal process and developing a predictive model based on it takes more follow-up work. So really most such trials are just a first step towards developing a predictive model at all--but even after such a model is developed, you then have to take the time to assemble a good predictive track record for the model.

Only after you have co-authored several papers with the very people you are trying to question, and earned a credential, it seems.

AstralStorm 6y ago
Not really. First you have to understand the paper in question, so you need at least partial background. Then you would have to find the authors. Many sensational paper authors or citation hunters don't like to answer critical questions. Nor is often raw data/algorithms/notes published so you can check if there is a problem.
- guscost 6y ago
  That practice of “not publishing all code and data” really sets off the BS detector.
  - roel_v 6y ago
    Are you a scientist? Because it really doesn't. For example, in most papers that I think harder than normal about, I don't care so much about 'the data' as I do about a description of what this 'data' is actually meant to represent, and how it was collected and processed. (I'm talking here about things that are a bit more complicated than e.g. railfall measurements or any other such lab-like, STEM topics) E.g., I do quite a bit of population modeling. You would think that 'population' is relatively easy to quantify, but it really isn't, and I can talk for hours about how cavalier people throw their 'population data' numbers into models and make all sorts of conclusions based on objectively wrong interpretations what this 'data' is. If the vast majority of papers can't even get that right, I don't care as much about the HN idea of what 'reproducibility' is - i.e. check if 'git clone <xyz> && run_model.sh && run_tests.sh' says 'All OK!' at the end.
    - 6gvONxR4sf7o 6y ago
      I agree in part here, but think it doesn’t apply to all papers. Those wrong interpretations you mention are generally because statistical inference is hard, and it’s an unfortunate reality that a lot of scientists are bad statisticians. There’s an article on the front page of HN about a nature retraction that came from someone asking authors of a paper for their data. That tells me that the data can be useful for fixing suspicions results. But also that simply asking for it can be sufficient. I wonder how often authors say no to data requests. My guess is only in the fishiest cases and in old cases. Publishing code and data isn’t high up the priority list, but in a lot of studies, it’s simple enough to be good ROI.

pdonis 6y ago

Sure: just look at whether the claimed "science" has an actual track record of correct predictions to back up its claims. The nice thing about this method is that it doesn't require you to understand anything about how the scientific model makes its predictions; it could be reading tea leaves or using a magic 8-ball for all you know. The model itself is just a black box that outputs predictions, and all you need to check is how well the predictions matched reality. The usual objection to this is that it's hard to assemble the data to make these evaluations. But imagine if society's standard response to this was to simply ignore any scientific claims not backed up by publicly available data sufficient to evaluate the predictions on which the claims are based? Note that physicists, for example, have no trouble making reams of data freely available online, or writing periodic review articles that summarize the current state of scientific models in a field, with copious references to the original research that verified the predictions.

andi999 6y ago
I think society should ignore any result not independently reproduced. Especially (although statistically significant) the actual difference was tiny. (What I mean something like this (making this up): Eating a bar of chocalates increase sleep duration by 3,5 minutes on average)
- pdonis 6y ago
  Independent reproduction of a successful prediction by a proposed scientific model would be a first step towards a predictive track record, yes. But only a first step. A solid predictive track record for a model has to be built up over a substantial period of time and a substantial number of accurate predictions, not just one trial and one replication. Note, also, that in many cases trials that produce statistically significant results, even when replicated, are not testing any actual model. They are just testing a hypothesis about a correlation. Correlation is not causation, and most such trials are not even capable of testing an actual causal model--they can test for the presence of a correlation, but not why it is present. Figuring out the actual causal process and developing a predictive model based on it takes more follow-up work. So really most such trials are just a first step towards developing a predictive model at all--but even after such a model is developed, you then have to take the time to assemble a good predictive track record for the model.