3 ms·

How do we prove that they didn't just look at mortality in all diseases and for all restaurants, and cherry-picked stroke-vs-ramen because it has the best signi

How do we prove that they didn't just look at mortality in all diseases and for all restaurants, and cherry-picked stroke-vs-ramen because it has the best significance? I am not accusing them of anything - I have no horse in this race. What I am curious about is: how do you prove that you haven't p-hacked, if you are going to do a study like this? Sure, you can pre-register your methods, but presumably you can get the data and do some data crunching on it way before you have to pre-register. How do you prove that your results are legit in the age of reproduction crisis?

shakna 7y ago
You don't at the level of this study. It's useful to suggest this is something that might be explored in a better, more clinical, setting. The news might pick up and run with this, which is unfortunate, but so long as you don't view this as anything more than a suggestion of a relationship, you should be fine.
ajross 7y ago
> How do you prove that your results are legit in the age of reproduction crisis? You... try to reproduce them. The "reproduction crisis" isn't a failure of the scientific method, it's just pointing out that we've picked all the low hanging fruit (i.e. highly significant results) in science while increasing the amount of science done by huge amounts. So there's a lot of junk. Possibly (hopefully! I love ramen) including this paper. But nothing has changed with how you prove your results are legit: do a similar study in a different location trying to find the same effect.
wongarsu 7y ago
I guess the only reliable way is to have an independent group reproduce it using different data and publish their result. Whenever theres something interesting shown in natural sciences you quickly hear about replication efforts being able or unable to replicate the results. In most other sciences there doesn't seem to be much glory in replication
- deleted 7y ago
  [deleted]
gryson 7y ago
Good faith is required to accept any type of reporting. Otherwise, your question can always be extended to "How do we know that they didn't just make it all up?" The researchers did not report conducting more comparisons and they did not adjust for this in their analysis, so we have to assume that they are telling the truth. Of course, there's always the chance that they are being deceitful, so we have to reserve some skepticism and take into consideration related studies before forming any firm conclusions. In this particular case, the correlations that they found have quite strong effect sizes with p < .001. Figures 2 and 3 are quite striking. These results are also supported by previous research. That doesn't mean the matter is settled by any means. It's just another piece in the puzzle. But there is no indication whatsoever that they are p-hacking or going on a fishing expedition.
- joe_the_user 7y ago
  I don't think you have to assume good faith at all in these situations. If someone comes up with a correlation of X statistic with Y statistic in a fashion seemingly coming from nowhere, they either say why they started studying the correlation or show the correlation is incredibly strong. I think that's pretty much what "extraordinary claims require extraordinary evidence" winds-up meaning.
- mattkrause 7y ago
  The correlation is so large that I find it literally incredible. Could ramen restaurant prevalence really explain a quarter to a third of the variance in mortality (which is what r=0.5 to 0.6 implies)? If so, the damn things are an outright menace. And why is it so different from fast food, which is equally unhealthy but supposedly makes people live longer?
  - perl4ever 7y ago
    What is the connection between a correlation and an explanation? I see none. Zero. Nada.
    - mattkrause 7y ago

shakna 7y ago

You don't at the level of this study. It's useful to suggest this is something that might be explored in a better, more clinical, setting. The news might pick up and run with this, which is unfortunate, but so long as you don't view this as anything more than a suggestion of a relationship, you should be fine.

ajross 7y ago

> How do you prove that your results are legit in the age of reproduction crisis? You... try to reproduce them. The "reproduction crisis" isn't a failure of the scientific method, it's just pointing out that we've picked all the low hanging fruit (i.e. highly significant results) in science while increasing the amount of science done by huge amounts. So there's a lot of junk. Possibly (hopefully! I love ramen) including this paper. But nothing has changed with how you prove your results are legit: do a similar study in a different location trying to find the same effect.

wongarsu 7y ago

I guess the only reliable way is to have an independent group reproduce it using different data and publish their result. Whenever theres something interesting shown in natural sciences you quickly hear about replication efforts being able or unable to replicate the results. In most other sciences there doesn't seem to be much glory in replication

deleted 7y ago
[deleted]

gryson 7y ago

Good faith is required to accept any type of reporting. Otherwise, your question can always be extended to "How do we know that they didn't just make it all up?" The researchers did not report conducting more comparisons and they did not adjust for this in their analysis, so we have to assume that they are telling the truth. Of course, there's always the chance that they are being deceitful, so we have to reserve some skepticism and take into consideration related studies before forming any firm conclusions. In this particular case, the correlations that they found have quite strong effect sizes with p < .001. Figures 2 and 3 are quite striking. These results are also supported by previous research. That doesn't mean the matter is settled by any means. It's just another piece in the puzzle. But there is no indication whatsoever that they are p-hacking or going on a fishing expedition.

joe_the_user 7y ago
I don't think you have to assume good faith at all in these situations. If someone comes up with a correlation of X statistic with Y statistic in a fashion seemingly coming from nowhere, they either say why they started studying the correlation or show the correlation is incredibly strong. I think that's pretty much what "extraordinary claims require extraordinary evidence" winds-up meaning.
mattkrause 7y ago
The correlation is so large that I find it literally incredible. Could ramen restaurant prevalence really explain a quarter to a third of the variance in mortality (which is what r=0.5 to 0.6 implies)? If so, the damn things are an outright menace. And why is it so different from fast food, which is equally unhealthy but supposedly makes people live longer?
- perl4ever 7y ago
  What is the connection between a correlation and an explanation? I see none. Zero. Nada.
  - mattkrause 7y ago