4 ms·

>I think people should be extremely skeptical of anything they read This is a mistake that many HN readers make; they think that if one is equipped with some a

>I think people should be extremely skeptical of anything they read This is a mistake that many HN readers make; they think that if one is equipped with some above-average level of intelligence, one can discern the validity of new research. But this is wrong. It usually takes many years of study before one can begin to clearly understand what is even being said, let alone whether it has any veracity. People of above-average intelligence often resent this fact, because it suggests that their smart opinion isn't as valuable as the opinion of an expert, but that is simply a sad fact of life. If they haven't put in the work in that field, then they don't know what they're talking about in that field, and they are incapable of applying skepticism in that field correctly. This is precisely and unfortunately why we are forced to place our trust in experts.

mike_hearn 3y ago
In practice that's not the case. Maybe for some fields it is, but there are definitely plenty of fields where minimal expertise is required to do basic sanity checks of a paper. And there are many generic 'tells' that are indicative of issues across a wide range of fields, e.g. if a paper is reporting mostly P = 0.049 results, well ... As an example of an issue that doesn't require much expertise to spot, I was reading a health paper a couple of days ago that reported on an intervention in a specific population. The paper said words to the effect of "[the intervention] was effective, especially for men". The associated chart not only had (somehow?!) got a legend that didn't match the actual chart lines, but whilst the line for men did indeed go down the line for women was very clearly flat! This should have been reported as "no effect in women but an effect in men" but wasn't. When the wording doesn't match the data being reported, that's a good sign in any field that the researchers know they're treading on thin ice. That particular claim was a correlation/causation fallacy anyway, which is very common in health. They didn't have any proof it was their intervention causing the reduction and there were a bunch of reasons to suspect it wasn't. But the intervention was long term and high effort so it's not a surprise they wanted to find something.
- SantalBlush 3y ago
  This is a great example of what I'm talking about. P-values of 0.049 is not a "tell"; that is an HN meme. You can have p-values above 5 percent and the paper can still convey novel and valid research, depending on the nature of the study. The 5-percent value is a common cutoff, but it is a largely arbitrary one. Sometimes there is justification for a higher one, and sometimes it should be lower. To gauge what sort of p-value suggests a useful model requires familiarity with the area and the broader context of the research.
  - mike_hearn 3y ago
    That's the point - it's an arbitrary cutoff, so the high frequency with which results cluster around that point is indicative of a problem. If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. Now think about how many papers get published, and many of them report more than one thing. The right threshold should really be an order of magnitude lower. It's not OK for scientists to report FPs at that rate, and that's a big part of the reason for declining confidence in science.
    - SantalBlush 3y ago
      >If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. That is not the correct interpretation of a p-value. See the ASA's statement on p-values.[0] >Researchers often wish to turn a p-value into a state- ment about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself. Moreover, while it's fine to suspect that p-hacking took place with p-values just under 5 percent, it is merely a suspicion and nothing more. Throwing out p-values you don't like without further evidence is a different sort of violation of the methodology; it is like p-hacking in the other direction. >The right threshold should really be an order of magnitude lower. This is false. Again, see the ASA's statement.[0] >Practices that reduce data analysis or scientific infer- ence to mechanical “bright-line” rules (such as “p < 0.05”) for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision making. A conclusion does not immediately become “true” on one side of the divide and “false” on the other. Researchers should bring many contextual factors into play to derive scientific inferences, including the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis. Pragmatic considerations often require binary, “yes-no” decisions, but this does not mean that p-values alone can ensure that a decision is correct or incorrect. The widespread use of “statistical significance” (generally interpreted as “p & 0.05”) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distor- tion of the scientific process. In short, these things often require a strong statistical literacy to interpret, which most people complaining about p-hacking do not possess. [0]https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108 https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016...

mike_hearn 3y ago

In practice that's not the case. Maybe for some fields it is, but there are definitely plenty of fields where minimal expertise is required to do basic sanity checks of a paper. And there are many generic 'tells' that are indicative of issues across a wide range of fields, e.g. if a paper is reporting mostly P = 0.049 results, well ... As an example of an issue that doesn't require much expertise to spot, I was reading a health paper a couple of days ago that reported on an intervention in a specific population. The paper said words to the effect of "[the intervention] was effective, especially for men". The associated chart not only had (somehow?!) got a legend that didn't match the actual chart lines, but whilst the line for men did indeed go down the line for women was very clearly flat! This should have been reported as "no effect in women but an effect in men" but wasn't. When the wording doesn't match the data being reported, that's a good sign in any field that the researchers know they're treading on thin ice. That particular claim was a correlation/causation fallacy anyway, which is very common in health. They didn't have any proof it was their intervention causing the reduction and there were a bunch of reasons to suspect it wasn't. But the intervention was long term and high effort so it's not a surprise they wanted to find something.

SantalBlush 3y ago
This is a great example of what I'm talking about. P-values of 0.049 is not a "tell"; that is an HN meme. You can have p-values above 5 percent and the paper can still convey novel and valid research, depending on the nature of the study. The 5-percent value is a common cutoff, but it is a largely arbitrary one. Sometimes there is justification for a higher one, and sometimes it should be lower. To gauge what sort of p-value suggests a useful model requires familiarity with the area and the broader context of the research.
- mike_hearn 3y ago
  That's the point - it's an arbitrary cutoff, so the high frequency with which results cluster around that point is indicative of a problem. If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. Now think about how many papers get published, and many of them report more than one thing. The right threshold should really be an order of magnitude lower. It's not OK for scientists to report FPs at that rate, and that's a big part of the reason for declining confidence in science.
  - SantalBlush 3y ago
    >If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. That is not the correct interpretation of a p-value. See the ASA's statement on p-values.[0] >Researchers often wish to turn a p-value into a state- ment about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself. Moreover, while it's fine to suspect that p-hacking took place with p-values just under 5 percent, it is merely a suspicion and nothing more. Throwing out p-values you don't like without further evidence is a different sort of violation of the methodology; it is like p-hacking in the other direction. >The right threshold should really be an order of magnitude lower. This is false. Again, see the ASA's statement.[0] >Practices that reduce data analysis or scientific infer- ence to mechanical “bright-line” rules (such as “p < 0.05”) for justifying scientific claims or conclusions can lead to erroneous beliefs and poor decision making. A conclusion does not immediately become “true” on one side of the divide and “false” on the other. Researchers should bring many contextual factors into play to derive scientific inferences, including the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis. Pragmatic considerations often require binary, “yes-no” decisions, but this does not mean that p-values alone can ensure that a decision is correct or incorrect. The widespread use of “statistical significance” (generally interpreted as “p & 0.05”) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distor- tion of the scientific process. In short, these things often require a strong statistical literacy to interpret, which most people complaining about p-hacking do not possess. [0]https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108 https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016...