6 ms·

How did this article, written by someone who clearly lacks an understanding of basic statistics, make it into the Upshot? They try to make it seem like the test

How did this article, written by someone who clearly lacks an understanding of basic statistics, make it into the Upshot? They try to make it seem like the test is wrong 85% of the time, but that's not necessarily the case. All we know from the article is that 85 / 100 positive results are false positives, which means the test could actually be quite accurate. If the test correctly identifies 100% of real cases, then that sounds like an excellent test. Just as an example, if 1/4000 people have the disease, and the test identifies 100% of these cases, then around 0.14% of test takers will get a false positive.

SpicyLemonZest 5y ago
Their infographics convince me that they understand the statistics. But one of the key issues here is that the statistics are radically counterintuitive in a way that most people don't understand - the patients, the testing companies, and even some medical staff all incorrectly believe that a positive test for a rare condition means you probably have the condition.
- halpert 5y ago
  Their graphics say the tests are “84% wrong.” Do you really feel that’s an accurate description? That doesn’t feel like an accurate description to me, and their usage of “wrong” in this context highlights that they don’t understand the distinction and importance of true positives, false positives, true negatives, and false negatives when measuring accuracy.
  - SpicyLemonZest 5y ago
    I really feel it's an accurate description. If you get a positive result on the test, there's a 16% chance your fetus has a 1p36 deletion and an 84% chance they don't.
    - halpert 5y ago
      As you said “if you get a positive result”. It’s true, if you ignore the 99.9% of the time the test is correct (true negative result), then you can say the test is 84% wrong.
      - SpicyLemonZest 5y ago
        84% of people who got a positive test result will end up telling their family "it's OK, the first test was wrong, my baby doesn't have a 1p36 deletion after all". The 99.9% of other people who got true negatives are important from a test design perspective, because specificity is closer to the actual levers you can pull on, but it's not super relevant to the decisionmaking process of someone who gets a positive result.
    - andreilys 5y ago
      Ignoring all the true and false negatives which themselves are markers of how accurate the test is. 16% precision is the correct statement, saying the test is wrong 84% of the time implies that those getting negative results might actually have positive results.
      - robbedpeter 5y ago
        He framed his statement correctly, limiting his observation to the condition that the test returned a positive result. Saying that 84% of positive results are false is correct if only 16% are true. You'd need to know false negative rates and base occurrence rates (modified by whatever other factors are unique to your situation) to inform the nature of information you get by performing the test.
  - isoprophlex 5y ago
    Going through something like this is very VERY stressful. When you get a negative you immediately forget about it. When you get a positive you die inside. Speaking from experience here. 84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course) 84% of people whose world is turned upside down are actually getting a wrong diagnosis.
    - deleted 5y ago
      [deleted]
    - andreilys 5y ago
      You’re talking about precision (true positive / true positive + false negative) but that’s only one part of the story. There is a real human cost to having a child born with a rare genetic disease (and I would argue is immensely more stressful). You can easily adjust the sensitivity to the test but at the cost of detecting actual true positive cases. The correct response to receiving a positive is to do another test to ensure it’s not a false positive. To say 84% wrong is clickbait and used to elicit a legislative response (FDA regulation), which will help the reporters career. The actual ratio to tell if something is “wrong” is accuracy (True positive + true negative) / (true positive + true negative + false positive + false negative)
      - mnw21cam 5y ago
        No, precision is true positive / (true positive + false positive). Your first equation is sensitivity.
    - fshbbdssbbgdd 5y ago
      If you get a negative result and then your child is born with the condition, you won’t forget quickly either.
ellisv 5y ago
I disagree. It is clear from the title, “When They Warn of Rare Disorders, These Prenatal Tests Are Usually Wrong”, and the lead that they’re focusing on false positives.
- halpert 5y ago
  It's true they are focusing on false positives, but the authors are using the ratio of false positives to true positives to paint a picture that the tests are inaccurate, when in reality the tests are accurate. What this article is looking at is called the "sensitivity" of a test: https://en.wikipedia.org/wiki/Sensitivity_and_specificity https://en.wikipedia.org/wiki/Sensitivity_and_specificity
  - ramraj07 5y ago
    Did they use the word accurate? You used the word accurate and then you yourself are going on a tirade about how that’s not correct? It’s clear the article is talking about why sensitivity is important in layman’s terms and while it could use better writing it’s a real problem in diagnostics. This is why you don’t ask men to take a pregnancy test to check for prostrate cancer. It is accurate but not sensitive.
    - halpert 5y ago
      They used the word “wrong”. Whether or not they used wrong to mean inaccurate, or wrong to mean not sensitive is up to the reader.
  - adjkant 5y ago
    While the author may not be well versed or focusing on the stats side, you're missing the human side here I think. > the tests are inaccurate, when in reality the tests are accurate If the test make someone consider terminating a pregnancy or even considering it, that's a lot of pain. So for that human, the test is failing its purpose potentially, depending on the value calculation of terminating a viable pregnancy vs the severity of the issue if it comes to term. For a human, accuracy as you defined it means little to nothing. Usefulness and helpfulness are far better metrics, and such a high false positive rate is clearly causing issues in respect to those, which is what the article is highlighting.
mcguire 5y ago
Would a test that reported 100% positive similarly be "quite accurate"? It would catch all true positives, right?
- aaron695 5y ago

SpicyLemonZest 5y ago

Their infographics convince me that they understand the statistics. But one of the key issues here is that the statistics are radically counterintuitive in a way that most people don't understand - the patients, the testing companies, and even some medical staff all incorrectly believe that a positive test for a rare condition means you probably have the condition.

halpert 5y ago
Their graphics say the tests are “84% wrong.” Do you really feel that’s an accurate description? That doesn’t feel like an accurate description to me, and their usage of “wrong” in this context highlights that they don’t understand the distinction and importance of true positives, false positives, true negatives, and false negatives when measuring accuracy.
- SpicyLemonZest 5y ago
  I really feel it's an accurate description. If you get a positive result on the test, there's a 16% chance your fetus has a 1p36 deletion and an 84% chance they don't.
  - halpert 5y ago
    As you said “if you get a positive result”. It’s true, if you ignore the 99.9% of the time the test is correct (true negative result), then you can say the test is 84% wrong.
    - SpicyLemonZest 5y ago
      84% of people who got a positive test result will end up telling their family "it's OK, the first test was wrong, my baby doesn't have a 1p36 deletion after all". The 99.9% of other people who got true negatives are important from a test design perspective, because specificity is closer to the actual levers you can pull on, but it's not super relevant to the decisionmaking process of someone who gets a positive result.
  - andreilys 5y ago
    Ignoring all the true and false negatives which themselves are markers of how accurate the test is. 16% precision is the correct statement, saying the test is wrong 84% of the time implies that those getting negative results might actually have positive results.
    - robbedpeter 5y ago
      He framed his statement correctly, limiting his observation to the condition that the test returned a positive result. Saying that 84% of positive results are false is correct if only 16% are true. You'd need to know false negative rates and base occurrence rates (modified by whatever other factors are unique to your situation) to inform the nature of information you get by performing the test.
- isoprophlex 5y ago
  Going through something like this is very VERY stressful. When you get a negative you immediately forget about it. When you get a positive you die inside. Speaking from experience here. 84% wrong sounds, to me, as an accurate description. Experiencing this from the inside out, only the false/true positive ratio matters. (Given sufficiently low false negative rates, of course) 84% of people whose world is turned upside down are actually getting a wrong diagnosis.
  - deleted 5y ago
    [deleted]
  - andreilys 5y ago
    You’re talking about precision (true positive / true positive + false negative) but that’s only one part of the story. There is a real human cost to having a child born with a rare genetic disease (and I would argue is immensely more stressful). You can easily adjust the sensitivity to the test but at the cost of detecting actual true positive cases. The correct response to receiving a positive is to do another test to ensure it’s not a false positive. To say 84% wrong is clickbait and used to elicit a legislative response (FDA regulation), which will help the reporters career. The actual ratio to tell if something is “wrong” is accuracy (True positive + true negative) / (true positive + true negative + false positive + false negative)
    - mnw21cam 5y ago
      No, precision is true positive / (true positive + false positive). Your first equation is sensitivity.
  - fshbbdssbbgdd 5y ago
    If you get a negative result and then your child is born with the condition, you won’t forget quickly either.

ellisv 5y ago

I disagree. It is clear from the title, “When They Warn of Rare Disorders, These Prenatal Tests Are Usually Wrong”, and the lead that they’re focusing on false positives.

halpert 5y ago
It's true they are focusing on false positives, but the authors are using the ratio of false positives to true positives to paint a picture that the tests are inaccurate, when in reality the tests are accurate. What this article is looking at is called the "sensitivity" of a test: https://en.wikipedia.org/wiki/Sensitivity_and_specificity https://en.wikipedia.org/wiki/Sensitivity_and_specificity
- ramraj07 5y ago
  Did they use the word accurate? You used the word accurate and then you yourself are going on a tirade about how that’s not correct? It’s clear the article is talking about why sensitivity is important in layman’s terms and while it could use better writing it’s a real problem in diagnostics. This is why you don’t ask men to take a pregnancy test to check for prostrate cancer. It is accurate but not sensitive.
  - halpert 5y ago
    They used the word “wrong”. Whether or not they used wrong to mean inaccurate, or wrong to mean not sensitive is up to the reader.
- adjkant 5y ago
  While the author may not be well versed or focusing on the stats side, you're missing the human side here I think. > the tests are inaccurate, when in reality the tests are accurate If the test make someone consider terminating a pregnancy or even considering it, that's a lot of pain. So for that human, the test is failing its purpose potentially, depending on the value calculation of terminating a viable pregnancy vs the severity of the issue if it comes to term. For a human, accuracy as you defined it means little to nothing. Usefulness and helpfulness are far better metrics, and such a high false positive rate is clearly causing issues in respect to those, which is what the article is highlighting.

mcguire 5y ago

Would a test that reported 100% positive similarly be "quite accurate"? It would catch all true positives, right?

aaron695 5y ago