4 ms·

I mean, part of the reason why I ask this question is that specificity and sensitivity tend to be inversely related: as a test becomes more sensitive, it tends

by nrr 2y ago

I mean, part of the reason why I ask this question is that specificity and sensitivity tend to be inversely related: as a test becomes more sensitive, it tends to become less specific, say. (While I'm here, in case folks are lacking the statistics background: specificity and sensitivity refer to the probabilities that a test will return with a negative test result in the absence of a condition or a positive test result in the presence of a condition respectively.) Most of the "promising" tests I've seen tend to be something like 80% sensitive and 20% specific (testing for Lyme disease comes to mind), which makes them no better than flipping a weighted coin. The fact that this Alzheimer's test beats that feels like a big deal on the probability merits alone, and I can't think of basically any other tests that do that. (Serum cardiac troponins maybe?)

Rhapso 2y ago
They don't offer any tests for "general population" situations, only "people we think might already have dementia". This is a really strong filter up-front. I suspect the test would be high sensitivity very low specificity in the general population, but that is an intuition not backed by any data.
- nrr 2y ago
  Aha, yeah, restricting your population could certainly have that effect on the specificity. Thanks for clarifying.
  - tptacek 2y ago
    For what it's worth, I'm not sure specificity/sensitivity is the limiting factor here (for doing broad population surveys with this test); it's the low base rate of Alzheimers relative to the specificity. At 90% true positive and 10% base rate, I think? (I suck at math) a positive test has like a coin flip chance of being right?
    - nrr 2y ago
      We're in the realm of probability, which is mysterious sometimes even to people who have a math background. I didn't fully grok it until I studied intensely for my first actuarial exams. (: This kind of thing is almost always a weighted coin toss: with sensitivity or specificity alone, you only have two possible outcomes (present/relevant, absent/irrelevant), and the thing that changes is the probability distribution of those outcomes. Combining the two gets you the full four: present and relevant; present and irrelevant; absent and relevant; and absent and irrelevant. Taking the uniform distribution, they're all 25% likely, but the idea is to find a probability distribution that makes the "present and relevant" and "absent and relevant" outcomes more likely. Since I myself don't work in a clinical setting, I simply hadn't considered that the clinician would want to exercise discretion in pre-screening for specificity before ordering the test in order to get there. Oops.
      - tptacek 2y ago
        I'm just saying, 90% true positive rate, 10% base rate, 1000 pts: * 100 true positives * 100 false positives (100-90=10% of 1000) That's still just a 10% false positive rate across the population, but if you get a positive test, it's only 50/50 correct. Did I do this wrong? I'm assuming I did this wrong.
        nrr 2y ago
        I think you may be close but likely for the wrong reasons. I had to sit down with this for a moment to feel comfortable with it. If we take your 10% base rate to be disease prevalence, that gives us 100 sick and 900 well. Of the 100 sick, something with 90% sensitivity should get me 90 true positive tests and 10 false negative tests. Of the 900 well, I should expect to see for a test with 90% specificity, what, 80 false positives and 810 true negatives? if I did my arithmetic right?

by nrr 2y ago

Rhapso 2y ago

They don't offer any tests for "general population" situations, only "people we think might already have dementia". This is a really strong filter up-front. I suspect the test would be high sensitivity very low specificity in the general population, but that is an intuition not backed by any data.

nrr 2y ago
Aha, yeah, restricting your population could certainly have that effect on the specificity. Thanks for clarifying.
- tptacek 2y ago
  For what it's worth, I'm not sure specificity/sensitivity is the limiting factor here (for doing broad population surveys with this test); it's the low base rate of Alzheimers relative to the specificity. At 90% true positive and 10% base rate, I think? (I suck at math) a positive test has like a coin flip chance of being right?
  - nrr 2y ago
    We're in the realm of probability, which is mysterious sometimes even to people who have a math background. I didn't fully grok it until I studied intensely for my first actuarial exams. (: This kind of thing is almost always a weighted coin toss: with sensitivity or specificity alone, you only have two possible outcomes (present/relevant, absent/irrelevant), and the thing that changes is the probability distribution of those outcomes. Combining the two gets you the full four: present and relevant; present and irrelevant; absent and relevant; and absent and irrelevant. Taking the uniform distribution, they're all 25% likely, but the idea is to find a probability distribution that makes the "present and relevant" and "absent and relevant" outcomes more likely. Since I myself don't work in a clinical setting, I simply hadn't considered that the clinician would want to exercise discretion in pre-screening for specificity before ordering the test in order to get there. Oops.
    - tptacek 2y ago
      I'm just saying, 90% true positive rate, 10% base rate, 1000 pts: * 100 true positives * 100 false positives (100-90=10% of 1000) That's still just a 10% false positive rate across the population, but if you get a positive test, it's only 50/50 correct. Did I do this wrong? I'm assuming I did this wrong.
      - nrr 2y ago
        I think you may be close but likely for the wrong reasons. I had to sit down with this for a moment to feel comfortable with it. If we take your 10% base rate to be disease prevalence, that gives us 100 sick and 900 well. Of the 100 sick, something with 90% sensitivity should get me 90 true positive tests and 10 false negative tests. Of the 900 well, I should expect to see for a test with 90% specificity, what, 80 false positives and 810 true negatives? if I did my arithmetic right?