5 ms·

Practicing hospital physician here. We never use the term "accuracy" to describe diagnostic tests, and whenever I see that term used, I'm very suspicious of th

Practicing hospital physician here. We never use the term "accuracy" to describe diagnostic tests, and whenever I see that term used, I'm very suspicious of the claim being made. In other words, this Bosch announcement can probably be ignored. We use the terms "sensitivity" and "specificity" to describe diagnostic test performance, and never to my knowledge talk about a test's "accuracy." Usually accuracy refers to the sensitivity, but there's a lot of liberty taken with that term in marketing talk. For example, the existing PCR test being used does not have ideal sensitivity, which for practical purposes means that there is a substantial false-negative rate. Which is why (among other reasons) we weren't testing asymptomatic patients in the early phase.

thinkharderdev 7y ago
So would sensitivity be the percentage of infected people who test positive and specificity be the percentage of non-infected who test negative (i.e. sensitivity = (1 - false negative probability) and specificity = (1 - false positive probability))?
- mehrdadn 7y ago
  Yes. The intuitive way I try to remember it is to think that, in just plain English, "How sensitive is this thing?" means "How likely is it to detect whatever it's supposed to detect?" (e.g.: How likely is this security camera to detect movement?)
  - fit2rule 7y ago
    "Sensitivity" is an inaccurate term, since for non-native English speakers, it could be misinterpreted as a term to mean some kind of emotional response. "Accuracy" is a bit more scientific- leaning and therefore easier to translate. Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;)
    - mehrdadn 7y ago
      > Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;) ...what? I was only trying to help the parent find a way to remember which one is sensitivity and which one is specificity...
      - fit2rule 7y ago
        Yes, and you perfectly demonstrated the confusion possible by selecting the wrong terms. Sure, 'specificity' and 'sensitivity' are scientific terms - but for marketing material, can be confused with emotional responses. Thus, I believe Bosch chose 'accuracy', since this translates into other languages more effectively.
        mehrdadn 7y ago
        ...marketing material? To the non-English world? I was literally just explaining a potentially helpful thought process to an English-speaking HN visitor dev who lists his(?) public key on his profile... not trying to produce marketing material for a French waiter or something. Are you sure you're not the one confused here rather than him?
        fit2rule 7y ago
        I'm only trying to point out that Bosch may have specifically chosen the term 'accuracy' instead of 'specificity' or 'sensitivity', which don't translate easily outside of the Anglo-sphere, and in my opinion your thread is a valid example of why those two terms don't get used to market this device, which is what kicked this thread off in the first place ..
        fhars 7y ago
        It is more likely that they were trying do deliberatly write a vague statement, their (native) German announcement uses the same "Genauigkeit", instead of the relevant terms "Sensitivität" and "Spezifizität".
mehrdadn 7y ago
Thank you! I've been bringing up the sensitivity issue a lot recently, since it seems to be disregarded by many upset at the situation. I have a question for you if you don't mind: when so many people complain about lack of testing in the US, do you know what false-negative rates would exist for the tests we'd expect to see (like what they see available en mass in other countries)? And would you know what that level would need to be for the test to be useful?
neuronic 7y ago
The substantial false-negative rate is also one of the reasons why covid patients weren't released from quarantine after a first negative test, since a single result doesn't pass the risk threshold. Multiple negatives in sequence give the desired certainty of someone really being covid-free.
_Microft 7y ago
According to Wikipedia, accuracy is actually defined in this context. This does not not mean that Bosch has not used everyday speech here but it is not clear that they have. Too bad that the article does not include links to additional information. https://en.wikipedia.org/wiki/Sensitivity_and_specificity#Definitions https://en.wikipedia.org/wiki/Sensitivity_and_specificity#De...
- halflings 7y ago
  Of course accuracy is defined, but it's not the right metric because you usually have a large bias in the tested population: even if you only test people with symptoms, you might only have 10% of people that actually have the disease; so a test that always says "NO" would have 90% accuracy. That's why you look at True Positive Rate (sensitivity) and True Negative Rate (specificity) to factor that bias out. If TPR is 95%, that means that 95% of the people that are actually infected (10% of the pop in the previous ex.) will be detected as such. IF TNR is 20%, that means you'll mistakenly detect 20% of people that don't have it as infected, which would be really bad. In other fields, people use Precision and Recall. Precision = % of times the test is right when it predicts a positive. Recall = how many of the real positives were detected as such.
  - buboard 7y ago
    plus in the current context, false negatives are much more dangerous than false positives
    - oldgradstudent 7y ago
      Not necessarily. Hospitals in an epidemic are major spreaders of the infection. If false positive results lead to patients being put together with real carriers, they will soon be infected. From a recent report[1] from Italy: > For example, we are learning that hospitals might be the main Covid-19 carriers, as they are rapidly populated by infected patients, facilitating transmission to uninfected patients. Patients are transported by our regional system, which also contributes to spreading the disease as its ambulances and personnel rapidly become vectors. Health workers are asymptomatic carriers or sick without surveillance; some might die, including young people, which increases the stress of those on the front line. It could mean that some of the major causes of the large death toll in Italy are the lack of personal protection equipment, and, possibly, laxity in following protocol. [1] https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0080 https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0080
fit2rule 7y ago
These terms don't translate very well on an International market. Bosch probably chose its terminology to suit its plans to market these devices outside the Anglo-sphere, as well...
zeroxfe 7y ago
Sounds like "Sensitivity" and "Specificity" are analogous to "Accuracy" and "Precision" typically used in data sciences.
- vibrio 7y ago
  I don't really know data science but in assay validation, sensitivity basically measures false negative rate, and specificity measures false positive rate. Accuracy and precision are measures of assay consistency or tightness. (statisticians may not purely agree with my simple explanation)
  - zeroxfe 7y ago
    Gotcha -- specificity sounded like tightness to me. Thanks for the explanation.
skissane 7y ago
Accuracy = (Sensitivity + Specificity) / 2. The measure does sometimes get used when discussing diagnostic tests - for an example from psychology/psychiatry see https://insar.confex.com/insar/2019/webprogram/Paper29391.html https://insar.confex.com/insar/2019/webprogram/Paper29391.ht...
- notascientist 7y ago
  No, accuracy is the same here it is for any binary classification. accuracy = (true positives + true negatives) / all samples. A largely useless measure. As a single score F1 score is much better, but sensitivity and specifity are specified separately for good reason. It makes napking math much easier, for one thing.
  - skissane 7y ago
    What confused me is there are actually two definitions of accuracy. Accuracy: ACC = (TP+TN)/(P+N) Balanced Accuracy: BA = (TPR+TNR)/2 The definition I gave is the second not the first. You are probably correct that the conference abstract I linked to is using it in the first sense not the second. (I'm not 100% sure though.) Source: https://en.wikipedia.org/wiki/Template:Confusion_matrix_terms https://en.wikipedia.org/wiki/Template:Confusion_matrix_term...
_fizz_buzz_ 7y ago
This page was obviously written by the marketing department. So, let's see what their actual tech specs will look like when they come out.

thinkharderdev 7y ago

So would sensitivity be the percentage of infected people who test positive and specificity be the percentage of non-infected who test negative (i.e. sensitivity = (1 - false negative probability) and specificity = (1 - false positive probability))?

mehrdadn 7y ago
Yes. The intuitive way I try to remember it is to think that, in just plain English, "How sensitive is this thing?" means "How likely is it to detect whatever it's supposed to detect?" (e.g.: How likely is this security camera to detect movement?)
- fit2rule 7y ago
  "Sensitivity" is an inaccurate term, since for non-native English speakers, it could be misinterpreted as a term to mean some kind of emotional response. "Accuracy" is a bit more scientific- leaning and therefore easier to translate. Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;)
  - mehrdadn 7y ago
    > Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;) ...what? I was only trying to help the parent find a way to remember which one is sensitivity and which one is specificity...
    - fit2rule 7y ago
      Yes, and you perfectly demonstrated the confusion possible by selecting the wrong terms. Sure, 'specificity' and 'sensitivity' are scientific terms - but for marketing material, can be confused with emotional responses. Thus, I believe Bosch chose 'accuracy', since this translates into other languages more effectively.
      - mehrdadn 7y ago
        ...marketing material? To the non-English world? I was literally just explaining a potentially helpful thought process to an English-speaking HN visitor dev who lists his(?) public key on his profile... not trying to produce marketing material for a French waiter or something. Are you sure you're not the one confused here rather than him?
        fit2rule 7y ago
        I'm only trying to point out that Bosch may have specifically chosen the term 'accuracy' instead of 'specificity' or 'sensitivity', which don't translate easily outside of the Anglo-sphere, and in my opinion your thread is a valid example of why those two terms don't get used to market this device, which is what kicked this thread off in the first place ..
        fhars 7y ago
        It is more likely that they were trying do deliberatly write a vague statement, their (native) German announcement uses the same "Genauigkeit", instead of the relevant terms "Sensitivität" and "Spezifizität".

mehrdadn 7y ago

Thank you! I've been bringing up the sensitivity issue a lot recently, since it seems to be disregarded by many upset at the situation. I have a question for you if you don't mind: when so many people complain about lack of testing in the US, do you know what false-negative rates would exist for the tests we'd expect to see (like what they see available en mass in other countries)? And would you know what that level would need to be for the test to be useful?

neuronic 7y ago

The substantial false-negative rate is also one of the reasons why covid patients weren't released from quarantine after a first negative test, since a single result doesn't pass the risk threshold. Multiple negatives in sequence give the desired certainty of someone really being covid-free.

_Microft 7y ago

According to Wikipedia, accuracy is actually defined in this context. This does not not mean that Bosch has not used everyday speech here but it is not clear that they have. Too bad that the article does not include links to additional information. https://en.wikipedia.org/wiki/Sensitivity_and_specificity#Definitions https://en.wikipedia.org/wiki/Sensitivity_and_specificity#De...

halflings 7y ago
Of course accuracy is defined, but it's not the right metric because you usually have a large bias in the tested population: even if you only test people with symptoms, you might only have 10% of people that actually have the disease; so a test that always says "NO" would have 90% accuracy. That's why you look at True Positive Rate (sensitivity) and True Negative Rate (specificity) to factor that bias out. If TPR is 95%, that means that 95% of the people that are actually infected (10% of the pop in the previous ex.) will be detected as such. IF TNR is 20%, that means you'll mistakenly detect 20% of people that don't have it as infected, which would be really bad. In other fields, people use Precision and Recall. Precision = % of times the test is right when it predicts a positive. Recall = how many of the real positives were detected as such.
- buboard 7y ago
  plus in the current context, false negatives are much more dangerous than false positives
  - oldgradstudent 7y ago
    Not necessarily. Hospitals in an epidemic are major spreaders of the infection. If false positive results lead to patients being put together with real carriers, they will soon be infected. From a recent report[1] from Italy: > For example, we are learning that hospitals might be the main Covid-19 carriers, as they are rapidly populated by infected patients, facilitating transmission to uninfected patients. Patients are transported by our regional system, which also contributes to spreading the disease as its ambulances and personnel rapidly become vectors. Health workers are asymptomatic carriers or sick without surveillance; some might die, including young people, which increases the stress of those on the front line. It could mean that some of the major causes of the large death toll in Italy are the lack of personal protection equipment, and, possibly, laxity in following protocol. [1] https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0080 https://catalyst.nejm.org/doi/full/10.1056/CAT.20.0080

fit2rule 7y ago

These terms don't translate very well on an International market. Bosch probably chose its terminology to suit its plans to market these devices outside the Anglo-sphere, as well...

zeroxfe 7y ago

Sounds like "Sensitivity" and "Specificity" are analogous to "Accuracy" and "Precision" typically used in data sciences.

vibrio 7y ago
I don't really know data science but in assay validation, sensitivity basically measures false negative rate, and specificity measures false positive rate. Accuracy and precision are measures of assay consistency or tightness. (statisticians may not purely agree with my simple explanation)
- zeroxfe 7y ago
  Gotcha -- specificity sounded like tightness to me. Thanks for the explanation.

skissane 7y ago

Accuracy = (Sensitivity + Specificity) / 2. The measure does sometimes get used when discussing diagnostic tests - for an example from psychology/psychiatry see https://insar.confex.com/insar/2019/webprogram/Paper29391.html https://insar.confex.com/insar/2019/webprogram/Paper29391.ht...

notascientist 7y ago
No, accuracy is the same here it is for any binary classification. accuracy = (true positives + true negatives) / all samples. A largely useless measure. As a single score F1 score is much better, but sensitivity and specifity are specified separately for good reason. It makes napking math much easier, for one thing.
- skissane 7y ago
  What confused me is there are actually two definitions of accuracy. Accuracy: ACC = (TP+TN)/(P+N) Balanced Accuracy: BA = (TPR+TNR)/2 The definition I gave is the second not the first. You are probably correct that the conference abstract I linked to is using it in the first sense not the second. (I'm not 100% sure though.) Source: https://en.wikipedia.org/wiki/Template:Confusion_matrix_terms https://en.wikipedia.org/wiki/Template:Confusion_matrix_term...

_fizz_buzz_ 7y ago

This page was obviously written by the marketing department. So, let's see what their actual tech specs will look like when they come out.