9 ms·

Press release: https://publishing.aip.org/publications/latest-content/would-you-prefer-a-mammogram-mri-or-saliva-on-a-test-strip/ https://publishing.aip.org/pub

Press release: https://publishing.aip.org/publications/latest-content/would-you-prefer-a-mammogram-mri-or-saliva-on-a-test-strip/ https://publishing.aip.org/publications/latest-content/would... Actual publication: https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/High-sensitivity-saliva-based-biosensor-in https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig... Experiment size is literally N=21, with 4 healthy participants, 3 in-situ breast cancers, and 14 invasive breast cancers. N=21 might as well be useless in my opinion. You can't draw any meaningful conclusions about statistical power of this test; if your priors were 10% for breast cancer, after taking this test, your posterior probably remains unchanged.

causal 3y ago
Yah headline jumping the gun a bit, but hopefully this motivates funding to get a bigger study / refine the technique.
deleted 3y ago
[deleted]
pvaldes 3y ago
> We tried it with 21 people The new Theranos is here again
- dheera 3y ago
  No, not at all. Polar opposites. Theranos was actual lies. TFA is just being honest about a low sample size.
  - pvaldes 3y ago
    If somebody claims: our device "accurately tests", and we discover later that "In fact the tests hadn't been done still" (at a meaningful way), It should be taken as a red flag. Specially when it came in the same package with "to make lots of tests would be really cheap" (but for some reason they didn't tried it). Maybe be a great, honest work, brilliant step?. Yes. But please don't claim that your model is reproducible if you didn't ever tried seriously to reproduce it first. What if the results are just a random effect?. What if the test says just positive most of the time? The test negative response has been tested in a control group of 4 people? This is 20 bucks well spent. --At this moment-- isn't different than other thousand projects that seemed too good to be true, took the money and soon vanished. I hope to be wrong.
    - dheera 3y ago
      > didn't ever tried seriously to reproduce it first Maybe they found something worth testing and just need more funding to actually do the tests? I have no problem with early free speech about "I found something interesting" as long as you're honest about the sample size and don't misrepresent it.
  - jacquesm 3y ago
    Without a larger n you can't really tell what the fp rate is and that means that the accuracy figure isn't very useful. If accuracy holds in light of a false positive rate that is much lower than other tests then this may well be very useful. But you can not conclude that at all based on the evidence presented.
- sonicanatidae 3y ago
  Not quite. Theranos was stating that they were capable of doing what was impossible prior. The real issue with Theranos was physics. A single drop of blood is simply not a large enough sample for them to do all the testing they claimed. As in, impossible with today's tech and in the near future. These folks are just stating that, "so far, this looks promising". At least, thats my read. YMMV.
  - chaxor 3y ago
    It is important to point out that what they claimed to do is not technically impossible actually. They just didn't do it, and lied about it. That was the real problem. It's very unfortunate, because it is technically possible, just very difficult to achieve, even in academia (so it won't be done first in industry). This was the absolute worst part of Theranos, is that they deter others from trying to make headway in the space at all.
    - sonicanatidae 3y ago
      I don't believe they could run a battery of 81+ tests on a drop of blood. There simply isn't enough there. Why do you think they take tubes of blood for testing, currently? The issue is concentrations and the presence of them in sufficient amounts to be detected reliably and consistently. edit: typo.. words are hard.
      - chaxor 3y ago
        It's because that's what the equipment is designed to use, and yes it provides more reliable results with the current technology. Much of the equipment used in healthcare relies on very old technology. This certainly does not mean it cannot be improved to work with less amount of material. There are hand held sequencers for your phone now (minIon) that are very cheap and work. There are also large multimillion dollar bench top machines on the same technology from nanopore. Both are very new technology compared to what we used in early 2000s, and offer various different features and certainties from the analysis. However, it is foolish to say that the minIon is impossible. It works. It's just less resolute than other systems. This is true for many systems which people cry "Theranos" about. Their issue was lying. That was the problem.
fuhrtf 3y ago
That’s a bit harsh. It’s an exploratory study testing a new paradigm.
- oliwarner 3y ago
  It's only as harsh as the the headline is rose-tinted. "Exploratory study shows promise" is better. When n=21, flipping a coin as about as accurate.
  - matthewmacleod 3y ago
    This statement belies a fundamental misunderstanding of statistics on your part.
    - ethanbond 3y ago
      If you're really good at statistics, you can just intuit what seems like a sufficient N. This is actually a powerful statistical method because you can vary your intuition depending on whether you want to believe the study was well-powered or not, enabling you to easily discard Bad Evidence and include Good Evidence. Few understand this.
      - jncfhnb 3y ago
        No, that’s utter nonsense. Statistical power is well defined and the required N to reliably detect an effect is a function of its effect size, which is the thing that wannabe statisticians don’t understand.
        lazyasciiart 3y ago
        > you can vary your intuition depending on whether you want to believe the study I suspect it is deliberate nonsense.
        solardev 3y ago
redder23 3y ago
Totally agree!
dheera 3y ago
It's still a nonzero sample size, and if anything, says two things: (a) We should do more tests to increase N (b) If a $5 device tests you positive, maybe you should go get checked out for $5000 or whatever the doctors charge you (because insurance often only pays AFTER "shit happens" and often does not pay to test "whether shit might happen"), that you wouldn't have thought of doing otherwise.
jncfhnb 3y ago
You can absolutely draw conclusions about the statistical power of the test. That’s what statistics is for. If this sample data is randomly sampled it looks like it will be a fairly high precision test for invasive cancers, with room for false negatives. You would have had to have gotten very unlucky to see such a difference in distributions even on a small sample like this. But sure, let’s get more samples.
- hn_throwaway_99 3y ago
  Thank you for this! I get frustrated with the "N of only 21? Might as well flip a coin!" responses. Like you say, the whole purpose of statistical testing is to give an accurate, numeric value that says how likely the results are due to chance. One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.
  - lofatdairy 3y ago
    Sensitivity matters in this case because the medium is saliva which contains a fraction of the antigens contained in serum. Specificity is challenging when the biomarkers are non-specific and you only have 1 testing modality but that seems to be not so much the case here.
  - jncfhnb 3y ago
    Well, depends. It’s bayes law fucking us over with the vast majority of people not having breast cancer but really, really needing to know if they do. This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly. Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases. The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.
lofatdairy 3y ago
I appreciate the links but I think this actually misses the point. The novelty isn't in diagnosis of cancer but in sensitivity/cost-efficacy in detection of known biomarkers for breast cancer (and associated risks of recurrence, etc). I'm not familiar with how common ELISA-based HER2 testing takes place, but it seems like it has some impact on drug decisions[^1]. In terms of applicability, it depends on whether or not ELISA is in fact the current standard of care, but it could be useful in low-resource settings where you don't have lab personnel trained to carry out those assays, and drug choice is also restricted by limited availability. Additionally, there's a point-of-care argument as well. Since breast cancer does benefit from early detection, I can see a future in which biomarker testing is a more regular thing, and high saliva concentrations are flagged. At the very least as something worth bringing up at one's next appointment or wtv. [^1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7033231/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7033231/
- tetramer 3y ago
  The standard of care in the US is immunohistochemistry (IHC), with FISH testing in equivocal cases so not really ELISA based. HER2 testing is done on all breast cancers as it affects treatment choices though the majority of breast cancers are not HER2 positive. (HER2 is also expressed in some normal tissue (notably cardiac) and is also seen in other cancers as well).
neuronexmachina 3y ago
> Table I shows the median and the range of digital readings by disease status and overall p-value using the Kruskal–Wallis test to examine if there exist statistically significant distinctions among two or more groups. The overall p-value is significant while the value for HER2 is 0.002, which show the probability of false-positive detection. This indicates that this sensor technology is an efficient way to detect HER2 biomarkers in saliva. > ... In Fig. 5, the test results for detecting CA15-3 of the human samples are displayed. The digital reading decreases from the healthy group to the invasive breast cancer group, indicating an increase in CA15-3 concentration. The median, the range by disease status, and overall p-values analyzed with the Kruskal–Wallis test for the CA15-3 test are listed in Table I. The overall p-value for CA15-3 is 0.005, indicating that this device provides an efficient way to detect the salivary biomarkers related to breast cancer.
- jacquesm 3y ago
  You'd need a much larger n to determine the false positive rate. Sensitivity by itself isn't very useful.
deleted 3y ago
[deleted]

causal 3y ago

Yah headline jumping the gun a bit, but hopefully this motivates funding to get a bigger study / refine the technique.

deleted 3y ago

[deleted]

pvaldes 3y ago

> We tried it with 21 people The new Theranos is here again

dheera 3y ago
No, not at all. Polar opposites. Theranos was actual lies. TFA is just being honest about a low sample size.
- pvaldes 3y ago
  If somebody claims: our device "accurately tests", and we discover later that "In fact the tests hadn't been done still" (at a meaningful way), It should be taken as a red flag. Specially when it came in the same package with "to make lots of tests would be really cheap" (but for some reason they didn't tried it). Maybe be a great, honest work, brilliant step?. Yes. But please don't claim that your model is reproducible if you didn't ever tried seriously to reproduce it first. What if the results are just a random effect?. What if the test says just positive most of the time? The test negative response has been tested in a control group of 4 people? This is 20 bucks well spent. --At this moment-- isn't different than other thousand projects that seemed too good to be true, took the money and soon vanished. I hope to be wrong.
  - dheera 3y ago
    > didn't ever tried seriously to reproduce it first Maybe they found something worth testing and just need more funding to actually do the tests? I have no problem with early free speech about "I found something interesting" as long as you're honest about the sample size and don't misrepresent it.
- jacquesm 3y ago
  Without a larger n you can't really tell what the fp rate is and that means that the accuracy figure isn't very useful. If accuracy holds in light of a false positive rate that is much lower than other tests then this may well be very useful. But you can not conclude that at all based on the evidence presented.
sonicanatidae 3y ago
Not quite. Theranos was stating that they were capable of doing what was impossible prior. The real issue with Theranos was physics. A single drop of blood is simply not a large enough sample for them to do all the testing they claimed. As in, impossible with today's tech and in the near future. These folks are just stating that, "so far, this looks promising". At least, thats my read. YMMV.
- chaxor 3y ago
  It is important to point out that what they claimed to do is not technically impossible actually. They just didn't do it, and lied about it. That was the real problem. It's very unfortunate, because it is technically possible, just very difficult to achieve, even in academia (so it won't be done first in industry). This was the absolute worst part of Theranos, is that they deter others from trying to make headway in the space at all.
  - sonicanatidae 3y ago
    I don't believe they could run a battery of 81+ tests on a drop of blood. There simply isn't enough there. Why do you think they take tubes of blood for testing, currently? The issue is concentrations and the presence of them in sufficient amounts to be detected reliably and consistently. edit: typo.. words are hard.
    - chaxor 3y ago
      It's because that's what the equipment is designed to use, and yes it provides more reliable results with the current technology. Much of the equipment used in healthcare relies on very old technology. This certainly does not mean it cannot be improved to work with less amount of material. There are hand held sequencers for your phone now (minIon) that are very cheap and work. There are also large multimillion dollar bench top machines on the same technology from nanopore. Both are very new technology compared to what we used in early 2000s, and offer various different features and certainties from the analysis. However, it is foolish to say that the minIon is impossible. It works. It's just less resolute than other systems. This is true for many systems which people cry "Theranos" about. Their issue was lying. That was the problem.

fuhrtf 3y ago

That’s a bit harsh. It’s an exploratory study testing a new paradigm.

oliwarner 3y ago
It's only as harsh as the the headline is rose-tinted. "Exploratory study shows promise" is better. When n=21, flipping a coin as about as accurate.
- matthewmacleod 3y ago
  This statement belies a fundamental misunderstanding of statistics on your part.
  - ethanbond 3y ago
    If you're really good at statistics, you can just intuit what seems like a sufficient N. This is actually a powerful statistical method because you can vary your intuition depending on whether you want to believe the study was well-powered or not, enabling you to easily discard Bad Evidence and include Good Evidence. Few understand this.
    - jncfhnb 3y ago
      No, that’s utter nonsense. Statistical power is well defined and the required N to reliably detect an effect is a function of its effect size, which is the thing that wannabe statisticians don’t understand.
      - lazyasciiart 3y ago
        > you can vary your intuition depending on whether you want to believe the study I suspect it is deliberate nonsense.
        solardev 3y ago

redder23 3y ago

Totally agree!

dheera 3y ago

It's still a nonzero sample size, and if anything, says two things: (a) We should do more tests to increase N (b) If a $5 device tests you positive, maybe you should go get checked out for $5000 or whatever the doctors charge you (because insurance often only pays AFTER "shit happens" and often does not pay to test "whether shit might happen"), that you wouldn't have thought of doing otherwise.

jncfhnb 3y ago

You can absolutely draw conclusions about the statistical power of the test. That’s what statistics is for. If this sample data is randomly sampled it looks like it will be a fairly high precision test for invasive cancers, with room for false negatives. You would have had to have gotten very unlucky to see such a difference in distributions even on a small sample like this. But sure, let’s get more samples.

hn_throwaway_99 3y ago
Thank you for this! I get frustrated with the "N of only 21? Might as well flip a coin!" responses. Like you say, the whole purpose of statistical testing is to give an accurate, numeric value that says how likely the results are due to chance. One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.
- lofatdairy 3y ago
  Sensitivity matters in this case because the medium is saliva which contains a fraction of the antigens contained in serum. Specificity is challenging when the biomarkers are non-specific and you only have 1 testing modality but that seems to be not so much the case here.
- jncfhnb 3y ago
  Well, depends. It’s bayes law fucking us over with the vast majority of people not having breast cancer but really, really needing to know if they do. This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly. Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases. The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.

lofatdairy 3y ago

I appreciate the links but I think this actually misses the point. The novelty isn't in diagnosis of cancer but in sensitivity/cost-efficacy in detection of known biomarkers for breast cancer (and associated risks of recurrence, etc). I'm not familiar with how common ELISA-based HER2 testing takes place, but it seems like it has some impact on drug decisions[^1]. In terms of applicability, it depends on whether or not ELISA is in fact the current standard of care, but it could be useful in low-resource settings where you don't have lab personnel trained to carry out those assays, and drug choice is also restricted by limited availability. Additionally, there's a point-of-care argument as well. Since breast cancer does benefit from early detection, I can see a future in which biomarker testing is a more regular thing, and high saliva concentrations are flagged. At the very least as something worth bringing up at one's next appointment or wtv. [^1]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7033231/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7033231/

tetramer 3y ago
The standard of care in the US is immunohistochemistry (IHC), with FISH testing in equivocal cases so not really ELISA based. HER2 testing is done on all breast cancers as it affects treatment choices though the majority of breast cancers are not HER2 positive. (HER2 is also expressed in some normal tissue (notably cardiac) and is also seen in other cancers as well).

neuronexmachina 3y ago

> Table I shows the median and the range of digital readings by disease status and overall p-value using the Kruskal–Wallis test to examine if there exist statistically significant distinctions among two or more groups. The overall p-value is significant while the value for HER2 is 0.002, which show the probability of false-positive detection. This indicates that this sensor technology is an efficient way to detect HER2 biomarkers in saliva. > ... In Fig. 5, the test results for detecting CA15-3 of the human samples are displayed. The digital reading decreases from the healthy group to the invasive breast cancer group, indicating an increase in CA15-3 concentration. The median, the range by disease status, and overall p-values analyzed with the Kruskal–Wallis test for the CA15-3 test are listed in Table I. The overall p-value for CA15-3 is 0.005, indicating that this device provides an efficient way to detect the salivary biomarkers related to breast cancer.

jacquesm 3y ago
You'd need a much larger n to determine the false positive rate. Sensitivity by itself isn't very useful.

deleted 3y ago

[deleted]