4 ms·

With so few controls, it’s still not a well designed comparison. Most scientists deal with low sample sizes, especially in first trials. However, what is often

With so few controls, it’s still not a well designed comparison. Most scientists deal with low sample sizes, especially in first trials. However, what is often overlooked is the need for sufficient numbers of negative controls. If they only had 4 control patients, that’s completely inadequate to draw any real conclusions. With access to a biobank at U of Florida, they should have been able to test more samples — especially with a $5 test. There are two other issues with this paper [1] that I quickly see. Their main figure lacks errors bars. It’s pretty clear to me that the groups would overlap quite a bit. More numbers would make this problem clearer (either to narrow error bars or clearly show an overlap). The lack of error bars across the paper make me think they didn’t do any technical replicates, which is also a problem. I’m also not sure a one way test is correct here, but I’m also not entirely sure how they are measuring the data. In this one way analysis, all you can tell is if one group is higher than the other. When the data are so unbalanced, what you don’t see is what the predictive value of the test is — false positive/negative. That’s the real issue here. It looks like you’d have a really high false negative rate, as the cancer samples have a much wider range than the controls. This is the worst thing you can have in a test like this. Finally — this paper was published in the “journal of vacuum sciences and technology B.” There is no way this paper got a valid peer review to make these claims. I don’t know anything about this journal, but I doubt it has much experience with cancer testing. [1] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/High-sensitivity-saliva-based-biosensor-in https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...

jncfhnb 3y ago
A small control sample is not “few controls”. Misleading term. “Negative controls” is not a thing either. Just the control group is fine. The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate. A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little. If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.
- mbreese 3y ago
  This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations. You can argue about the terms "negative control" (which is appropriate here, as there is a positive control test in the paper), but there are only 4 non-cancer samples tested. That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer. But, that's not really the point. They aren't trying to diagnose cancer vs. healthy. Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample. I also mis-read part of the paper the first time. For the HER2 cases, there aren't 4 negative samples -- there are 20. There is only one positive sample. Part of the problem is really how they are presenting the data -- it is not all clear what they are testing. But, there is only one HER2+ sample in the mix. One... N=1. Samples include: * Non-cancer: 4 * In situ cancer: 3 * Invasive cancer, HER2-: 13 * Invasive cancer, HER2+: 1 What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions). Note: I'm not trying to say that the authors of the study are wrong or are trying to deliberately mislead people. There is so much here that could have been corrected to make this a much stronger paper. To me, this seems like a paper where the authors are likely engineers and not that well versed in biomedical statistics. The paper is published in a physics journal, so the journal itself is not a good place to make some of these arguments. Is the idea of a non-invasive test worthwhile? Yes! Absolutely. But they didn't show that it was a good test of clinical utility. They showed that it could measure differences in protein concentrations from saliva. That's not nothing, but that's it. Now, if that is an appropriate way to differentiate patients is a completely different question and requires substantially more testing (and orders of magnitude more patients).
  - jncfhnb 3y ago
    > This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations. Complete Nonsense > That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer. True. But it is a promising initial signal that the distribution of non cancerous folks is probably very different from the invasive cancer folks. The effect size here is huge. “Range” is less interesting than “Distribution” > Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. You don’t really need error bars when you’re showing all of a small number of data points. But sure whatever > Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample. Would be nice. Sounds like they did ten measurements per. > What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions). I’m not clear why you’re saying HER+ vs HER- is the important difference here.

jncfhnb 3y ago

A small control sample is not “few controls”. Misleading term. “Negative controls” is not a thing either. Just the control group is fine. The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate. A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little. If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.

mbreese 3y ago
This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations. You can argue about the terms "negative control" (which is appropriate here, as there is a positive control test in the paper), but there are only 4 non-cancer samples tested. That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer. But, that's not really the point. They aren't trying to diagnose cancer vs. healthy. Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample. I also mis-read part of the paper the first time. For the HER2 cases, there aren't 4 negative samples -- there are 20. There is only one positive sample. Part of the problem is really how they are presenting the data -- it is not all clear what they are testing. But, there is only one HER2+ sample in the mix. One... N=1. Samples include: * Non-cancer: 4 * In situ cancer: 3 * Invasive cancer, HER2-: 13 * Invasive cancer, HER2+: 1 What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions). Note: I'm not trying to say that the authors of the study are wrong or are trying to deliberately mislead people. There is so much here that could have been corrected to make this a much stronger paper. To me, this seems like a paper where the authors are likely engineers and not that well versed in biomedical statistics. The paper is published in a physics journal, so the journal itself is not a good place to make some of these arguments. Is the idea of a non-invasive test worthwhile? Yes! Absolutely. But they didn't show that it was a good test of clinical utility. They showed that it could measure differences in protein concentrations from saliva. That's not nothing, but that's it. Now, if that is an appropriate way to differentiate patients is a completely different question and requires substantially more testing (and orders of magnitude more patients).
- jncfhnb 3y ago
  > This is an unbalanced study. As such, it doesn't tell you anything about the ability to differentiate between the three populations. Complete Nonsense > That is not enough to be able to adequately know range of measurable values in the population of patients w/o cancer. True. But it is a promising initial signal that the distribution of non cancerous folks is probably very different from the invasive cancer folks. The effect size here is huge. “Range” is less interesting than “Distribution” > Error bars here are absolutely necessary. Two reasons: First, you want to know the approximate ranges for each group in Figures 3 and 5. Not showing them is misleading. You don’t really need error bars when you’re showing all of a small number of data points. But sure whatever > Secondly -- you actually also want error bars for each patient sample. I'd expect for there to be at least three replicates for each saliva sample to show that the strips are able to consistently measure a known value from each sample. Would be nice. Sounds like they did ten measurements per. > What you'd really like to show is that the HER2+ patients could be differentiated from HER2- patients. Which, does look really good, but with only one HER2+ sample, you really can't tell much. (And the presence of so much signal in the HER2- samples raises some very interesting biological/mechanistic questions). I’m not clear why you’re saying HER+ vs HER- is the important difference here.