4 ms·

You can absolutely draw conclusions about the statistical power of the test. That’s what statistics is for. If this sample data is randomly sampled it looks li

You can absolutely draw conclusions about the statistical power of the test. That’s what statistics is for. If this sample data is randomly sampled it looks like it will be a fairly high precision test for invasive cancers, with room for false negatives. You would have had to have gotten very unlucky to see such a difference in distributions even on a small sample like this. But sure, let’s get more samples.

hn_throwaway_99 3y ago
Thank you for this! I get frustrated with the "N of only 21? Might as well flip a coin!" responses. Like you say, the whole purpose of statistical testing is to give an accurate, numeric value that says how likely the results are due to chance. One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.
- lofatdairy 3y ago
  Sensitivity matters in this case because the medium is saliva which contains a fraction of the antigens contained in serum. Specificity is challenging when the biomarkers are non-specific and you only have 1 testing modality but that seems to be not so much the case here.
- jncfhnb 3y ago
  Well, depends. It’s bayes law fucking us over with the vast majority of people not having breast cancer but really, really needing to know if they do. This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly. Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases. The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.
- ryandrake 3y ago
  No matter what the value of N is, someone always comes out of the woodwork to complain about it. It's a pretty common criticism of any study that relies on statistics, and one anyone can make in a few sentences.
- wolverine876 3y ago
  People on HN (maybe elsewhere too) devote a lot of attention to sample sizes. I don't know the upthread commenter at all, but in general I suspect it's because that is an easy thing to understand about research and statistics, and it's a valid critique they've seen professionals use. A normal human fallacy is to focus on the thing you understand, that is easy to understand (e.g., easy to quantify), and overlook the difficult issues that are far more important. There is much more, of much more importance going on in most of these studies, and in their statistics and validity, than the sample size.
  - treflop 3y ago
    Elsewhere too I first took stats in high school and we read a bunch of valid small sample size studies so I don’t know where people are failing to get educated. There were so many factors going into whether a study was good.
    - ajb 3y ago
      Exactly. Famous example: Lady tasting Tea [1]. N=1 or N=8 depending on how you look at it. Still significant. This is the one that proves that it's possible to tell whether you added the milk first or second. [1] https://en.wikipedia.org/wiki/Lady_tasting_tea https://en.wikipedia.org/wiki/Lady_tasting_tea
- renewiltord 3y ago
  This is why science has gotten harder these days. When there were only half a billion people, sampling 500 got you 1 in a million coverage. Now there are 8 billion people, so it's worth 16 times less. As more people are made, science will suffer because the percentage is so much less. In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people. Sometimes, it's even better to just sample people in one house. 100% coverage.
  - hn_throwaway_99 3y ago
    TBH, I can't tell if this is sarcastic or not, because there is so much in this comment that belies a fundamental misunderstanding of statistical testing and statistical power, which is the point I was trying to make. > Now there are 8 billion people, so it's worth 16 times less. That is not how statistics works, at all. > In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people. These are the sentences that made me think this had to be satire (presumably you want "science" to apply to places outside of Colma...), but in all honesty these days it's really hard to tell.
- serial_dev 3y ago
  My assumption is that 98% of people have no idea what the right sample size is for this particular experiment, including myself of course. To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?
  - hn_throwaway_99 3y ago
    > To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much? This is the point that you are fundamentally misunderstanding. Nobody is making the argument that "21 is such a perfect number in this case". The rest of your sentences ("Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?") seem to point to a belief that people are pulling numbers based on a finger in the wind. The whole point of (a lot of) statistical testing is that it allows you to come up with a specific number that determines how likely a particular result is due to chance. That is what the p < .05 "standard" is about - it's a determination that the results have less than a 5% likelihood to be due to random chance (though that 5% number used as "significant" is just basically pulled out of thin air, and p-hacking is another topic...) That is, I and the comment I replied to aren't making the argument that 21 "is such a perfect number". We're making the argument that even with a small sample size it's possible to determine the relative error bars, with precision, using statistical methods, not just pulling a number based on feels. Yes, larger sample sizes reduce the size of those error bars. But often not in ways that are "intuitive". You have to do the math. None of what I wrote above is meant to imply that statistical tests can't be misused, or that they often require assumptions about the underlying population distribution that may not be correct.
  - jncfhnb 3y ago
    It is a function of the effect size, not the experiment. When the effect size is large, you need fewer samples to detect it reliably. In this case it looks pretty large. Would you want more people before rolling it out? Of course. But it’s very unlikely to be vaporware provided the samples are random.
- kurthr 3y ago
  If you only flip a coin to the same side 21 times in a row, it could just be chance! It's only 1 in a million, but at about 10 the same I'd start to figure the coin wasn't fair.
kenjackson 3y ago
And this result probably helps the funding to get more patients. I’ve never worked in clinical trials but it seems like it would be so tiresome.
- bookofjoe 3y ago
  I ran many clinical trials during my career as an academic neurosurgical anesthesiologist. "... it seems like it would be so tiresome" is accurate; it's also exhausting in terms of the huge amount of time and effort required to get a study approved by the institutional review board; getting informed consent from prospective patients after an exhaustive explanation repeated over and over to each individual; actually doing the study; organizing the results; doing the statistical analysis required; writing the paper; waiting months to hear back from the journal's reviewers, often receiving a rejection letter; resubmitting the paper to another journal, sometimes several more; after having it accepted, revising the paper per the reviewers' comments before resubmission, sometimes going back and forth for months. You can find my published papers here: https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en
  - wolverine876 3y ago
    So how do you feel about it? :) Seriously, what are you feelings about it after all those experiences? You forgot the end of the story: After all that, it's done and published, and then a random person with no expertise and who barely read the paper posts on HN: the sample size is too small - as if you were in your first week of statistics 101 - and therefore the whole thing must be invalid! :)
    - bookofjoe 3y ago
      How do I feel about it? My feelings after all those experiences? I dunno... that was me then, I guess: hard core academic with a drive/compulsion to publish good work. I admit to being amused by comments here about statistics by people who wouldn't know a Bonferroni correction from a bonfire.
mbreese 3y ago
With so few controls, it’s still not a well designed comparison. Most scientists deal with low sample sizes, especially in first trials. However, what is often overlooked is the need for sufficient numbers of negative controls. If they only had 4 control patients, that’s completely inadequate to draw any real conclusions. With access to a biobank at U of Florida, they should have been able to test more samples — especially with a $5 test. There are two other issues with this paper [1] that I quickly see. Their main figure lacks errors bars. It’s pretty clear to me that the groups would overlap quite a bit. More numbers would make this problem clearer (either to narrow error bars or clearly show an overlap). The lack of error bars across the paper make me think they didn’t do any technical replicates, which is also a problem. I’m also not sure a one way test is correct here, but I’m also not entirely sure how they are measuring the data. In this one way analysis, all you can tell is if one group is higher than the other. When the data are so unbalanced, what you don’t see is what the predictive value of the test is — false positive/negative. That’s the real issue here. It looks like you’d have a really high false negative rate, as the cancer samples have a much wider range than the controls. This is the worst thing you can have in a test like this. Finally — this paper was published in the “journal of vacuum sciences and technology B.” There is no way this paper got a valid peer review to make these claims. I don’t know anything about this journal, but I doubt it has much experience with cancer testing. [1] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/High-sensitivity-saliva-based-biosensor-in https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...
- jncfhnb 3y ago
  A small control sample is not “few controls”. Misleading term. “Negative controls” is not a thing either. Just the control group is fine. The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate. A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little. If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.
e63f67dd-065b 3y ago
It's been a while since I took statistics, but isn't the whole point of <thing> testing that it needs to have a high Bayes factor, and to be confident that said Bayes factor is high? In this case, and my lack of understanding in both bio and stats is showing here, we're trying to develop a test for breast cancer. An ideal test will have high sensitivity and specificity, and a tight confidence interval for both of those numbers. This way we can be confident that a positive/negative test actually moves our priors meaningfully in either direction. I guess my question/comment is more on the fact that I don't see how any of the results shown actually translate, as the headline suggests, into a cancer test of high power. The priors for any kind of cancer is pretty low from what I can find, so we need high power tests in both the positive and negative direction to meaningfully effect health outcomes. I can't find any CI numbers in the paper, which may just be me not reading it closely enough, but it doesn't help my confidence.
- jncfhnb 3y ago
  Suppose you had cancer and could roll a die, and if the die came up six, it would tell you with high certainty you had cancer, and if it came up as anything else, it would tell you nothing. If the test is very cheap, it’s probably a great test presuming you get to see the dice roll itself. The cost and invasiveness of the test is important.
an_d_rew 3y ago
The problem here is that the underlying and unknown correlations in the sample (aka people) is that they are NOT independent. The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations. The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy". So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.
- jncfhnb 3y ago
  I feel like you’re taking a very roundabout way to describe the concept of sample bias. So long as it’s a random sample, this is accounted for in the statistics. Yes, we should still get more data all the same.

hn_throwaway_99 3y ago

Thank you for this! I get frustrated with the "N of only 21? Might as well flip a coin!" responses. Like you say, the whole purpose of statistical testing is to give an accurate, numeric value that says how likely the results are due to chance. One thing I'd note, though, is that the paper's title is "High sensitivity saliva-based biosensor in detection of breast cancer biomarkers: HER2 and CA15-3". My understanding is that sensitivity was never really the problem with breast cancer detection - it's specificity that is the real challenge with all types of broadly-deployed medical screening tools.

lofatdairy 3y ago
Sensitivity matters in this case because the medium is saliva which contains a fraction of the antigens contained in serum. Specificity is challenging when the biomarkers are non-specific and you only have 1 testing modality but that seems to be not so much the case here.
jncfhnb 3y ago
Well, depends. It’s bayes law fucking us over with the vast majority of people not having breast cancer but really, really needing to know if they do. This is why it’s better to have tiers of confidence for actual policy decisions and not just confusion matrix results frankly. Like if you run this test and you’re in a cohort that is 99.999% likely to be cancer, that’s useful info. It’s not a problem, per se, if the test instead comes back with an answer of mixed certainty. We just need to be comfortable with getting neither a positive nor a negative prediction; nor expecting it to cover all cases. The policy driving thresholds on tests need to consider the weights of each possible outcome to determine what to do.
ryandrake 3y ago
No matter what the value of N is, someone always comes out of the woodwork to complain about it. It's a pretty common criticism of any study that relies on statistics, and one anyone can make in a few sentences.
wolverine876 3y ago
People on HN (maybe elsewhere too) devote a lot of attention to sample sizes. I don't know the upthread commenter at all, but in general I suspect it's because that is an easy thing to understand about research and statistics, and it's a valid critique they've seen professionals use. A normal human fallacy is to focus on the thing you understand, that is easy to understand (e.g., easy to quantify), and overlook the difficult issues that are far more important. There is much more, of much more importance going on in most of these studies, and in their statistics and validity, than the sample size.
- treflop 3y ago
  Elsewhere too I first took stats in high school and we read a bunch of valid small sample size studies so I don’t know where people are failing to get educated. There were so many factors going into whether a study was good.
  - ajb 3y ago
    Exactly. Famous example: Lady tasting Tea [1]. N=1 or N=8 depending on how you look at it. Still significant. This is the one that proves that it's possible to tell whether you added the milk first or second. [1] https://en.wikipedia.org/wiki/Lady_tasting_tea https://en.wikipedia.org/wiki/Lady_tasting_tea
renewiltord 3y ago
This is why science has gotten harder these days. When there were only half a billion people, sampling 500 got you 1 in a million coverage. Now there are 8 billion people, so it's worth 16 times less. As more people are made, science will suffer because the percentage is so much less. In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people. Sometimes, it's even better to just sample people in one house. 100% coverage.
- hn_throwaway_99 3y ago
  TBH, I can't tell if this is sarcastic or not, because there is so much in this comment that belies a fundamental misunderstanding of statistical testing and statistical power, which is the point I was trying to make. > Now there are 8 billion people, so it's worth 16 times less. That is not how statistics works, at all. > In fact, this is why I always trust science in small towns more. In Colma, for instance, you sample 500 and you have covered 33% of the people. These are the sentences that made me think this had to be satire (presumably you want "science" to apply to places outside of Colma...), but in all honesty these days it's really hard to tell.
serial_dev 3y ago
My assumption is that 98% of people have no idea what the right sample size is for this particular experiment, including myself of course. To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?
- hn_throwaway_99 3y ago
  > To all who criticize and ridicule someone who would like to have more samples, why do you think 21 is such a perfect number in this case? Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much? This is the point that you are fundamentally misunderstanding. Nobody is making the argument that "21 is such a perfect number in this case". The rest of your sentences ("Wouldn't 15 be enough, if statistics and all applies? 10? 1? Would 30 be too much?") seem to point to a belief that people are pulling numbers based on a finger in the wind. The whole point of (a lot of) statistical testing is that it allows you to come up with a specific number that determines how likely a particular result is due to chance. That is what the p < .05 "standard" is about - it's a determination that the results have less than a 5% likelihood to be due to random chance (though that 5% number used as "significant" is just basically pulled out of thin air, and p-hacking is another topic...) That is, I and the comment I replied to aren't making the argument that 21 "is such a perfect number". We're making the argument that even with a small sample size it's possible to determine the relative error bars, with precision, using statistical methods, not just pulling a number based on feels. Yes, larger sample sizes reduce the size of those error bars. But often not in ways that are "intuitive". You have to do the math. None of what I wrote above is meant to imply that statistical tests can't be misused, or that they often require assumptions about the underlying population distribution that may not be correct.
- jncfhnb 3y ago
  It is a function of the effect size, not the experiment. When the effect size is large, you need fewer samples to detect it reliably. In this case it looks pretty large. Would you want more people before rolling it out? Of course. But it’s very unlikely to be vaporware provided the samples are random.
kurthr 3y ago
If you only flip a coin to the same side 21 times in a row, it could just be chance! It's only 1 in a million, but at about 10 the same I'd start to figure the coin wasn't fair.

kenjackson 3y ago

And this result probably helps the funding to get more patients. I’ve never worked in clinical trials but it seems like it would be so tiresome.

bookofjoe 3y ago
I ran many clinical trials during my career as an academic neurosurgical anesthesiologist. "... it seems like it would be so tiresome" is accurate; it's also exhausting in terms of the huge amount of time and effort required to get a study approved by the institutional review board; getting informed consent from prospective patients after an exhaustive explanation repeated over and over to each individual; actually doing the study; organizing the results; doing the statistical analysis required; writing the paper; waiting months to hear back from the journal's reviewers, often receiving a rejection letter; resubmitting the paper to another journal, sometimes several more; after having it accepted, revising the paper per the reviewers' comments before resubmission, sometimes going back and forth for months. You can find my published papers here: https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en https://scholar.google.com/citations?user=5DdrMc8AAAAJ&hl=en
- wolverine876 3y ago
  So how do you feel about it? :) Seriously, what are you feelings about it after all those experiences? You forgot the end of the story: After all that, it's done and published, and then a random person with no expertise and who barely read the paper posts on HN: the sample size is too small - as if you were in your first week of statistics 101 - and therefore the whole thing must be invalid! :)
  - bookofjoe 3y ago
    How do I feel about it? My feelings after all those experiences? I dunno... that was me then, I guess: hard core academic with a drive/compulsion to publish good work. I admit to being amused by comments here about statistics by people who wouldn't know a Bonferroni correction from a bonfire.

mbreese 3y ago

With so few controls, it’s still not a well designed comparison. Most scientists deal with low sample sizes, especially in first trials. However, what is often overlooked is the need for sufficient numbers of negative controls. If they only had 4 control patients, that’s completely inadequate to draw any real conclusions. With access to a biobank at U of Florida, they should have been able to test more samples — especially with a $5 test. There are two other issues with this paper [1] that I quickly see. Their main figure lacks errors bars. It’s pretty clear to me that the groups would overlap quite a bit. More numbers would make this problem clearer (either to narrow error bars or clearly show an overlap). The lack of error bars across the paper make me think they didn’t do any technical replicates, which is also a problem. I’m also not sure a one way test is correct here, but I’m also not entirely sure how they are measuring the data. In this one way analysis, all you can tell is if one group is higher than the other. When the data are so unbalanced, what you don’t see is what the predictive value of the test is — false positive/negative. That’s the real issue here. It looks like you’d have a really high false negative rate, as the cancer samples have a much wider range than the controls. This is the worst thing you can have in a test like this. Finally — this paper was published in the “journal of vacuum sciences and technology B.” There is no way this paper got a valid peer review to make these claims. I don’t know anything about this journal, but I doubt it has much experience with cancer testing. [1] https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/High-sensitivity-saliva-based-biosensor-in https://pubs.aip.org/avs/jvb/article/42/2/023202/3262988/Hig...

jncfhnb 3y ago
A small control sample is not “few controls”. Misleading term. “Negative controls” is not a thing either. Just the control group is fine. The error bars are not really needed. Your eyeballs are doing just fine. Yes, they overlap. Yes, it would likely have a high false negative rate. A false negative rate is not a problem for a cheap test. This test is not meant to replace higher quality and more invasive tests. A high precision, low recall test still has significant value. You merely have to accept that a negative result tells you very little. If we imagined for instance, that the false positive rate is very low despite the overwhelmingly larger population of people without cancer, then this would be of enormous value.

e63f67dd-065b 3y ago

It's been a while since I took statistics, but isn't the whole point of <thing> testing that it needs to have a high Bayes factor, and to be confident that said Bayes factor is high? In this case, and my lack of understanding in both bio and stats is showing here, we're trying to develop a test for breast cancer. An ideal test will have high sensitivity and specificity, and a tight confidence interval for both of those numbers. This way we can be confident that a positive/negative test actually moves our priors meaningfully in either direction. I guess my question/comment is more on the fact that I don't see how any of the results shown actually translate, as the headline suggests, into a cancer test of high power. The priors for any kind of cancer is pretty low from what I can find, so we need high power tests in both the positive and negative direction to meaningfully effect health outcomes. I can't find any CI numbers in the paper, which may just be me not reading it closely enough, but it doesn't help my confidence.

jncfhnb 3y ago
Suppose you had cancer and could roll a die, and if the die came up six, it would tell you with high certainty you had cancer, and if it came up as anything else, it would tell you nothing. If the test is very cheap, it’s probably a great test presuming you get to see the dice roll itself. The cost and invasiveness of the test is important.

an_d_rew 3y ago

The problem here is that the underlying and unknown correlations in the sample (aka people) is that they are NOT independent. The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations. The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy". So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.

jncfhnb 3y ago
I feel like you’re taking a very roundabout way to describe the concept of sample bias. So long as it’s a random sample, this is accounted for in the statistics. Yes, we should still get more data all the same.