4 ms·

> Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical researc

by ttymck 6y ago

> Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0]. I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger. [0]: https://stats.stackexchange.com/questions/2541/what-references-should-be-cited-to-support-using-30-as-a-large-enough-sample-siz#:~:text=4%20Answers&text=The%20choice%20of%20n%20%3D%2030,greater%20than%2025%20or%2030%22 https://stats.stackexchange.com/questions/2541/what-referenc.... [1]: https://www.datarobot.com/wiki/training-validation-holdout/ https://www.datarobot.com/wiki/training-validation-holdout/

Blikkentrekker 6y ago
“significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.
- mbreese 6y ago
  From Supplemental Table S2 in the paper: Non cancer patients: 26, Cancer patients: 25 So, they only had 51 individuals in the study. Then they actually divided the patients in to 3 groups: normal, PCa with pre-DRE, and PCa with post-DRE. The cancer patients thus had 2 samples, pre and post-exam: 26 + (2*25) = 76. Given the false positive rate of PSA screening, I would imagine that including more normal samples would have been more beneficial here. The method of analysis is pretty interesting though, which is where I think the novelty is for this paper. I assume that a more thorough clinical trial is planned with better numbers. This paper looks like it is more of a validation of the device than anything else.
dekhn 6y ago
There's a great paper about small sample sizes in the context of experimental surgeries (where usually the work is done at a single regional hospital on 3 patients): "If nobody died, is everything OK?" It's cited in Glantz's Primer on Biostatistics.
thegginthesky 6y ago
That's right, sample size can vary a lot depending on several factors such as: * Experiment design * Sampling techniques * Methodology being used on the model But normally with a small sample size a bigger emphasis should be put on performing the appropriate statistical tests on the sample and validating it carefully, before applying any model. There are a lot of assumptions that should be validated if you are working with small data. Unfortunately, I don't have access to the complete paper to verify the methodology used, so take it all with a grain of salt.
- samizdis 6y ago
  The paper may be read here: https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....
z77dj3kl 6y ago
It really depends on the model. If you're estimating a mean from a distribution with low variance, 76 can be quite enough. But for any even half-involved ML model, that's nothing.
ska 6y ago
Your interpretation is wrong, unfortunately. The paper is really about developing a multi-marker sensor approach for screening. The analysis, however was done on a small set "Obtained data from 76 urine specimens were partitioned randomly into a training data set (70% of total) and a test data set (30% of total). " Later on it states the 76 samples are from 51 individuals. So the GP complaint is correct in this case. They are comparing a RF approach and a NN approach - there isn't much detail in the paper but it's most likely the analysis is problematic. I suspect this was done because the group is a collection of hardware and clinical folks, not statisticians, and the paper would have benefited by review by a biostats person. You are quite right generally that for some models and some effects, a relatively small verification set can be effective.

by ttymck 6y ago

Blikkentrekker 6y ago

“significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.

mbreese 6y ago
From Supplemental Table S2 in the paper: Non cancer patients: 26, Cancer patients: 25 So, they only had 51 individuals in the study. Then they actually divided the patients in to 3 groups: normal, PCa with pre-DRE, and PCa with post-DRE. The cancer patients thus had 2 samples, pre and post-exam: 26 + (2*25) = 76. Given the false positive rate of PSA screening, I would imagine that including more normal samples would have been more beneficial here. The method of analysis is pretty interesting though, which is where I think the novelty is for this paper. I assume that a more thorough clinical trial is planned with better numbers. This paper looks like it is more of a validation of the device than anything else.

dekhn 6y ago

There's a great paper about small sample sizes in the context of experimental surgeries (where usually the work is done at a single regional hospital on 3 patients): "If nobody died, is everything OK?" It's cited in Glantz's Primer on Biostatistics.

thegginthesky 6y ago

That's right, sample size can vary a lot depending on several factors such as: * Experiment design * Sampling techniques * Methodology being used on the model But normally with a small sample size a bigger emphasis should be put on performing the appropriate statistical tests on the sample and validating it carefully, before applying any model. There are a lot of assumptions that should be validated if you are working with small data. Unfortunately, I don't have access to the complete paper to verify the methodology used, so take it all with a grain of salt.

samizdis 6y ago
The paper may be read here: https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....

z77dj3kl 6y ago

It really depends on the model. If you're estimating a mean from a distribution with low variance, 76 can be quite enough. But for any even half-involved ML model, that's nothing.

ska 6y ago

Your interpretation is wrong, unfortunately. The paper is really about developing a multi-marker sensor approach for screening. The analysis, however was done on a small set "Obtained data from 76 urine specimens were partitioned randomly into a training data set (70% of total) and a test data set (30% of total). " Later on it states the 76 samples are from 51 individuals. So the GP complaint is correct in this case. They are comparing a RF approach and a NN approach - there isn't much detail in the paper but it's most likely the analysis is problematic. I suspect this was done because the group is a collection of hardware and clinical folks, not statisticians, and the paper would have benefited by review by a biostats person. You are quite right generally that for some models and some effects, a relatively small verification set can be effective.