3 ms·

This article reads more like PR than research. Research is based on 76 urine samples which is way too little. In the paper, there's little info on model hyper-p

This article reads more like PR than research. Research is based on 76 urine samples which is way too little. In the paper, there's little info on model hyper-parameters as well as how they trained and tested them. I suspect they overfitted their model, meaning it just learned all the samples. If that's not the case, the only other explanation for such high accuracy is that there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI.

ghastmaster 6y ago
Agreed. This article raises more questions than answers. How were the samples obtained? What does "almost 100 percent accuracy" mean? What was the source of non positive samples? How many total samples were there?
ttymck 6y ago
> Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0]. I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger. [0]: https://stats.stackexchange.com/questions/2541/what-references-should-be-cited-to-support-using-30-as-a-large-enough-sample-siz#:~:text=4%20Answers&text=The%20choice%20of%20n%20%3D%2030,greater%20than%2025%20or%2030%22 https://stats.stackexchange.com/questions/2541/what-referenc.... [1]: https://www.datarobot.com/wiki/training-validation-holdout/ https://www.datarobot.com/wiki/training-validation-holdout/
- Blikkentrekker 6y ago
  “significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.
  - mbreese 6y ago
    From Supplemental Table S2 in the paper: Non cancer patients: 26, Cancer patients: 25 So, they only had 51 individuals in the study. Then they actually divided the patients in to 3 groups: normal, PCa with pre-DRE, and PCa with post-DRE. The cancer patients thus had 2 samples, pre and post-exam: 26 + (2*25) = 76. Given the false positive rate of PSA screening, I would imagine that including more normal samples would have been more beneficial here. The method of analysis is pretty interesting though, which is where I think the novelty is for this paper. I assume that a more thorough clinical trial is planned with better numbers. This paper looks like it is more of a validation of the device than anything else.
- dekhn 6y ago
  There's a great paper about small sample sizes in the context of experimental surgeries (where usually the work is done at a single regional hospital on 3 patients): "If nobody died, is everything OK?" It's cited in Glantz's Primer on Biostatistics.
- thegginthesky 6y ago
  That's right, sample size can vary a lot depending on several factors such as: * Experiment design * Sampling techniques * Methodology being used on the model But normally with a small sample size a bigger emphasis should be put on performing the appropriate statistical tests on the sample and validating it carefully, before applying any model. There are a lot of assumptions that should be validated if you are working with small data. Unfortunately, I don't have access to the complete paper to verify the methodology used, so take it all with a grain of salt.
  - samizdis 6y ago
    The paper may be read here: https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....
- z77dj3kl 6y ago
  It really depends on the model. If you're estimating a mean from a distribution with low variance, 76 can be quite enough. But for any even half-involved ML model, that's nothing.
- ska 6y ago
  Your interpretation is wrong, unfortunately. The paper is really about developing a multi-marker sensor approach for screening. The analysis, however was done on a small set "Obtained data from 76 urine specimens were partitioned randomly into a training data set (70% of total) and a test data set (30% of total). " Later on it states the 76 samples are from 51 individuals. So the GP complaint is correct in this case. They are comparing a RF approach and a NN approach - there isn't much detail in the paper but it's most likely the analysis is problematic. I suspect this was done because the group is a collection of hardware and clinical folks, not statisticians, and the paper would have benefited by review by a biostats person. You are quite right generally that for some models and some effects, a relatively small verification set can be effective.
Nowado 6y ago
> there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI IF (big if) that's the case, arguably you still prefer something that runs on some electricity in few servers instead of having thousands of human diagnosticians who need to be housed, fed, educated, entertained and later retired.
trhway 6y ago
>The team developed an ultrasensitive semiconductor sensor system capable of simultaneously measuring trace amounts of four selected cancer factors in urine for diagnosing prostate cancer. it would be interesting if they run as a control a human looking at those measurements. One can also wonder how many other cancers can be detected by dipping a chip in urine and whether those chips can be just made into a smartphone accessory for home/field use.
caycep 6y ago
Granted...through some weird HHS/medicare loophole, this is commercializable. Our center has a AI "stroke/clot" detector algorithm software, that's based on a training set of 50 CTs. It's ok, say at a 2nd year medical student level at reading images, i.e. not to the point anyone would trust them. But if the clinical writes in their documentation that the software was utilized to analyze the images, the hospital gets an extra couple grand of reimbursement from the insurance company, apparently. Exactly what rules apply here, I don't know.

ghastmaster 6y ago

Agreed. This article raises more questions than answers. How were the samples obtained? What does "almost 100 percent accuracy" mean? What was the source of non positive samples? How many total samples were there?

ttymck 6y ago

> Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0]. I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger. [0]: https://stats.stackexchange.com/questions/2541/what-references-should-be-cited-to-support-using-30-as-a-large-enough-sample-siz#:~:text=4%20Answers&text=The%20choice%20of%20n%20%3D%2030,greater%20than%2025%20or%2030%22 https://stats.stackexchange.com/questions/2541/what-referenc.... [1]: https://www.datarobot.com/wiki/training-validation-holdout/ https://www.datarobot.com/wiki/training-validation-holdout/

Blikkentrekker 6y ago
“significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.
- mbreese 6y ago
  From Supplemental Table S2 in the paper: Non cancer patients: 26, Cancer patients: 25 So, they only had 51 individuals in the study. Then they actually divided the patients in to 3 groups: normal, PCa with pre-DRE, and PCa with post-DRE. The cancer patients thus had 2 samples, pre and post-exam: 26 + (2*25) = 76. Given the false positive rate of PSA screening, I would imagine that including more normal samples would have been more beneficial here. The method of analysis is pretty interesting though, which is where I think the novelty is for this paper. I assume that a more thorough clinical trial is planned with better numbers. This paper looks like it is more of a validation of the device than anything else.
dekhn 6y ago
There's a great paper about small sample sizes in the context of experimental surgeries (where usually the work is done at a single regional hospital on 3 patients): "If nobody died, is everything OK?" It's cited in Glantz's Primer on Biostatistics.
thegginthesky 6y ago
That's right, sample size can vary a lot depending on several factors such as: * Experiment design * Sampling techniques * Methodology being used on the model But normally with a small sample size a bigger emphasis should be put on performing the appropriate statistical tests on the sample and validating it carefully, before applying any model. There are a lot of assumptions that should be validated if you are working with small data. Unfortunately, I don't have access to the complete paper to verify the methodology used, so take it all with a grain of salt.
- samizdis 6y ago
  The paper may be read here: https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....
z77dj3kl 6y ago
It really depends on the model. If you're estimating a mean from a distribution with low variance, 76 can be quite enough. But for any even half-involved ML model, that's nothing.
ska 6y ago
Your interpretation is wrong, unfortunately. The paper is really about developing a multi-marker sensor approach for screening. The analysis, however was done on a small set "Obtained data from 76 urine specimens were partitioned randomly into a training data set (70% of total) and a test data set (30% of total). " Later on it states the 76 samples are from 51 individuals. So the GP complaint is correct in this case. They are comparing a RF approach and a NN approach - there isn't much detail in the paper but it's most likely the analysis is problematic. I suspect this was done because the group is a collection of hardware and clinical folks, not statisticians, and the paper would have benefited by review by a biostats person. You are quite right generally that for some models and some effects, a relatively small verification set can be effective.

Nowado 6y ago

> there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI IF (big if) that's the case, arguably you still prefer something that runs on some electricity in few servers instead of having thousands of human diagnosticians who need to be housed, fed, educated, entertained and later retired.

trhway 6y ago

>The team developed an ultrasensitive semiconductor sensor system capable of simultaneously measuring trace amounts of four selected cancer factors in urine for diagnosing prostate cancer. it would be interesting if they run as a control a human looking at those measurements. One can also wonder how many other cancers can be detected by dipping a chip in urine and whether those chips can be just made into a smartphone accessory for home/field use.

caycep 6y ago

Granted...through some weird HHS/medicare loophole, this is commercializable. Our center has a AI "stroke/clot" detector algorithm software, that's based on a training set of 50 CTs. It's ok, say at a 2nd year medical student level at reading images, i.e. not to the point anyone would trust them. But if the clinical writes in their documentation that the software was utilized to analyze the images, the hospital gets an extra couple grand of reimbursement from the insurance company, apparently. Exactly what rules apply here, I don't know.