92 ms·

Prostate cancer can be precisely diagnosed using a urine test with AI

213 points by samizdis 6y ago

brundolf 6y ago
How do clinical trials work for ML? Since the system is largely a black-box, I imagine that the slightest alteration (training on more data, tweaking any parameters) would require totally re-doing a full gamut of trials, wouldn't it?
- swsieber 6y ago
  I'm not sure why you would need to re-do a full gamut of trials for anything except deciding that the current type of data collected was insufficient. Presumably you record all data, and just split it into test and train sets, and you're off to the races.
  - brundolf 6y ago
    I guess it makes sense that unlike for actual treatments, for diagnostic systems you could just "re-use" the test data from past trials and not have to gather a whole new dataset
- 5440 6y ago
  I'm a regulatory consultant for AI software and submit several AI/ML FDA submissions per week. Overall, I've submitted 100+ AI/ML submissions to date. Generally, FDA has been really focused on locked algorithms even though there has been some guidance lately stating the the new Good Machine Learning Practice (GMLP) will provide some allowance for iterations without submitting new 510ks. I will say that a number of companies are slightly tuning their algorithms without FDA resubmission within the confines of the FDA Guidance "https://www.fda.gov/regulatory-information/search-fda-guidance-documents/deciding-when-submit-510k-software-change-existing-device https://www.fda.gov/regulatory-information/search-fda-guidan...". That said, most companies most companies are just leaving their specifications at sensitivity/specificity of 80% which gives them some leeway to improve above those limits. In truth most companies are above 97%-99%. While most of the submissions I work on are MRI/CT image related, I'm starting to get a lot more in the predictive space centered around ovarian (CA125), breast or PSA cancer scores. Lately pathology AI is escalating rapidly.
  - brundolf 6y ago
    Thanks for the informed and informative answer! I love how common it is on HN to hear from someone with first-hand knowledge of a topic
- Kuinox 6y ago
  A lot of thing are blackbox. There also things that we think we understand, but we do not, thats why we have methodologies like double blind testing to assert that the blackbox is indeed working, and it's behavior is stable enough to be used as a tool.
m00dy 6y ago
where can I find their paper ? note: I've got the link https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://pubs.acs.org/doi/10.1021/acsnano.0c06946
- harveywi 6y ago
  I would recommend a quiet, distraction-free environment.
- ackbar03 6y ago
  You can read it on a boat; You can read it with a goat; You can read it in the rain; You can read it on a train; You can read it in a box; You can read it with a fox; You can read it in a house; You can read it with a mouse; You can read it here or there; You can read it anywhere!
- samizdis 6y ago
  https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....
  - boogies 6y ago
    ([PDF], if blocked paste your url into sci-hub.st or .se, or whatever URLs are listed on https://en.wikipedia.org/wiki/Sci-Hub https://en.wikipedia.org/wiki/Sci-Hub — which got them from https://nitter.dark.fail/sci_hub https://nitter.dark.fail/sci_hub before https://news.ycombinator.com/item?id=25779367 https://news.ycombinator.com/item?id=25779367. They came for the big bad orange man, too few spoke up, and now they’ve come for open access to other knowledge.)
- deleted 6y ago
  [deleted]
gfxgirl 6y ago
So this is super awesome if true and I hope they can use the technique for other cancers. I'm curious though, my bf says his doctor asks if he wants a PSA at his checkup. His doctor then goes on to explain that pretty much all men eventually get prostate cancer and there's not much you can do about it and you're far more likely to die of something else first so, not really much reason to get the PSA Anyone have any other info? Is there actually treatment and is it worth while?
- aclsid 6y ago
  That doesn't seem to match reality, where you have some men living past 100+ years. But I'm not a doctor so...
  - inglor_cz 6y ago
    Not every cancer is created equal. I helped an urologist typeset his PhD thesis about prostate cancer treatment in TeX (he used some math equations within) and he told me the same. Some cancers are aggressive and some are slooooow. In younger patients, cancers tend to move fast and kill fast. But a slow prostate cancer in a 70 y.o. may be better left as it is, because the risks of the operation may actually be higher. This, of course, is a difficult judgment call and belongs to the experts only.
    - ChefboyOG 6y ago
      I always wonder about bias in these statistics around age. It makes intuitive sense to me that a cancer diagnosed in a young patient, who is below the common screening age, is probably being diagnosed because it is presenting serious symptoms (i.e. is growing fast). Cancers diagnosed in a 70 year old, on the other hand, would seem much more likely to be diagnosed while they are asymptomatic and relatively contained, or to be cancers that have been growing slowly for a long time (more likely as the 70 year old has been alive longer). Obviously, I don't know if this is the case or not. If anyone has experience in this field, I'd love to hear about it.
- e40 6y ago
zerof1l 6y ago
This article reads more like PR than research. Research is based on 76 urine samples which is way too little. In the paper, there's little info on model hyper-parameters as well as how they trained and tested them. I suspect they overfitted their model, meaning it just learned all the samples. If that's not the case, the only other explanation for such high accuracy is that there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI.
- ghastmaster 6y ago
  Agreed. This article raises more questions than answers. How were the samples obtained? What does "almost 100 percent accuracy" mean? What was the source of non positive samples? How many total samples were there?
- ttymck 6y ago
  > Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0]. I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger. [0]: https://stats.stackexchange.com/questions/2541/what-references-should-be-cited-to-support-using-30-as-a-large-enough-sample-siz#:~:text=4%20Answers&text=The%20choice%20of%20n%20%3D%2030,greater%20than%2025%20or%2030%22 https://stats.stackexchange.com/questions/2541/what-referenc.... [1]: https://www.datarobot.com/wiki/training-validation-holdout/ https://www.datarobot.com/wiki/training-validation-holdout/
  - Blikkentrekker 6y ago
    “significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.
m3kw9 6y ago
How does a test get standardized? Is there something like drug trials where you need to show accuracy instead of efficacy?
williesleg 6y ago
Just put your mouth diaper back on.
rozab 6y ago
So they used a neural net with only 4 parameters?? And they haven't compared the performance to a simple regression?
bcatanzaro 6y ago
This isn't AI. It's a random forest with FOUR inputs and ONE output. The fact they put a huge big blue AI brain in their paper to represent this algorithm tells me this is mostly hype.
- plaidfuji 6y ago
  What's the cutoff on number of inputs/outputs/parameters for an empirical model to be considered "AI"?
randcraw 6y ago
This news begs a big question: How much does this new test cost? That will decide how it might be adopted. This test requires a chip rather than a simple lab assay. If the cost of the test is comparably as low as PSA ($50) it could entirely replace PSA for screening. However, if the chip is more expensive than PSA, it still may be very useful -- as a second test after a PSA screening test reports a high number but before a biopsy. If this new test could diminish the number of infections from unnecessary biopsies that worry policymakers, routine PSA screening combined with this test could become a low-risk no-brainer new standard.
- elicash 6y ago
  My dad had prostate cancer a few years ago, which was treated successfully with radiation. He now gets his PSA score twice a year -- once with his annual physical, and 6 months later with his cancer doc. I wonder whether the biosensor could be something patients like him could have at home. (Also, now I'm imagining Apple telling customers they can urinate on the next generation of the watch.)
jl2718 6y ago
90% of male cadavers have cancerous tumors in their prostate. The cost of treating men would be enormous. The medical opinion will be that it is harmful diagnostic because the patient (and the medical system) are better off not knowing. If you think this is a joke, it is because your doctor has never said this to you.
- TYMorningCoffee 6y ago
  I'm surprised it's that high so I looked for some papers. Only two references I found so far are below which show much lower than 90% in autopsies. I'm not in the medical field so please help me understand. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4485977/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4485977/ https://academic.oup.com/jnci/article/105/14/1050/965030 https://academic.oup.com/jnci/article/105/14/1050/965030
  - jl2718 6y ago
    Last I saw that stat was 2008, and I think the study was much older. I shouldn’t be so sloppy. Looks like it’s 60% for men 79+, but this is the no prior screening group, which specifically excludes known cases, so maybe a bit higher. Breast cancer is 12% and all cancer is 40% lifetime diagnosis risk in the USA.

213 points by samizdis 6y ago

How do clinical trials work for ML? Since the system is largely a black-box, I imagine that the slightest alteration (training on more data, tweaking any parameters) would require totally re-doing a full gamut of trials, wouldn't it?

swsieber 6y ago
I'm not sure why you would need to re-do a full gamut of trials for anything except deciding that the current type of data collected was insufficient. Presumably you record all data, and just split it into test and train sets, and you're off to the races.
- brundolf 6y ago
  I guess it makes sense that unlike for actual treatments, for diagnostic systems you could just "re-use" the test data from past trials and not have to gather a whole new dataset
5440 6y ago
I'm a regulatory consultant for AI software and submit several AI/ML FDA submissions per week. Overall, I've submitted 100+ AI/ML submissions to date. Generally, FDA has been really focused on locked algorithms even though there has been some guidance lately stating the the new Good Machine Learning Practice (GMLP) will provide some allowance for iterations without submitting new 510ks. I will say that a number of companies are slightly tuning their algorithms without FDA resubmission within the confines of the FDA Guidance "https://www.fda.gov/regulatory-information/search-fda-guidance-documents/deciding-when-submit-510k-software-change-existing-device https://www.fda.gov/regulatory-information/search-fda-guidan...". That said, most companies most companies are just leaving their specifications at sensitivity/specificity of 80% which gives them some leeway to improve above those limits. In truth most companies are above 97%-99%. While most of the submissions I work on are MRI/CT image related, I'm starting to get a lot more in the predictive space centered around ovarian (CA125), breast or PSA cancer scores. Lately pathology AI is escalating rapidly.
- brundolf 6y ago
  Thanks for the informed and informative answer! I love how common it is on HN to hear from someone with first-hand knowledge of a topic
Kuinox 6y ago
A lot of thing are blackbox. There also things that we think we understand, but we do not, thats why we have methodologies like double blind testing to assert that the blackbox is indeed working, and it's behavior is stable enough to be used as a tool.

m00dy 6y ago

where can I find their paper ? note: I've got the link https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://pubs.acs.org/doi/10.1021/acsnano.0c06946

harveywi 6y ago
I would recommend a quiet, distraction-free environment.
ackbar03 6y ago
You can read it on a boat; You can read it with a goat; You can read it in the rain; You can read it on a train; You can read it in a box; You can read it with a fox; You can read it in a house; You can read it with a mouse; You can read it here or there; You can read it anywhere!
samizdis 6y ago
https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano.0c06946 https://sci-hub.do/https://pubs.acs.org/doi/10.1021/acsnano....
- boogies 6y ago
  ([PDF], if blocked paste your url into sci-hub.st or .se, or whatever URLs are listed on https://en.wikipedia.org/wiki/Sci-Hub https://en.wikipedia.org/wiki/Sci-Hub — which got them from https://nitter.dark.fail/sci_hub https://nitter.dark.fail/sci_hub before https://news.ycombinator.com/item?id=25779367 https://news.ycombinator.com/item?id=25779367. They came for the big bad orange man, too few spoke up, and now they’ve come for open access to other knowledge.)
deleted 6y ago
[deleted]

gfxgirl 6y ago

So this is super awesome if true and I hope they can use the technique for other cancers. I'm curious though, my bf says his doctor asks if he wants a PSA at his checkup. His doctor then goes on to explain that pretty much all men eventually get prostate cancer and there's not much you can do about it and you're far more likely to die of something else first so, not really much reason to get the PSA Anyone have any other info? Is there actually treatment and is it worth while?

aclsid 6y ago
That doesn't seem to match reality, where you have some men living past 100+ years. But I'm not a doctor so...
- inglor_cz 6y ago
  Not every cancer is created equal. I helped an urologist typeset his PhD thesis about prostate cancer treatment in TeX (he used some math equations within) and he told me the same. Some cancers are aggressive and some are slooooow. In younger patients, cancers tend to move fast and kill fast. But a slow prostate cancer in a 70 y.o. may be better left as it is, because the risks of the operation may actually be higher. This, of course, is a difficult judgment call and belongs to the experts only.
  - ChefboyOG 6y ago
    I always wonder about bias in these statistics around age. It makes intuitive sense to me that a cancer diagnosed in a young patient, who is below the common screening age, is probably being diagnosed because it is presenting serious symptoms (i.e. is growing fast). Cancers diagnosed in a 70 year old, on the other hand, would seem much more likely to be diagnosed while they are asymptomatic and relatively contained, or to be cancers that have been growing slowly for a long time (more likely as the 70 year old has been alive longer). Obviously, I don't know if this is the case or not. If anyone has experience in this field, I'd love to hear about it.
e40 6y ago

zerof1l 6y ago

This article reads more like PR than research. Research is based on 76 urine samples which is way too little. In the paper, there's little info on model hyper-parameters as well as how they trained and tested them. I suspect they overfitted their model, meaning it just learned all the samples. If that's not the case, the only other explanation for such high accuracy is that there's a very obvious correlation and a human would be able to diagnose cancer with high accuracy also, meaning you don't need any AI.

ghastmaster 6y ago
Agreed. This article raises more questions than answers. How were the samples obtained? What does "almost 100 percent accuracy" mean? What was the source of non positive samples? How many total samples were there?
ttymck 6y ago
> Research is based on 76 urine samples which is way too little I'm sure I'm out of my depth, but I feel compelled to point out that reputable clinical research can be done on even smaller samples[0]. I want to make sure we aren't conflating the training set and the test/validation/holdout set [1] -- I interpreted the article as stating "76 samples in the test set" and assumed the training set is far larger. [0]: https://stats.stackexchange.com/questions/2541/what-references-should-be-cited-to-support-using-30-as-a-large-enough-sample-siz#:~:text=4%20Answers&text=The%20choice%20of%20n%20%3D%2030,greater%20than%2025%20or%2030%22 https://stats.stackexchange.com/questions/2541/what-referenc.... [1]: https://www.datarobot.com/wiki/training-validation-holdout/ https://www.datarobot.com/wiki/training-validation-holdout/
- Blikkentrekker 6y ago
  “significant results” can be yielded from 76 samples, provided they all be independent, and one can practically bet that they are not. But this is a claim of accurate diagnosis which requires more than simple significance. The a.i. being demonstrated to simply be better than chance would be a significant result, which is obviously not a spectacular standard for diagnoses. The issue is that the meaning of “significant” is very often overrated. It means nothing more than “We are highly confident that there is a stronger effect than zero.”; — this does not mean this effect is particularly large, and most importantly, it does not mean the effect is universal.

m3kw9 6y ago

How does a test get standardized? Is there something like drug trials where you need to show accuracy instead of efficacy?

williesleg 6y ago

Just put your mouth diaper back on.

rozab 6y ago

So they used a neural net with only 4 parameters?? And they haven't compared the performance to a simple regression?

bcatanzaro 6y ago

This isn't AI. It's a random forest with FOUR inputs and ONE output. The fact they put a huge big blue AI brain in their paper to represent this algorithm tells me this is mostly hype.

plaidfuji 6y ago
What's the cutoff on number of inputs/outputs/parameters for an empirical model to be considered "AI"?

randcraw 6y ago

This news begs a big question: How much does this new test cost? That will decide how it might be adopted. This test requires a chip rather than a simple lab assay. If the cost of the test is comparably as low as PSA ($50) it could entirely replace PSA for screening. However, if the chip is more expensive than PSA, it still may be very useful -- as a second test after a PSA screening test reports a high number but before a biopsy. If this new test could diminish the number of infections from unnecessary biopsies that worry policymakers, routine PSA screening combined with this test could become a low-risk no-brainer new standard.

elicash 6y ago
My dad had prostate cancer a few years ago, which was treated successfully with radiation. He now gets his PSA score twice a year -- once with his annual physical, and 6 months later with his cancer doc. I wonder whether the biosensor could be something patients like him could have at home. (Also, now I'm imagining Apple telling customers they can urinate on the next generation of the watch.)

jl2718 6y ago

90% of male cadavers have cancerous tumors in their prostate. The cost of treating men would be enormous. The medical opinion will be that it is harmful diagnostic because the patient (and the medical system) are better off not knowing. If you think this is a joke, it is because your doctor has never said this to you.

TYMorningCoffee 6y ago
I'm surprised it's that high so I looked for some papers. Only two references I found so far are below which show much lower than 90% in autopsies. I'm not in the medical field so please help me understand. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4485977/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4485977/ https://academic.oup.com/jnci/article/105/14/1050/965030 https://academic.oup.com/jnci/article/105/14/1050/965030
- jl2718 6y ago
  Last I saw that stat was 2008, and I think the study was much older. I shouldn’t be so sloppy. Looks like it’s 60% for men 79+, but this is the no prior screening group, which specifically excludes known cases, so maybe a bit higher. Breast cancer is 12% and all cancer is 40% lifetime diagnosis risk in the USA.