8 ms·
Many FDA-approved AI medical devices are not trained on real patient data
- AbstractH24 2y agoAs long as these are diagnostic aids, not treatments like a pacemaker or medication, I don't see this as so concerning.
- Sevii 2y agoNot a surprise. I worked on a ‘AI’ health insurance product for several years and even getting access to data was a struggle.
- dylan604 2y agoI really wish you had used "impossible" instead of "a struggle".
- deleted 2y ago[deleted]
- LifeIsBio 2y agoYep, I'm in the rare disease space. "impossible" is pretty appropriate. It's tricky. On the one hand, it's obviously not appropriate to be flippant about patient privacy. On the other, it's clearly that advancements in human health are being hindered by our current approach to (dis)allowing researchers access to data.
- AlexandrB 2y agoFor me it's a situation of "once bitten, twice shy". What are the odds the medical data intended for research will be handled correctly and not used outside of its intended purpose?
- LifeIsBio 2y agoNonzero, for sure.
- kfajdsl 2y agoWhat are the potential downsides to misuse of health data? Genuinely asking - I'm not sure what someone malicious would do with my health records, especially if it's anonymized.
- BadHumans 2y agoThere are several examples of anonymized data not being so anonymized after all and able to be traced back to the person. As far as what could someone malicious do with health records, you have a contingent of people in some states hunting down women for having abortions so that might be something you don't want getting out there. Or you might be someone in a very religious area and you don't want people finding out you're getting AIDS treatment.
- bpodgursky 2y agoI understand the theoretical concerns in these cases, but IMO it does not weigh heavily against the (conservatively) hundreds of thousands of annual deaths due to hindered medical research. It's hard to overstress enough how impossible it is to do even basic research across institutional health datasets, even you're a giant organization with a compliance team. It's soul-draining and frankly the reason a lot of smart people jump ship and work in finance or crypto or whatever, where you can accomplish something even if it's goofy.
- JTyQZSnP3cQGa8B 2y agoNitpicking but he gave you practical examples of stuff that already happened, not theoretical ones.
- bpodgursky 2y ago
- dylan604 2y agoIf a researcher can get the data, then so could someone else with less altruistic motives. So the good actor is slowed because of the bad actor. Unfortunately, there's very little way to prove the good is good and not crossing their fingers behind their back.
- vasco 2y agoPerhaps opening up access to data 5 years after the patient is dead would fix this.
- anitil 2y agoI'm not sure if this is relevant, but Ben Goldachre in the UK is working on how to get access to NHS patient data in a privacy-preserving manner [0]. My understanding is that you essentially submit your analysis and it is run against live data but you only recieve summary results. I'm not sure if this could be adapted to training. [0] https://assets.publishing.service.gov.uk/media/624ea0ade90e072a014d508a/goldacre-review-using-health-data-for-research-and-analysis.pdf https://assets.publishing.service.gov.uk/media/624ea0ade90e0...
- deusexml 2y agoThe problem with this approach is it is very hard to do data science with messy clinical data when you have no mechanism of investigating the data yourself.
- mandevil 2y agoI interviewed with a company that built a Electronic Medical Records system optimized for cancer treatment, licensed it for cheap, then took the data and paid for medical professionals to spend time normalizing the free-text notes data from the doctors (this was ~a decade ago, when LLM's were not really a thing, so they got trained professionals to do it instead). That was all a play to get the level of data they wanted for training AI to make new discoveries. They got acquired by a major pharmaceutical company, but I don't know of any major discoveries made from their data. Because even with real data this is a hard problem.
- w_for_wumbo 2y agoMaybe we could have an option to be a data-donor as opposed to only having the option to donate your organs for the purpose of science.
- thecosas 2y agoSomething like this or something more akin to "when I die, please take all my healthcare records, data from smart watch, etc and make that available to research"? https://www.tidepool.org/bigdata https://www.tidepool.org/bigdata
- morkalork 2y ago23andme tried that but well you know
- jprete 2y agoThat's an inherently biased sample, so not that useful for research.
- deleted 2y ago[deleted]
- Rychard 2y agoI fully expect the data to be de-anonymized and then used against people that have familial links to those who have donated their data. "Oh, your great great-great-grandfather had <disease>, so now you're classified as a high-risk individual and you have to pay a higher monthly fee for insurance" Only they won't tell you why you have to pay more, just that you do.
- darth_avocado 2y agoThat is essentially why HIPAA exists. And the negative and limiting effects of HIPAA on clinical trials is well acknowledged and is considered as an acceptable cost of HIPAA.
- 2y ago
- amiroo 2y agoIn the spirit of the authors' point of sharing one's supporting evidence, could you please share the data and code for scraping the list of devices from the FDA website? :)
- y-curious 2y agoHuge problem is data protections in the USA that do not allow for easy sharing of patient data for these purposes. The liability is huge and the upside is very little. Furthermore, if it's a college-affiliated hospital, that data is not going anywhere except to the internal teams within the college.
- iancmceachern 2y agoThese are good things. People don't want their info used for this without their permission. Freedom
- mannyv 2y agoYour data is valueless when it's by itself. That's the fundamental mistake they made with HIPAA.
- iancmceachern 2y agoThat's not true. Knowing if you have an expensive condition could be invaluable to say, an insurance company. Knowing something else may give people the opportunity for blackmail. There are lots of reasons one's medical history is valuable. There is more that is valuable than just money.
- funnygiraffe 2y ago"226 of 521 FDA-approved medical devices, or approximately 43%, lacked published clinical validation data." The lack of "published" clinical validation studies implies neither that the AI developer performed no clinical validation nor that the FDA hasn't seen it. So, it is not clear if the problem is with the lack of clinical validation or the lack of reporting. For some reason the title exaggerates yet further (half of FDA-approved AI not "trained" on real patient data).
- insane_dreamer 2y ago> AI developer performed no clinical validation nor that the FDA hasn't seen it if the developer went through the trouble and expense of performing clinical validation you can be sure that they would publish the results __unless__ the results reflected negatively on them
- meghnards 2y agoGiven that the design and endpoints of clinical validation studies is priceless information for developers of similar devices (i.e., competitors) applying for FDA clearance via the 510(k) pathway, it would not surprise me at all if this information was purposefully kept secret no matter how flattering it is. Especially for relatively new technology like AI
- iancmceachern 2y agoI've been part of bringing many medical devices to market and this isn't true. Sometimes they do publish the validation data, especially in pharma etc. Most times they don't, why would you? It would only help your competitors.
- seehafer 2y ago510(k) summaries are always a negotiation between FDA (which wants as much data as possible in them) and manufacturers (which want as little data as possible, as it is priceless competitive intelligence)
- blackeyeblitzar 2y agoHow can anyone except big incumbents who are immune to competition get access to data? I feel this is an unfortunate situation where the opportunity to be an innovator is locked away. Some of the reasons are good but it’s not a great outcome for us all.
- tomrod 2y agoThere is a healthy market for claims and ehr data resale. That can be part of the solution for out-of-sample outcomes comparisons.
- prashp 2y agoWhat's missing here is the risk of the medical device not performing as expected on real patients. These risks are usually mitigated reasonably by medical device manufacturers and designers such that it doesn't matter that "AI medical devices" are not trained on real patient data. The FDA keeps a database of all adverse events reported by manufacturers or healthcare systems [1], so we can check in a few years if these AI medical devices are causing an uptick of complaints. [1] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/search.cfm https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/s...
- can16358p 2y agoDon't want to go too off-topic but: Why does this has to be so hard? I mean what if the regulators went the opposite approach: make ALL the data publicly available by anonymizing it. What privacy concerns would anyone have if the whole data is completely anonymized? Wouldn't it create far more benefit by accelerating innovation in healthcare? I strongly believe the privacy concerns (which shouldn't exist after stripping out any PII) are outweighed by an order of magnitude by the upsides, which are literally saving millions of lives in the longer term. But again, regulation, as always, cripple innovation, this time slowing down developments that would literally save lives. Great.
- defrost 2y agoWithout getting into the weeds; privacy concerns remain after "stripping out any PII" simply due to the power of post analysis and combining masses of data looking for things that line up. There are various long articles on this and it remains the case that patients have a right to confidentiality.
- freetanga 2y agoElizabeth Holmes (in coarse voice): “Hold my beer”
- romesmoke 2y agoHad a chat with a pharmaceutical company whistleblower recently. They told me that FDA has nothing to do with actual product testing, e.g., human trials. What the "approval" guarantees is proper facilities and other second-order criteria. I'm writing this comment in the hopes of learning that I have misunderstood something crucial here. Because if it is indeed the case that the companies themselves are the only ones vouching for drug safety, that old story about capitalism and incentives implies we're seriously fucked.
- zomg 2y agoFDA clearance only means a company can legally market the device. additionally, product testing != human trials. they are two very different things. final product testing is called "design verification and validation" (aka dV+V) where a statistically significant number of devices are tested against several different categories. human trials can run the gauntlet from a small IDE studies to phase I through IV. the drug approval process is MUCH different than the medical device process.
- harish1977 2y agogod save us.. hope they didn't use omniverse replicator.
- uslic001 2y agoAs a gastroenterologist I have used Medtronic GI Genius AI to do colonoscopies for the past 8 months. It is mildly helpful but has way too many false positive alarms for "polyps" that are not polyps. It needs better training in the real world.
- zomg 2y agoanother confounding problem is how the FDA classifies and approves devices for clearance. most regulatory strategies have the company strive to find a predicate and prove they are substantially equivalent (SESE), which generally means a class II device, which does NOT require clinical data for clearance. it's frustrating that the article uses the term "approve" which is specific to PMA devices (class III), whereas class II devices are "cleared" by the FDA. - a medical device executive