4 ms·
It's ridiculous that a public-facing agency like the FDA does not make every piece of data available in a database that is easy to access and downloadable. You
by j_m_b 3y ago
It's ridiculous that a public-facing agency like the FDA does not make every piece of data available in a database that is easy to access and downloadable. You can download all of PubMed in compressed files, ClinicalTrials.gov gives you both API access as well as a publicly available read-only Postgres database, and the National Cancer Institute has excellent publicly available databases. There simply isn't an excuse for requiring someone to spend ~4 months to download this!
- wcedmisten 3y agoHi, author here! Completely agree, I would love to see the FDA publish an official dataset of predicate device relationships. I'm not sure if they have this data available internally, but if not, they could validate my dataset and republish it, which might be easier than starting from scratch.
- pfisherman 3y agoYou did not have to scrape the pdfs using the website link. They have bulk downloads indexed by year (IIRC). I remember grabbing these because I had to OCR a bunch of summaries to extract data for some NLP I was running. Also, IIRC grabbing the predicates was pretty easy w/ tesseract.
- wcedmisten 3y agoI'd love to take a look, do you have a link? I couldn't find any bulk downloads for this, but that would definitely be useful!
- j_m_b 3y agoI guess I spoke too soon. I found a resource here: https://open.fda.gov/apis/downloads/ https://open.fda.gov/apis/downloads/
- wcedmisten 3y agoI do use this resource for 510k.fyi to populate the device and recall data, but unfortunately it does not contain the 510(k) summaries, or even links to them. The web scraping was still required to get those documents so that I could run OCR on them. Since the website is open source, there's even a github issue confirming this: https://github.com/FDA/openfda/issues/200 https://github.com/FDA/openfda/issues/200
- jaybrendansmith 3y agoYes, it's all about how easy it is to access this data. Many of these older device submissions are scanned PDFs that need to be OCRd.
- iancmceachern 3y agoI wonder if the data could be requested via a public records request?