9 ms·

FDA launches openFDA to provide easy access to valuable FDA public data

173 points by loopasam 12y ago

DrugCite 12y ago
Hey guys, I'm an owner of DrugCite.com, we've talked to the FDA a few times over the last few years while building our site (They contacted us at various points with questions). They let us know last August they would be releasing this site but I guess I never thought it would so close to ours. Examples: Our Drug Page: http://www.drugcite.com/?q=ABILIFY http://www.drugcite.com/?q=ABILIFY FDA Drug Page http://open.fda.gov/drug/event/ http://open.fda.gov/drug/event/ Looks like they're including some data we didn't previous have access to or know about. Anyways, make no mistake this is huge and will be incredibly useful for doctors and patients everywhere, this is some great data. This is the type of data that should be investigated before you take any medication, prescription or not. I can see some of this data becoming common label information shortly.
- droope 12y ago
  I am sorry I don't see any similarities between the two pages.
ef4 12y ago
If you're interested in this, you should also check out the UMLS published by the National Institutes of Health. https://www.nlm.nih.gov/research/umls/ https://www.nlm.nih.gov/research/umls/ It's a very big semantic database of health terminology. Among other things, it has a subset called RxNorm that contains all currently prescribe-able prescription drugs. I've been very impressed with it and I feel like not enough people have heard of it.
sogen 12y ago
Impressed, very very useful. Thanks!
seanherron 12y ago
Great to see this on HN! I'm one of the openFDA core team members and would love to help people who are interested in using the public drug adverse event API. It's good to note that we've also released all of the source code behind the platform (https://github.com/fda https://github.com/fda) and are actively interested in having members of the community help us make improvements. Please do ping me if you have any questions about the API or want to learn more! sean.herron@fda.hhs.gov Also, here's a direct link to the API documentation: https://open.fda.gov/drug/event https://open.fda.gov/drug/event
- loopasam 12y ago
  The post mentions that "The FDA will continually work to identify additional public datasets to make available through openFDA" - do you guys already have an idea on what datasets are coming next?
  - seanherron 12y ago
    We're going to focus on product recall and product labeling data next - expect to see some more releases throughout the summer.
    - palidanx 12y ago
      For product labeling, which data will be released for that?
      - seanherron 12y ago
        Structured Product Labels (http://www.fda.gov/forindustry/datastandards/structuredproductlabeling/default.htm http://www.fda.gov/forindustry/datastandards/structuredprodu...) that will get the same cleaning and harmonizing that the drug adverse events data received.
- Osiris 12y ago
  I'm sure a lot of people, myself included, feel that government projects would be better and cheaper if they were developed as open source rather the the typical proprietary solutions developed by contractors that we see today. What's your take on government and open source projects like this one?
gbinal 12y ago
This is awesome. Adding to the list of US Federal APIs - http://18f.github.io/API-All-the-X/pages/status http://18f.github.io/API-All-the-X/pages/status
danso 12y ago
It's really encouraging to see that there's someone in the U.S. gov't who not only cares about open source and the associated effects of transparency, but has some practical experience* in it. The openFDA website is built on Jekyll (https://github.com/FDA/open.fda.gov https://github.com/FDA/open.fda.gov) and its API is powered with Python and Node.js (https://github.com/FDA/openfda https://github.com/FDA/openfda)...It's not just the framework/current-tooling that is nice, but that such systems use open, readable formats (such as Markdown for the web pages). The current administration has always paid lip-service toward open-source...they won't satisfy people who think "open source" and "government" means hand over just about everything...but they're doing a good job making inroads on the parts of the U.S. data interfaces that were well-intended, but so obfuscated by poor design that it was a job in itself to parse/scrape their sites. (FDA has always had really exhaustive dumps of their data...strewn about their legacy site...the API isn't as interesting to me as the documentation for the API and the pipeline of data) * I don't want to just slag on Drupal...but Drupal was what Obama's head tech officer wanted in place, and to their credit, they did open-source parts of their custom Drupal modules...which were not particularly useful, because of the particulars of Drupal's module system and its quickly changing API...nevermind being only useful for other Drupal installations. But a lot of credit has to go to the U.S. gov't for pivoting off of Drupal to a mix of WordPress, Jekyll, and even node.js sites with less coupled components. It's been only about two or so years since Data.gov open-sourced its Drupal components before promptly switching to WordPress and CKAN modules...considering how a non-significant number of the fed sites are built on 12+ year-old code...the turnaround in the U.S. gov's stack is pretty amazing...(when it's not attempted on a service-critical site, such as healthcare.gov)
lucidrains 12y ago
Ok. This is actually pretty surprising and welcomed news!
RA_Fisher 12y ago
I have some history working with the Adverse Event Reporting data. The API is nice, but it just exposes what they already offer in flat files .... stale data. Does it seem reasonable to you that the FDA runs a year behind on this data? http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/ucm082193.htm http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInforma... They don't respond: https://twitter.com/statwonk/status/413355130461761536 https://twitter.com/statwonk/status/413355130461761536
Istof 12y ago
I hope the NSA will do the same and also open their datasets... maybe someone could catch someone who plans some "terroristic" actions before they happen...
skram 12y ago
My colleague and I have developed a search interface for web browsers at http://openfdasearch.herokuapp.com/ http://openfdasearch.herokuapp.com/ Feel free to tweet at us at @SocialHealthIs, @Geek_Nurse, and/or @Skram
bello 12y ago
Awesome initiative! As I was testing it, I noticed that the response consists of prettified JSON. I'm guessing that all that whitespace can be removed to save bandwidth?
- mmohebbi 12y ago
  Thanks! We support gzip on the api json response for clients that support it. Given that, I'd expect the size improvements would be minimal for whitespace stripping but let us know if you have evidence to the contrary!
  - bello 12y ago
    Good point! I always blindly assumed that removing whitespaces would lead to a decent size improvement, even after gzipping, so I ran a small test (gzip on linux, default parameters): http://i.imgur.com/U1O4Xg5.png http://i.imgur.com/U1O4Xg5.png The raw json contains the whitespaces, while they were removed in the minified json. So there is a 47% improvement for the uncompressed version, and a 21% improvement for the compressed version. What would be interesting to see is how the second (compressed) number scales with the filesize (I don't know enough about compression algorithms to guess that). EDIT: I really don't know how to format a table in plaintext...

173 points by loopasam 12y ago

Hey guys, I'm an owner of DrugCite.com, we've talked to the FDA a few times over the last few years while building our site (They contacted us at various points with questions). They let us know last August they would be releasing this site but I guess I never thought it would so close to ours. Examples: Our Drug Page: http://www.drugcite.com/?q=ABILIFY http://www.drugcite.com/?q=ABILIFY FDA Drug Page http://open.fda.gov/drug/event/ http://open.fda.gov/drug/event/ Looks like they're including some data we didn't previous have access to or know about. Anyways, make no mistake this is huge and will be incredibly useful for doctors and patients everywhere, this is some great data. This is the type of data that should be investigated before you take any medication, prescription or not. I can see some of this data becoming common label information shortly.

droope 12y ago
I am sorry I don't see any similarities between the two pages.

ef4 12y ago

If you're interested in this, you should also check out the UMLS published by the National Institutes of Health. https://www.nlm.nih.gov/research/umls/ https://www.nlm.nih.gov/research/umls/ It's a very big semantic database of health terminology. Among other things, it has a subset called RxNorm that contains all currently prescribe-able prescription drugs. I've been very impressed with it and I feel like not enough people have heard of it.

sogen 12y ago

Impressed, very very useful. Thanks!

seanherron 12y ago

Great to see this on HN! I'm one of the openFDA core team members and would love to help people who are interested in using the public drug adverse event API. It's good to note that we've also released all of the source code behind the platform (https://github.com/fda https://github.com/fda) and are actively interested in having members of the community help us make improvements. Please do ping me if you have any questions about the API or want to learn more! sean.herron@fda.hhs.gov Also, here's a direct link to the API documentation: https://open.fda.gov/drug/event https://open.fda.gov/drug/event

loopasam 12y ago
The post mentions that "The FDA will continually work to identify additional public datasets to make available through openFDA" - do you guys already have an idea on what datasets are coming next?
- seanherron 12y ago
  We're going to focus on product recall and product labeling data next - expect to see some more releases throughout the summer.
  - palidanx 12y ago
    For product labeling, which data will be released for that?
    - seanherron 12y ago
      Structured Product Labels (http://www.fda.gov/forindustry/datastandards/structuredproductlabeling/default.htm http://www.fda.gov/forindustry/datastandards/structuredprodu...) that will get the same cleaning and harmonizing that the drug adverse events data received.
Osiris 12y ago
I'm sure a lot of people, myself included, feel that government projects would be better and cheaper if they were developed as open source rather the the typical proprietary solutions developed by contractors that we see today. What's your take on government and open source projects like this one?

gbinal 12y ago

This is awesome. Adding to the list of US Federal APIs - http://18f.github.io/API-All-the-X/pages/status http://18f.github.io/API-All-the-X/pages/status

danso 12y ago

It's really encouraging to see that there's someone in the U.S. gov't who not only cares about open source and the associated effects of transparency, but has some practical experience* in it. The openFDA website is built on Jekyll (https://github.com/FDA/open.fda.gov https://github.com/FDA/open.fda.gov) and its API is powered with Python and Node.js (https://github.com/FDA/openfda https://github.com/FDA/openfda)...It's not just the framework/current-tooling that is nice, but that such systems use open, readable formats (such as Markdown for the web pages). The current administration has always paid lip-service toward open-source...they won't satisfy people who think "open source" and "government" means hand over just about everything...but they're doing a good job making inroads on the parts of the U.S. data interfaces that were well-intended, but so obfuscated by poor design that it was a job in itself to parse/scrape their sites. (FDA has always had really exhaustive dumps of their data...strewn about their legacy site...the API isn't as interesting to me as the documentation for the API and the pipeline of data) * I don't want to just slag on Drupal...but Drupal was what Obama's head tech officer wanted in place, and to their credit, they did open-source parts of their custom Drupal modules...which were not particularly useful, because of the particulars of Drupal's module system and its quickly changing API...nevermind being only useful for other Drupal installations. But a lot of credit has to go to the U.S. gov't for pivoting off of Drupal to a mix of WordPress, Jekyll, and even node.js sites with less coupled components. It's been only about two or so years since Data.gov open-sourced its Drupal components before promptly switching to WordPress and CKAN modules...considering how a non-significant number of the fed sites are built on 12+ year-old code...the turnaround in the U.S. gov's stack is pretty amazing...(when it's not attempted on a service-critical site, such as healthcare.gov)

lucidrains 12y ago

Ok. This is actually pretty surprising and welcomed news!

RA_Fisher 12y ago

I have some history working with the Adverse Event Reporting data. The API is nice, but it just exposes what they already offer in flat files .... stale data. Does it seem reasonable to you that the FDA runs a year behind on this data? http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/ucm082193.htm http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInforma... They don't respond: https://twitter.com/statwonk/status/413355130461761536 https://twitter.com/statwonk/status/413355130461761536

Istof 12y ago

I hope the NSA will do the same and also open their datasets... maybe someone could catch someone who plans some "terroristic" actions before they happen...

skram 12y ago

My colleague and I have developed a search interface for web browsers at http://openfdasearch.herokuapp.com/ http://openfdasearch.herokuapp.com/ Feel free to tweet at us at @SocialHealthIs, @Geek_Nurse, and/or @Skram

bello 12y ago

Awesome initiative! As I was testing it, I noticed that the response consists of prettified JSON. I'm guessing that all that whitespace can be removed to save bandwidth?

mmohebbi 12y ago
Thanks! We support gzip on the api json response for clients that support it. Given that, I'd expect the size improvements would be minimal for whitespace stripping but let us know if you have evidence to the contrary!
- bello 12y ago
  Good point! I always blindly assumed that removing whitespaces would lead to a decent size improvement, even after gzipping, so I ran a small test (gzip on linux, default parameters): http://i.imgur.com/U1O4Xg5.png http://i.imgur.com/U1O4Xg5.png The raw json contains the whitespaces, while they were removed in the minified json. So there is a 47% improvement for the uncompressed version, and a 21% improvement for the compressed version. What would be interesting to see is how the second (compressed) number scales with the filesize (I don't know enough about compression algorithms to guess that). EDIT: I really don't know how to format a table in plaintext...