4 ms·

Unfortunately it's not a simple endeavor. (a) For healthcare data specifically, the data is sensitive. You need to follow HIPAA and usually 21 CFR 11 regulation

Unfortunately it's not a simple endeavor. (a) For healthcare data specifically, the data is sensitive. You need to follow HIPAA and usually 21 CFR 11 regulations, and you always face potentially high liability in case of breach. (b) In part for that reason, it becomes expensive. Even for non-healthcare biomedical data (research-only data), most academic labs will not or cannot pay outside firms to do the work, regardless of whether it would be done better or faster.

lj3 10y ago
> Unfortunately it's not a simple endeavor. The HIPAA and other regulations aren't any more annoying than any other modern programming practices these days. As for liability in the case of a breach, that's what E&O insurance is for. It does raise the barrier of entry a little, but it's not by any means prohibitive. I think the bigger issue is what format this data is in. Most medical and medical records data is in a variety of proprietary, non-open and difficult to integrate technologies. One look at HL7 is usually enough to send a programmer back into the loving embrace of anything else. Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost.
- op00to 10y ago
  > Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost. It's not worth the cost 99% of the time. The only reason we're still making real breakthroughs is because of the research institutes doing basic research on public funding. Even then, they're swinging for the low hanging fruit.
  - lj3 10y ago
    I read your other comment too. That whole situation breaks my brain. How can you do meaningful long term research if you're not aggregating data? > I couldn't have gotten out of there fast enough for the way saner land of tech companies. You know you have a problem when tech companies seem saner in comparison.
    - op00to 10y ago
      Researchers are aggregating data. There's all kinds of ways you as a layperson can mash up genomic data, and it's easier than ever. The data that's most accessible are well-defined genomic sequences, not cutting edge stuff. Much of what 23 and Me is doing is stems off of these publicly funded datasets. In many cases scientists are required to submit data to journals when they publish. However, data is a funny term! Using genomics data, there are many levels of data that you'll encounter - everything from raw image files (many gene sequencers are actually automated digital cameras, taking pictures of florescent markers attached to the DNA strands) to intermediate sequence files to formatted pieces of selected data. Then, there's the whole toolchain used to go from sample to formatted final analysis? What's required to be submitted? How long should it be kept? Who checks all this to make sure it's not NES roms or Shakespeare instead of the right data? Finally, there's the big questio: how can we be sure that the data captured in the intermediate or final steps of analysis actually originates from the raw data? Should scientists store the raw data (TBs and TB), intermediates (GBs and GBs), or final analysis (MBs). For how long? To answer your question about meaningful long term research - I've personally seen grad students' careers effectively ruined due to shitty data storage hygiene.
      - hash-set 10y ago
        We're wasting too much $$ on stupid shit like Uber.
    - deleted 10y ago
      [deleted]
- zbjornson 10y ago
  > regulations aren't any more annoying than any other modern programming practices these days As someone in the field I beg to differ :). And regardless of who pays in the event of a breach, the effects may be sufficient enough to shut down the company for future projects.
  - lj3 10y ago
    Can you expand on that? Everyone I've spoken with about HIPAA say that most companies don't even bother to comply and that nobody is enforcing it. Then again, these people work for companies that are small enough they've never had a breach.
    - zbjornson 10y ago
      That's rather scary to hear, and I can't imagine that they manage to secure access to any of the major datasets e.g. as contractors for hospitals or insurance companies. You can basically self-certify, but most serious companies will bring in an outside contractor on an ongoing basis to certify compliance. Staff needs to be trained, computers need to be managed, software changes have to be very thoroughly reviewed, updates become slow. It makes it pretty unattractive to enter into for a lot of devs.

lj3 10y ago

> Unfortunately it's not a simple endeavor. The HIPAA and other regulations aren't any more annoying than any other modern programming practices these days. As for liability in the case of a breach, that's what E&O insurance is for. It does raise the barrier of entry a little, but it's not by any means prohibitive. I think the bigger issue is what format this data is in. Most medical and medical records data is in a variety of proprietary, non-open and difficult to integrate technologies. One look at HL7 is usually enough to send a programmer back into the loving embrace of anything else. Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost.

op00to 10y ago
> Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost. It's not worth the cost 99% of the time. The only reason we're still making real breakthroughs is because of the research institutes doing basic research on public funding. Even then, they're swinging for the low hanging fruit.
- lj3 10y ago
  I read your other comment too. That whole situation breaks my brain. How can you do meaningful long term research if you're not aggregating data? > I couldn't have gotten out of there fast enough for the way saner land of tech companies. You know you have a problem when tech companies seem saner in comparison.
  - op00to 10y ago
    Researchers are aggregating data. There's all kinds of ways you as a layperson can mash up genomic data, and it's easier than ever. The data that's most accessible are well-defined genomic sequences, not cutting edge stuff. Much of what 23 and Me is doing is stems off of these publicly funded datasets. In many cases scientists are required to submit data to journals when they publish. However, data is a funny term! Using genomics data, there are many levels of data that you'll encounter - everything from raw image files (many gene sequencers are actually automated digital cameras, taking pictures of florescent markers attached to the DNA strands) to intermediate sequence files to formatted pieces of selected data. Then, there's the whole toolchain used to go from sample to formatted final analysis? What's required to be submitted? How long should it be kept? Who checks all this to make sure it's not NES roms or Shakespeare instead of the right data? Finally, there's the big questio: how can we be sure that the data captured in the intermediate or final steps of analysis actually originates from the raw data? Should scientists store the raw data (TBs and TB), intermediates (GBs and GBs), or final analysis (MBs). For how long? To answer your question about meaningful long term research - I've personally seen grad students' careers effectively ruined due to shitty data storage hygiene.
    - hash-set 10y ago
      We're wasting too much $$ on stupid shit like Uber.
  - deleted 10y ago
    [deleted]
zbjornson 10y ago
> regulations aren't any more annoying than any other modern programming practices these days As someone in the field I beg to differ :). And regardless of who pays in the event of a breach, the effects may be sufficient enough to shut down the company for future projects.
- lj3 10y ago
  Can you expand on that? Everyone I've spoken with about HIPAA say that most companies don't even bother to comply and that nobody is enforcing it. Then again, these people work for companies that are small enough they've never had a breach.
  - zbjornson 10y ago
    That's rather scary to hear, and I can't imagine that they manage to secure access to any of the major datasets e.g. as contractors for hospitals or insurance companies. You can basically self-certify, but most serious companies will bring in an outside contractor on an ongoing basis to certify compliance. Staff needs to be trained, computers need to be managed, software changes have to be very thoroughly reviewed, updates become slow. It makes it pretty unattractive to enter into for a lot of devs.