13 ms·

Big Data Coming in Faster Than Biomedical Researchers Can Process It

158 points by happy-go-lucky 10y ago

searine 10y ago
Yeah, that happens when you prioritize buying sequencers and building genome centers while pretending that analysts grow on trees. The post-docs and graduate students who do the heavy lifting on all of these projects don't make a living wage. They can't raise a family, buy a house, or save for the future. The people in charge made them indentured servants and now those leaders are going to reap the whirlwind.
- op00to 10y ago
  As a tech worker in research, I think my salary was at least 50% higher than many of the "staff scientists" and postdocs. This was a huge problem, as you can imagine. 80% of postdocs I ran into came from either China or Russia. Almost all of them were "disposable" from our point of view. Long hours, low pay, little reward. Despite reaching out to them to build friendships, it was incredibly rare for them to do anything but socialize in their small postdoc circles consisting of people from the same province in China or whatever. The best part is, from their point of view, they were here to take advantage of our "advanced" research infrastructure until they had enough experience to duplicate it back home. Fair trade, I think.
- Declanomous 10y ago
  Pretty much what you said. I majored in Biology and took most of the CS track at my school, with the intention of going into bioinformatics. I gave up on that plan, because the money was terrible. To make a reasonable amount of money in bioinformatics you need to have a PhD and be the person in charge of the grant. At that point, you'll still probably be making less money than anybody who is doing CRUD applications for any medium sized organization. Huge shame, because I love biology. I'd love to work on these projects, but I'm not going to get a PhD just so I can make what I'm making now.
- XiaomiFan 10y ago
  I was offered a coop position at the Singapore Genomics Institute this summer, but I turned it down because the pay was truly terrible. I would've been paid 1500 Sinagpore dollars a month IF I was able to successfully apply for the A-star scholarship, otherwise it would've been only 1000. I was seriously consider taking the offer, because I believe the experience would've been very rewarding, especially since I don't know a lick of biology, and was hired because of my quantatitve toolkit and modelling experiences, but declined when I found out they don't even pay for the plane ticket.
- toufka 10y ago
  Yep. Money is strangely spent. Incentives and historical attitudes of the field regarding money are hard to change. $500k for a new microscope, no problem! It's a fancy, four channel live microscope. So it takes four 1Mb images each frame, and you're running a 10min long experiment taking an image 5 times per second. So that's like 12Gb per imageset. And you take like 10/15 replicates per experiment under 3-5 different conditions. That data for that experiment which under-girds a 6-7 figure grant is now stored on a $100 3TB USB disk from Best Buy. Oh, and trying to process that 12Gb image over USB2.0 using MatLab on a student's personal macbook air is horribly inefficient - but there is no other option for the student really. The students collecting the data and storing it on their local HD or laptop hard drive have no place to archive their data even if they wanted to. There are no repositories capable of generically storing that kind of huge data that needs to be frequently accessed at the price the students/labs are willing to pay (nothing). And this speaks nothing about the code every student reinvents in MatLab to do basic scientific analysis. Or worse, does not reinvent and instead reuses 20-year old code written by some long-forgotten student who wanted to try their hand at a 'new' programming language like IDL. The 'students' and postdocs are paid nearly minimum wage to do the high-tech biomedical research. There are no computer-scientists to be seen because they would be fools to give up making 5-8x money across the street at Twitter. On the other hand, the scientists know their work really well, and it will take a truly integrated team to solve these issues. A computer scientist can't just come over for a day and write up an app to help out. The code will have to make scientific assumptions and must be custom for many/most projects. But it's very hard to build a capable team when the market salary for certain kinds of team members is multiples of another on the same team.
  - IndianAstronaut 10y ago
    I am surprised they don't use something like S3 to scale up the storage withouf buying hardware.
    - jerven 10y ago
      From the perspective of the student S3 over university wifi is not "better" than local USB2 hard drives. It actually is a risk budget wise.
zo7 10y ago
I've heard about this a lot from people I know who work in healthcare. It seems that one could make a successful business simply by hiring a bunch of data scientists to offer analytics and data processing services for healthcare, but what's preventing that? Is there a lack of expertise, funding, too much regulation, or something else?
- zbjornson 10y ago
  Unfortunately it's not a simple endeavor. (a) For healthcare data specifically, the data is sensitive. You need to follow HIPAA and usually 21 CFR 11 regulations, and you always face potentially high liability in case of breach. (b) In part for that reason, it becomes expensive. Even for non-healthcare biomedical data (research-only data), most academic labs will not or cannot pay outside firms to do the work, regardless of whether it would be done better or faster.
  - lj3 10y ago
    > Unfortunately it's not a simple endeavor. The HIPAA and other regulations aren't any more annoying than any other modern programming practices these days. As for liability in the case of a breach, that's what E&O insurance is for. It does raise the barrier of entry a little, but it's not by any means prohibitive. I think the bigger issue is what format this data is in. Most medical and medical records data is in a variety of proprietary, non-open and difficult to integrate technologies. One look at HL7 is usually enough to send a programmer back into the loving embrace of anything else. Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost.
    - op00to 10y ago
      > Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost. It's not worth the cost 99% of the time. The only reason we're still making real breakthroughs is because of the research institutes doing basic research on public funding. Even then, they're swinging for the low hanging fruit.
aisofteng 10y ago
Isn't Watson Health in exactly this space?
- searine 10y ago
  Waston can only categorize what is already known about genomes given existing research. What we need is to have people who can ask questions and think critically. Good, hypothesis driven science is how we discover entirely new concepts and mechanisms.
mjevans 10y ago
If that much data is being collected it's time to start asking what we're looking for within it, and if the rate of collection, retention, and range if inputs is worth it. Maybe it only makes sense to store sections which deviate in a significant way from a range of error (lossy compression). Maybe some of those inputs just don't make sense for the questions being asked. A concrete reasoning for why the data should be kept needs to be presented, and THAT is what should call for the funding to back that need.
- zitterbewegung 10y ago
  Also, the data can be a real toxic asset if it is ever leaked or hacked into. Even with HIPPA not everyone is compliant.
- dnautics 10y ago
  Genomic data is basically stored as "diffs" from the reference genome. For humans, that's "some guy from Buffalo", as a UCSC professor put it.
  - mjevans 10y ago
    My take was that this was mostly 'telemetry' data from all of the vital signs monitors hooked up to a patient.
yahyaheee 10y ago
Anyone know of open datasets one could play around with?
- acomjean 10y ago
  for genetic data ensemble.org has a lot. http://useast.ensembl.org/info/about/species.html http://useast.ensembl.org/info/about/species.html for example Humans: You end up here ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/dna/ there is a read me which helps figure out what this stuff is, or ask your friendly neighborhood biologist. ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/dna/README
  - jerven 10y ago
    To be honest, you should probably look at their RDF dumps if you are from a CS background or you will end up in Ensembls Perl/MySQL swamp.
- searine 10y ago
  All US publicly funded sequencing data is required to be posted on the NIH's Sequence Read Archive within a few months of it's generation. There are petabytes of open data there for you pick from every kind of organism. Go nuts. https://www.ncbi.nlm.nih.gov/sra https://www.ncbi.nlm.nih.gov/sra
- abetusk 10y ago
  The Harvard Personal Genome Project Over 200 whole genomes and I think over 500 genotyping data (23andMe and the like) released under a CC0 license with sporadic phenotype data [1]. Open Humans [2] has a bunch of data with a convenient API [3]. OpenSNP has a lot of genotyping data (23andMe etc.) available for download [4]. For a more comprehensive list check out one of the many "Awesome Public Datasets" [5] (biology section). [1] https://my.pgp-hms.org/public_genetic_data https://my.pgp-hms.org/public_genetic_data [2] https://www.openhumans.org/members/ https://www.openhumans.org/members/ [3] https://www.openhumans.org/public-data-api/ https://www.openhumans.org/public-data-api/ [4] https://opensnp.org/ https://opensnp.org/ [5] https://github.com/caesar0301/awesome-public-datasets#biology https://github.com/caesar0301/awesome-public-datasets#biolog...
danso 10y ago
It seems like there's good opportunity for skilled data scientists and engineers to make a real difference here. I do think that laypersons (to both medicine and engineering) think that practitioners in medicine and biology have mastered such mundane things like data pipelines, because you have to be so smart to be in medicine/biology, but my limited experience has been more along the lines of what Neil Saunders describes as the inspiration for his coding+bioinformatics blog: https://nsaunders.wordpress.com/about-2/about/ https://nsaunders.wordpress.com/about-2/about/ > You may be wondering about the title of this blog. > Early in my bioinformatics career, I gave a talk to my department. It was fairly basic stuff – how to identify genes in a genome sequence, standalone BLAST, annotation, data munging with Perl and so on. Come question time, a member of the audience raised her hand and said: > “It strikes me that what you’re doing is rather desperate. Wouldn’t you be better off doing some experiments?” > It was one of the few times in my life when my jaw literally dropped and swung uselessly on its hinges. Ultimately though, her question did make a great blog title. edit: To add an anecdote that I believe I read on HN; regarding the topic of the huge datamine of DNA and other health data provided by the U.S. government, a commenter said that the reason it was all on FTP was because professors couldn't download large datasets via their web browser, or some such technical hiccup. I won't say that putting data on the Web makes it automatically more accessible, but data discovery through FTP requires a bit of scripting skill that I imagine the average biomedical scientist does not have.
- brandonb 10y ago
  I agree--the impact of a great engineer working in healthcare is very high, particularly if you partner with medical experts. We're a small startup that has partnered with UCSF Cardiology to detect abnormal heart rhythms, and other conditions, using deep learning on Apple Watch heart rate data: https://wsj.com/articles/new-study-seeks-to-use-deep-learning-to-detect-heart-disease-1458240739 https://blog.cardiogr.am/three-challenges-for-artificial-intelligence-in-medicine-dfb9993ae750 https://a16z.com/2016/10/20/cardiogram/ We have about 10B sensor data points so far. If you're a machine learning engineer and interested in working on this type of problem, feel free to email me: brandon@cardiogr.am.
  - 10y ago
sien 10y ago
This isn't just in Biomedical fields. In Earth Sciences, Astronomy and presumably quite a few other fields there is a staggering amount of data coming in and the number of people who have the required domain knowledge, math knowledge and programming knowledge is not growing as fast as the data is. Teams can and do help, but well, there is lots of work out there.
- IndianAstronaut 10y ago
  The LSST is a telescope project that is a sky survey and is to generate 15tb of data a day. Staggering amount of data coming in. The data processing pipelines need to be very efficient and questions clear to tackle these. Not to mention the processing algorithms for this data.
mavsman 10y ago
This isn't very surprising to me. Data is relatively cheap and the (effective) analytics of it is where the value lies. This typically the case, isn't it? That said, this is a good reminder that data scientists are in high demand and can make a difference.
dekhn 10y ago
I'm a biologist by training. Eventually my research hit a data wall (my simulations produced too much data for my storage and processing system). I had read a paper on GFS and Mapreduce and Bigtable from Google, and decided to go work there. I got hired onto an SRE (production ops) team and spent my 20% time learning how production data processing works at scale. After a few years I understood stuff better and moved my pipelines to MapReduce. And I built a bigger simulator (Exacycle). It was easy to process 100+T datasets in an hour or so. It wasn't a lot of work, really. We converted external data forms to protobufs and stored them in various container files. Then we ported the code that computed various parameters from the simulation to MapReduce. I took this knowledge, looked at the market, and heard "storing genomic data is hard". After some research, I found that storing genomic data isn't hard at all. People spend all their time complaining about storage and performance, but when you look, they're using tin can telephones and wind up toy cars. This is because most scientists specialize in their science, not in data processing. So, based on this I built a product called 'Google Cloud Genomics' which stores petabytes of data (some public, some private for customers). Our customers love it- they do all their processing in Google Cloud, with fast access to petabytes of data. We've turned something that required them to hire expensive sysadmins and data scientists into something their regular scientists can just use (for example, from BigQuery or Python). One of the things that really irked me about genomic data is that for several years people were predicting exponential growth of sequencing and similar rates of storage needs. They made ludicrous projections and complained that not enough hard drives were made to store their forthcoming volumes. oh, and the storage cost too much, too. Well, the reality is that genomic data doesn't ahve enough value to archive it for long times (sorry, folks, for those that believe it: your BAM files don't have value enough for you to pay the incredibly low rates storage providers charge! Also, we can just order more harddrives, Seagate just produces drive to meet demand, so if there is a real demand signal and money behind it, the drives will be made. Actual genomic data is tiny compared to cat videos. The real issue is that most researchers don't have the tools or incentives to properly collect, store, and use big data. Until that is fixed, the field will continue in a crisis.
- ams6110 10y ago
  Question from ignorance: how do you get "petabytes of data" into the Google Cloud in a reasonable time? I find copying a mere few TB can take days and that's on a local network not over the internet.
Balgair 10y ago
One point: Most Bio type people have not had multi-variable calculus, and many have not had calculus at all. So, it's not really that they can't process it fast enough, it's that their techniques for processing it are stereotypical. They figured out how to do that T-Test (or something else) once, and they stick with it, because they really don't know he math behind it. Also, though there is a TON of 'data' coming in, most of it is not useful. For example, I have a 500Gb file of a stack of .tiff images per fish that I have imaged in a confocal microscope. I have a GFP filter on the scope and therefore only get the green part of the .tiff files exposed, the red and blue are just background noise. Also, most of the image is the dish I have the fishes in. I tickle the fish, they flick their tails, and I see this all in 120fps. Now, I measure how much of an angle the fish made their tails flick, all in 3-D, because that's what the scope records in. I have a half TB per fish to comb through, and I have ~20 fishes, say ~10TB. At the end, I get a single graph comparing the fish with some gene to those without it, and I have 10TB of 'data' left over. Yeah, someone could comb through it all and find something else to look at. But i forgot to record the precise temperatures, the orientation of the fish, the fish that I knew later died, etc. I had that all in my head. And, hey, what do you know?, the p-value is ~.45 and therefore there is no 'real' difference in the fish and we can't include this in a paper. Now all that 'data' is being kept on a drive on some computer somewhere and is counted towards the budget that the lab has on the shared spaces. It's not really 'data' anymore, in that it is useful to advancing knowledge for anyone (it counts as practice I guess), but it still clogs up space.
- spynxic 10y ago
  It seems that you may have been able to extract specific information about things like "how much of an angle the fish made their tails flick" as opposed to storing the raw files. Such a technique would've definitely eliminated extra irrelevant information like "the dish [you] have the fishes in" as well as cut back on the size of the data.
  - Balgair 10y ago
    True! That info is what the paper you are trying to write is all about. You take these TBs of data, and effectively compress it into a single graph of maybe 5kB, the thing you really care about. However, you still have to keep all the data around for possible re-analysis depending on the terms of the paper you submitted to, for up to 10 years or more in some cases, along with all the equipment and maybe some frozen fish samples in LN2 down in the basement. Now, not all labs are so well funded, and sometimes accidents happen and grad students are not always so informed about these rules, but you should be keeping it all ready for re-testing for some number of years. You simply cannot get rid of it all for the sake of science. If you really want to help out bio-peepz, then helping them program is a great way to do so. Spaghetti does not even come close to describing their 'code'. If anyone really wanted to reproduce the experiment, then they would have to wade into the fetid swamp of 'code' that produced it. Most bio people cannot even begin to tell you how to code or what it even means. They can PCR and Western Blot better than Jesus himself, but code? No way. It is a real hindrance to the sciences, actually. The helping hand that code was meant to be has become a ball-n-chain that drags peoples minds away from them and 'lets the computer do it' for them.
dcgoss 10y ago
If you're interested in working on an open source project involving big data, machine learning and cancer, check out cognoma.org. It is sponsored by UPenn's greenelab.com
nonbel 10y ago
To most here: Try to make sure you aren't just helping people publish papers for the sake of it. As soon as you sense that, stop working with them. It is a very, very bad thing.

158 points by happy-go-lucky 10y ago

Yeah, that happens when you prioritize buying sequencers and building genome centers while pretending that analysts grow on trees. The post-docs and graduate students who do the heavy lifting on all of these projects don't make a living wage. They can't raise a family, buy a house, or save for the future. The people in charge made them indentured servants and now those leaders are going to reap the whirlwind.

op00to 10y ago
As a tech worker in research, I think my salary was at least 50% higher than many of the "staff scientists" and postdocs. This was a huge problem, as you can imagine. 80% of postdocs I ran into came from either China or Russia. Almost all of them were "disposable" from our point of view. Long hours, low pay, little reward. Despite reaching out to them to build friendships, it was incredibly rare for them to do anything but socialize in their small postdoc circles consisting of people from the same province in China or whatever. The best part is, from their point of view, they were here to take advantage of our "advanced" research infrastructure until they had enough experience to duplicate it back home. Fair trade, I think.
Declanomous 10y ago
Pretty much what you said. I majored in Biology and took most of the CS track at my school, with the intention of going into bioinformatics. I gave up on that plan, because the money was terrible. To make a reasonable amount of money in bioinformatics you need to have a PhD and be the person in charge of the grant. At that point, you'll still probably be making less money than anybody who is doing CRUD applications for any medium sized organization. Huge shame, because I love biology. I'd love to work on these projects, but I'm not going to get a PhD just so I can make what I'm making now.
XiaomiFan 10y ago
I was offered a coop position at the Singapore Genomics Institute this summer, but I turned it down because the pay was truly terrible. I would've been paid 1500 Sinagpore dollars a month IF I was able to successfully apply for the A-star scholarship, otherwise it would've been only 1000. I was seriously consider taking the offer, because I believe the experience would've been very rewarding, especially since I don't know a lick of biology, and was hired because of my quantatitve toolkit and modelling experiences, but declined when I found out they don't even pay for the plane ticket.
toufka 10y ago
Yep. Money is strangely spent. Incentives and historical attitudes of the field regarding money are hard to change. $500k for a new microscope, no problem! It's a fancy, four channel live microscope. So it takes four 1Mb images each frame, and you're running a 10min long experiment taking an image 5 times per second. So that's like 12Gb per imageset. And you take like 10/15 replicates per experiment under 3-5 different conditions. That data for that experiment which under-girds a 6-7 figure grant is now stored on a $100 3TB USB disk from Best Buy. Oh, and trying to process that 12Gb image over USB2.0 using MatLab on a student's personal macbook air is horribly inefficient - but there is no other option for the student really. The students collecting the data and storing it on their local HD or laptop hard drive have no place to archive their data even if they wanted to. There are no repositories capable of generically storing that kind of huge data that needs to be frequently accessed at the price the students/labs are willing to pay (nothing). And this speaks nothing about the code every student reinvents in MatLab to do basic scientific analysis. Or worse, does not reinvent and instead reuses 20-year old code written by some long-forgotten student who wanted to try their hand at a 'new' programming language like IDL. The 'students' and postdocs are paid nearly minimum wage to do the high-tech biomedical research. There are no computer-scientists to be seen because they would be fools to give up making 5-8x money across the street at Twitter. On the other hand, the scientists know their work really well, and it will take a truly integrated team to solve these issues. A computer scientist can't just come over for a day and write up an app to help out. The code will have to make scientific assumptions and must be custom for many/most projects. But it's very hard to build a capable team when the market salary for certain kinds of team members is multiples of another on the same team.
- IndianAstronaut 10y ago
  I am surprised they don't use something like S3 to scale up the storage withouf buying hardware.
  - jerven 10y ago
    From the perspective of the student S3 over university wifi is not "better" than local USB2 hard drives. It actually is a risk budget wise.

zo7 10y ago

I've heard about this a lot from people I know who work in healthcare. It seems that one could make a successful business simply by hiring a bunch of data scientists to offer analytics and data processing services for healthcare, but what's preventing that? Is there a lack of expertise, funding, too much regulation, or something else?

zbjornson 10y ago
Unfortunately it's not a simple endeavor. (a) For healthcare data specifically, the data is sensitive. You need to follow HIPAA and usually 21 CFR 11 regulations, and you always face potentially high liability in case of breach. (b) In part for that reason, it becomes expensive. Even for non-healthcare biomedical data (research-only data), most academic labs will not or cannot pay outside firms to do the work, regardless of whether it would be done better or faster.
- lj3 10y ago
  > Unfortunately it's not a simple endeavor. The HIPAA and other regulations aren't any more annoying than any other modern programming practices these days. As for liability in the case of a breach, that's what E&O insurance is for. It does raise the barrier of entry a little, but it's not by any means prohibitive. I think the bigger issue is what format this data is in. Most medical and medical records data is in a variety of proprietary, non-open and difficult to integrate technologies. One look at HL7 is usually enough to send a programmer back into the loving embrace of anything else. Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost.
  - op00to 10y ago
    > Working with this data is time intensive and expensive and I'm guessing most heathcare companies don't see it as worth the cost. It's not worth the cost 99% of the time. The only reason we're still making real breakthroughs is because of the research institutes doing basic research on public funding. Even then, they're swinging for the low hanging fruit.

aisofteng 10y ago

Isn't Watson Health in exactly this space?

searine 10y ago
Waston can only categorize what is already known about genomes given existing research. What we need is to have people who can ask questions and think critically. Good, hypothesis driven science is how we discover entirely new concepts and mechanisms.

mjevans 10y ago

If that much data is being collected it's time to start asking what we're looking for within it, and if the rate of collection, retention, and range if inputs is worth it. Maybe it only makes sense to store sections which deviate in a significant way from a range of error (lossy compression). Maybe some of those inputs just don't make sense for the questions being asked. A concrete reasoning for why the data should be kept needs to be presented, and THAT is what should call for the funding to back that need.

zitterbewegung 10y ago
Also, the data can be a real toxic asset if it is ever leaked or hacked into. Even with HIPPA not everyone is compliant.
dnautics 10y ago
Genomic data is basically stored as "diffs" from the reference genome. For humans, that's "some guy from Buffalo", as a UCSC professor put it.
- mjevans 10y ago
  My take was that this was mostly 'telemetry' data from all of the vital signs monitors hooked up to a patient.

yahyaheee 10y ago

Anyone know of open datasets one could play around with?

acomjean 10y ago
for genetic data ensemble.org has a lot. http://useast.ensembl.org/info/about/species.html http://useast.ensembl.org/info/about/species.html for example Humans: You end up here ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/dna/ there is a read me which helps figure out what this stuff is, or ask your friendly neighborhood biologist. ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/dna/README
- jerven 10y ago
  To be honest, you should probably look at their RDF dumps if you are from a CS background or you will end up in Ensembls Perl/MySQL swamp.
searine 10y ago
All US publicly funded sequencing data is required to be posted on the NIH's Sequence Read Archive within a few months of it's generation. There are petabytes of open data there for you pick from every kind of organism. Go nuts. https://www.ncbi.nlm.nih.gov/sra https://www.ncbi.nlm.nih.gov/sra
abetusk 10y ago
The Harvard Personal Genome Project Over 200 whole genomes and I think over 500 genotyping data (23andMe and the like) released under a CC0 license with sporadic phenotype data [1]. Open Humans [2] has a bunch of data with a convenient API [3]. OpenSNP has a lot of genotyping data (23andMe etc.) available for download [4]. For a more comprehensive list check out one of the many "Awesome Public Datasets" [5] (biology section). [1] https://my.pgp-hms.org/public_genetic_data https://my.pgp-hms.org/public_genetic_data [2] https://www.openhumans.org/members/ https://www.openhumans.org/members/ [3] https://www.openhumans.org/public-data-api/ https://www.openhumans.org/public-data-api/ [4] https://opensnp.org/ https://opensnp.org/ [5] https://github.com/caesar0301/awesome-public-datasets#biology https://github.com/caesar0301/awesome-public-datasets#biolog...

danso 10y ago

It seems like there's good opportunity for skilled data scientists and engineers to make a real difference here. I do think that laypersons (to both medicine and engineering) think that practitioners in medicine and biology have mastered such mundane things like data pipelines, because you have to be so smart to be in medicine/biology, but my limited experience has been more along the lines of what Neil Saunders describes as the inspiration for his coding+bioinformatics blog: https://nsaunders.wordpress.com/about-2/about/ https://nsaunders.wordpress.com/about-2/about/ > You may be wondering about the title of this blog. > Early in my bioinformatics career, I gave a talk to my department. It was fairly basic stuff – how to identify genes in a genome sequence, standalone BLAST, annotation, data munging with Perl and so on. Come question time, a member of the audience raised her hand and said: > “It strikes me that what you’re doing is rather desperate. Wouldn’t you be better off doing some experiments?” > It was one of the few times in my life when my jaw literally dropped and swung uselessly on its hinges. Ultimately though, her question did make a great blog title. edit: To add an anecdote that I believe I read on HN; regarding the topic of the huge datamine of DNA and other health data provided by the U.S. government, a commenter said that the reason it was all on FTP was because professors couldn't download large datasets via their web browser, or some such technical hiccup. I won't say that putting data on the Web makes it automatically more accessible, but data discovery through FTP requires a bit of scripting skill that I imagine the average biomedical scientist does not have.

brandonb 10y ago
I agree--the impact of a great engineer working in healthcare is very high, particularly if you partner with medical experts. We're a small startup that has partnered with UCSF Cardiology to detect abnormal heart rhythms, and other conditions, using deep learning on Apple Watch heart rate data: https://wsj.com/articles/new-study-seeks-to-use-deep-learning-to-detect-heart-disease-1458240739 https://blog.cardiogr.am/three-challenges-for-artificial-intelligence-in-medicine-dfb9993ae750 https://a16z.com/2016/10/20/cardiogram/ We have about 10B sensor data points so far. If you're a machine learning engineer and interested in working on this type of problem, feel free to email me: brandon@cardiogr.am.
- 10y ago

sien 10y ago

This isn't just in Biomedical fields. In Earth Sciences, Astronomy and presumably quite a few other fields there is a staggering amount of data coming in and the number of people who have the required domain knowledge, math knowledge and programming knowledge is not growing as fast as the data is. Teams can and do help, but well, there is lots of work out there.

IndianAstronaut 10y ago
The LSST is a telescope project that is a sky survey and is to generate 15tb of data a day. Staggering amount of data coming in. The data processing pipelines need to be very efficient and questions clear to tackle these. Not to mention the processing algorithms for this data.

mavsman 10y ago

This isn't very surprising to me. Data is relatively cheap and the (effective) analytics of it is where the value lies. This typically the case, isn't it? That said, this is a good reminder that data scientists are in high demand and can make a difference.

dekhn 10y ago

I'm a biologist by training. Eventually my research hit a data wall (my simulations produced too much data for my storage and processing system). I had read a paper on GFS and Mapreduce and Bigtable from Google, and decided to go work there. I got hired onto an SRE (production ops) team and spent my 20% time learning how production data processing works at scale. After a few years I understood stuff better and moved my pipelines to MapReduce. And I built a bigger simulator (Exacycle). It was easy to process 100+T datasets in an hour or so. It wasn't a lot of work, really. We converted external data forms to protobufs and stored them in various container files. Then we ported the code that computed various parameters from the simulation to MapReduce. I took this knowledge, looked at the market, and heard "storing genomic data is hard". After some research, I found that storing genomic data isn't hard at all. People spend all their time complaining about storage and performance, but when you look, they're using tin can telephones and wind up toy cars. This is because most scientists specialize in their science, not in data processing. So, based on this I built a product called 'Google Cloud Genomics' which stores petabytes of data (some public, some private for customers). Our customers love it- they do all their processing in Google Cloud, with fast access to petabytes of data. We've turned something that required them to hire expensive sysadmins and data scientists into something their regular scientists can just use (for example, from BigQuery or Python). One of the things that really irked me about genomic data is that for several years people were predicting exponential growth of sequencing and similar rates of storage needs. They made ludicrous projections and complained that not enough hard drives were made to store their forthcoming volumes. oh, and the storage cost too much, too. Well, the reality is that genomic data doesn't ahve enough value to archive it for long times (sorry, folks, for those that believe it: your BAM files don't have value enough for you to pay the incredibly low rates storage providers charge! Also, we can just order more harddrives, Seagate just produces drive to meet demand, so if there is a real demand signal and money behind it, the drives will be made. Actual genomic data is tiny compared to cat videos. The real issue is that most researchers don't have the tools or incentives to properly collect, store, and use big data. Until that is fixed, the field will continue in a crisis.

ams6110 10y ago
Question from ignorance: how do you get "petabytes of data" into the Google Cloud in a reasonable time? I find copying a mere few TB can take days and that's on a local network not over the internet.

Balgair 10y ago

One point: Most Bio type people have not had multi-variable calculus, and many have not had calculus at all. So, it's not really that they can't process it fast enough, it's that their techniques for processing it are stereotypical. They figured out how to do that T-Test (or something else) once, and they stick with it, because they really don't know he math behind it. Also, though there is a TON of 'data' coming in, most of it is not useful. For example, I have a 500Gb file of a stack of .tiff images per fish that I have imaged in a confocal microscope. I have a GFP filter on the scope and therefore only get the green part of the .tiff files exposed, the red and blue are just background noise. Also, most of the image is the dish I have the fishes in. I tickle the fish, they flick their tails, and I see this all in 120fps. Now, I measure how much of an angle the fish made their tails flick, all in 3-D, because that's what the scope records in. I have a half TB per fish to comb through, and I have ~20 fishes, say ~10TB. At the end, I get a single graph comparing the fish with some gene to those without it, and I have 10TB of 'data' left over. Yeah, someone could comb through it all and find something else to look at. But i forgot to record the precise temperatures, the orientation of the fish, the fish that I knew later died, etc. I had that all in my head. And, hey, what do you know?, the p-value is ~.45 and therefore there is no 'real' difference in the fish and we can't include this in a paper. Now all that 'data' is being kept on a drive on some computer somewhere and is counted towards the budget that the lab has on the shared spaces. It's not really 'data' anymore, in that it is useful to advancing knowledge for anyone (it counts as practice I guess), but it still clogs up space.

spynxic 10y ago
It seems that you may have been able to extract specific information about things like "how much of an angle the fish made their tails flick" as opposed to storing the raw files. Such a technique would've definitely eliminated extra irrelevant information like "the dish [you] have the fishes in" as well as cut back on the size of the data.
- Balgair 10y ago
  True! That info is what the paper you are trying to write is all about. You take these TBs of data, and effectively compress it into a single graph of maybe 5kB, the thing you really care about. However, you still have to keep all the data around for possible re-analysis depending on the terms of the paper you submitted to, for up to 10 years or more in some cases, along with all the equipment and maybe some frozen fish samples in LN2 down in the basement. Now, not all labs are so well funded, and sometimes accidents happen and grad students are not always so informed about these rules, but you should be keeping it all ready for re-testing for some number of years. You simply cannot get rid of it all for the sake of science. If you really want to help out bio-peepz, then helping them program is a great way to do so. Spaghetti does not even come close to describing their 'code'. If anyone really wanted to reproduce the experiment, then they would have to wade into the fetid swamp of 'code' that produced it. Most bio people cannot even begin to tell you how to code or what it even means. They can PCR and Western Blot better than Jesus himself, but code? No way. It is a real hindrance to the sciences, actually. The helping hand that code was meant to be has become a ball-n-chain that drags peoples minds away from them and 'lets the computer do it' for them.

dcgoss 10y ago

If you're interested in working on an open source project involving big data, machine learning and cancer, check out cognoma.org. It is sponsored by UPenn's greenelab.com

nonbel 10y ago

To most here: Try to make sure you aren't just helping people publish papers for the sake of it. As soon as you sense that, stop working with them. It is a very, very bad thing.