4 ms·
I'm a biologist by training. Eventually my research hit a data wall (my simulations produced too much data for my storage and processing system). I had read a
by dekhn 10y ago
I'm a biologist by training. Eventually my research hit a data wall (my simulations produced too much data for my storage and processing system). I had read a paper on GFS and Mapreduce and Bigtable from Google, and decided to go work there. I got hired onto an SRE (production ops) team and spent my 20% time learning how production data processing works at scale.
After a few years I understood stuff better and moved my pipelines to MapReduce. And I built a bigger simulator (Exacycle). It was easy to process 100+T datasets in an hour or so. It wasn't a lot of work, really. We converted external data forms to protobufs and stored them in various container files. Then we ported the code that computed various parameters from the simulation to MapReduce.
I took this knowledge, looked at the market, and heard "storing genomic data is hard". After some research, I found that storing genomic data isn't hard at all. People spend all their time complaining about storage and performance, but when you look, they're using tin can telephones and wind up toy cars. This is because most scientists specialize in their science, not in data processing. So, based on this I built a product called 'Google Cloud Genomics' which stores petabytes of data (some public, some private for customers). Our customers love it- they do all their processing in Google Cloud, with fast access to petabytes of data. We've turned something that required them to hire expensive sysadmins and data scientists into something their regular scientists can just use (for example, from BigQuery or Python).
One of the things that really irked me about genomic data is that for several years people were predicting exponential growth of sequencing and similar rates of storage needs. They made ludicrous projections and complained that not enough hard drives were made to store their forthcoming volumes. oh, and the storage cost too much, too. Well, the reality is that genomic data doesn't ahve enough value to archive it for long times (sorry, folks, for those that believe it: your BAM files don't have value enough for you to pay the incredibly low rates storage providers charge! Also, we can just order more harddrives, Seagate just produces drive to meet demand, so if there is a real demand signal and money behind it, the drives will be made. Actual genomic data is tiny compared to cat videos.
The real issue is that most researchers don't have the tools or incentives to properly collect, store, and use big data. Until that is fixed, the field will continue in a crisis.
- ams6110 10y agoQuestion from ignorance: how do you get "petabytes of data" into the Google Cloud in a reasonable time? I find copying a mere few TB can take days and that's on a local network not over the internet.
- fnbr 10y agoI'd also be interested to hear this. I'm running a project that's 10gb in size and uploading the data to AWS S3 was absurdly slow. Is there any way to speed up the upload that you found? 10 GB was painful though, I can't imagine uploading terabytes.
- xapata 10y agoThe fastest way to upload is to ship hard drives in an airplane.
- ams6110 10y agoOK -- Tannenbaum's "station wagon full of tapes" updated for the 21st century.
- dekhn 10y agoTannenbaum always forgot to include the time writing and reading tapes. Typical 10TB hard drives (which most people use for data interchange instead of tapes) only have ~100MB/sec bandwidth (~ same as 1Gbit NIC).
- phillc73 10y agoI don't work in this specific field, but did previously, during the first decade of this century, in broadcast video distribution. At the time, UDP based tools such as Aspera[1], Signiant[2] and FileCatalyst[3] were all the rage for punting large amount of data over the public Internet. [1] http://asperasoft.com/ http://asperasoft.com/ [2] http://www.signiant.com/ http://www.signiant.com/ [3] http://filecatalyst.com/ http://filecatalyst.com/