5 ms·

It seems to be more like 1.5GB rather than 100 GB, and someone else commented that compressed is around 800 MB. https://bitesizebio.com/8378/how-much-informati

It seems to be more like 1.5GB rather than 100 GB, and someone else commented that compressed is around 800 MB. https://bitesizebio.com/8378/how-much-information-is-stored-in-the-human-genome/ https://bitesizebio.com/8378/how-much-information-is-stored-... Even at 1.5 GB, this doesn't seem like an insurmountable problem at the current cost of storage. 750,000 GB = 750 TB, and you can buy an 8 TB hard drive for $150. So for $15,000 you can store 500,000 human genomes. If $15,000 is too expensive for a company, that company is doing something very wrong, when any salary will be many multiples of that.

evandijk70 8y ago
The raw sequencing data is 100 GB, and that's what needs to be stored, as the algorithms deducing the sequence are not perfect and under continuous development. Moreover, storing a genome in one location on a consumer hard drive is asking for trouble. In practice the genomes are stored backed up in multiple locations and in RAID systems on enterprise hard drives, so your estimate is of by about a factor of 1,000.
sofaofthedamned 8y ago
When working on a genome project I saw files up to 150gb, and as I understand it these are compressed csv files.its a lot of storage, especially if you unrl this and index it further for performance reasons. Also the more you compress, the longer the processing time. Some jobs took days, so there was a trade-off. I'm not a genome expert btw, just a devops nerd on a related project.
mmt 8y ago
> 750,000 GB = 750 TB, and you can buy an 8 TB hard drive for $150. So for $15,000 you can store 500,000 human genomes. As others have mentioned [1], the raw sequencing data (still) has to be stored, so 1.5GB is off by about 30x. Even at 6:1 compression, that's still 5x. More importantly, though, the cost of storage, especially at scale, is significantly higher than just the price of bare drives. Backblaze famously complained about this [2] in 2009. Even their particularly cheap, particularly low-performance solution claimed to add almost 45% on top of the cost of the bare drives. A higher-performancd storage system (using, say, standard SAS expander backplanes instead of their non-standard SATA multipliers) has, in my experience, added 70% on top of the bare drives (and that's being frugal, which seems remarkably rare, even among startups). Your $15k is now $128k, minimum. That's just purchase cost, and, while operating cost isn't huge for that little storage, it's not zero. The cost gets much, much worse, if the orgnization can't/won't hire someone like me or won't allow that someone to implement such a frugal DIY assembly (which is the vast majority of organizations today). That leaves "enterprise" storage or cloud, which are both approximately as expensive. I recently heard an estimate of half a million dollars per petabyte for NetApp, which would translate to $1,875,000 here. [1] Well detailed by https://news.ycombinator.com/item?id=17821523 https://news.ycombinator.com/item?id=17821523 [2] famous on HN, anyway. https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...
- wenc 8y ago
  Cloud: just did a quick calculation for AWS. 75 PB (= 150GB * 500,000) of storage costs $300,000 on AWS Glacier, and $1.57mil on S3 Standard. And that's before accounting for any read/write costs. Wow.
  - mmt 8y ago
    I was using 3.75PB (assuming 45GB compressed 6:1 so 7.5GB per person). I'm not certain, but my intuition is that, even paying AWS for the CPU time for compression, that would be cheaper than paying for the uncompressed transfer charges. But, yes, even for my lower storage numbers, for "EU (London)", even for just 3 years (a conservative enough lifespan for an HDD), S3-Standard would be almost $2.97M, S3-Standard-IA would be $1.77M, S3-OneZone-IA $1.4M, and Glacier would be merely $608k. Of course, that excludes transfer charges.
tofof 8y ago
> If $15,000 is too expensive for a company, that company is doing something very wrong, when any salary will be many multiples of that. You're very correct that salary is often the single largest expense - true even in academia. This is consistently reflected in grants and other funding awards. However, a hardware expense of $15,000 would indeed be out of reach of many biology labs. How can these both be true? Simply: graduate students are responsible for the vast majority of the lab's research output. Graduate student salaries in biology are very far from many multiples of $15,000. In fact, the median salary is certainly less than the lowest multiple of that. The NRSA, (a coveted, extremely competetive grant well-beyond the reach of the majority of graduate students) pays $24,324 per year. [0] For another example, the University of Florida publishes their stipends [1] - but if you are unfamiliar with the (always) distorted presentation of the numbers, you'll have to take my word on the specifics. The standard appointment will be at .50 FTE for 11 months, a combination you won't find listed. Such a researcher's compensation for the preceding 12 months would total $20,078x11/12 = $18,404, and that's before almost certainly some $500 of mandatory fees and other unsubsidized non-tuition expenses that all graduate students get to pay tax on but immediately hand back to their employer (university). It's not much better in academia even after becoming a doctorate. Salaries in biology for post-doctoral (ie, already possessing PhD) researchers are about $36,000. [2] And that's after a delay averaging 7 years [3] earning 1/2 to 2/3 that figure at best. Those numbers are perfectly in line with graduate student compensation here at the University of Illinois, for what that's additionally worth. [0] https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-175.html https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-1... [1] https://hr.ufl.edu/manager-resources/recruitment-staffing/hiring-center/preparing-an-offer/requirements-for-an-appointment/#salaries https://hr.ufl.edu/manager-resources/recruitment-staffing/hi... [2] http://www.sciencemag.org/careers/2015/12/new-phd-incomes-surprisingly-low http://www.sciencemag.org/careers/2015/12/new-phd-incomes-su... [3] http://www.slate.com/articles/health_and_science/science/2012/08/what_is_the_value_of_a_science_phd_is_graduate_school_worth_the_effort_.html http://www.slate.com/articles/health_and_science/science/201...

evandijk70 8y ago

The raw sequencing data is 100 GB, and that's what needs to be stored, as the algorithms deducing the sequence are not perfect and under continuous development. Moreover, storing a genome in one location on a consumer hard drive is asking for trouble. In practice the genomes are stored backed up in multiple locations and in RAID systems on enterprise hard drives, so your estimate is of by about a factor of 1,000.

sofaofthedamned 8y ago

When working on a genome project I saw files up to 150gb, and as I understand it these are compressed csv files.its a lot of storage, especially if you unrl this and index it further for performance reasons. Also the more you compress, the longer the processing time. Some jobs took days, so there was a trade-off. I'm not a genome expert btw, just a devops nerd on a related project.

mmt 8y ago

> 750,000 GB = 750 TB, and you can buy an 8 TB hard drive for $150. So for $15,000 you can store 500,000 human genomes. As others have mentioned [1], the raw sequencing data (still) has to be stored, so 1.5GB is off by about 30x. Even at 6:1 compression, that's still 5x. More importantly, though, the cost of storage, especially at scale, is significantly higher than just the price of bare drives. Backblaze famously complained about this [2] in 2009. Even their particularly cheap, particularly low-performance solution claimed to add almost 45% on top of the cost of the bare drives. A higher-performancd storage system (using, say, standard SAS expander backplanes instead of their non-standard SATA multipliers) has, in my experience, added 70% on top of the bare drives (and that's being frugal, which seems remarkably rare, even among startups). Your $15k is now $128k, minimum. That's just purchase cost, and, while operating cost isn't huge for that little storage, it's not zero. The cost gets much, much worse, if the orgnization can't/won't hire someone like me or won't allow that someone to implement such a frugal DIY assembly (which is the vast majority of organizations today). That leaves "enterprise" storage or cloud, which are both approximately as expensive. I recently heard an estimate of half a million dollars per petabyte for NetApp, which would translate to $1,875,000 here. [1] Well detailed by https://news.ycombinator.com/item?id=17821523 https://news.ycombinator.com/item?id=17821523 [2] famous on HN, anyway. https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...

wenc 8y ago
Cloud: just did a quick calculation for AWS. 75 PB (= 150GB * 500,000) of storage costs $300,000 on AWS Glacier, and $1.57mil on S3 Standard. And that's before accounting for any read/write costs. Wow.
- mmt 8y ago
  I was using 3.75PB (assuming 45GB compressed 6:1 so 7.5GB per person). I'm not certain, but my intuition is that, even paying AWS for the CPU time for compression, that would be cheaper than paying for the uncompressed transfer charges. But, yes, even for my lower storage numbers, for "EU (London)", even for just 3 years (a conservative enough lifespan for an HDD), S3-Standard would be almost $2.97M, S3-Standard-IA would be $1.77M, S3-OneZone-IA $1.4M, and Glacier would be merely $608k. Of course, that excludes transfer charges.

tofof 8y ago

> If $15,000 is too expensive for a company, that company is doing something very wrong, when any salary will be many multiples of that. You're very correct that salary is often the single largest expense - true even in academia. This is consistently reflected in grants and other funding awards. However, a hardware expense of $15,000 would indeed be out of reach of many biology labs. How can these both be true? Simply: graduate students are responsible for the vast majority of the lab's research output. Graduate student salaries in biology are very far from many multiples of $15,000. In fact, the median salary is certainly less than the lowest multiple of that. The NRSA, (a coveted, extremely competetive grant well-beyond the reach of the majority of graduate students) pays $24,324 per year. [0] For another example, the University of Florida publishes their stipends [1] - but if you are unfamiliar with the (always) distorted presentation of the numbers, you'll have to take my word on the specifics. The standard appointment will be at .50 FTE for 11 months, a combination you won't find listed. Such a researcher's compensation for the preceding 12 months would total $20,078x11/12 = $18,404, and that's before almost certainly some $500 of mandatory fees and other unsubsidized non-tuition expenses that all graduate students get to pay tax on but immediately hand back to their employer (university). It's not much better in academia even after becoming a doctorate. Salaries in biology for post-doctoral (ie, already possessing PhD) researchers are about $36,000. [2] And that's after a delay averaging 7 years [3] earning 1/2 to 2/3 that figure at best. Those numbers are perfectly in line with graduate student compensation here at the University of Illinois, for what that's additionally worth. [0] https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-175.html https://grants.nih.gov/grants/guide/notice-files/NOT-OD-18-1... [1] https://hr.ufl.edu/manager-resources/recruitment-staffing/hiring-center/preparing-an-offer/requirements-for-an-appointment/#salaries https://hr.ufl.edu/manager-resources/recruitment-staffing/hi... [2] http://www.sciencemag.org/careers/2015/12/new-phd-incomes-surprisingly-low http://www.sciencemag.org/careers/2015/12/new-phd-incomes-su... [3] http://www.slate.com/articles/health_and_science/science/2012/08/what_is_the_value_of_a_science_phd_is_graduate_school_worth_the_effort_.html http://www.slate.com/articles/health_and_science/science/201...