3 ms·

I completely agree. I have always hated the phrase "big data". It is a meaningless marketing phrase. As the article points out, very few IT companies have singl

by rpedela 14y ago

I completely agree. I have always hated the phrase "big data". It is a meaningless marketing phrase. As the article points out, very few IT companies have single data sets which are in terabyte or larger scale. Although the total size of their data may be in terabytes or higher, the size of an analyzed data set is usually in megabytes or gigabytes. You know who actually does "big data"? Physicists. There are some telescopes that generate terabytes of data per second. There are other scientific fields which generate very large, single data sets such as genomics, proteomics, climate, etc.

twic 14y ago
Even genomics isn't really that big. The human genome is three billion base pairs; that's 750 MB if packed as tightly as can be. The largest genome (that of _Paris japonica_) is 150 billion base pairs, 37.5 GB. You might be storing multiples of that amount of data if you are doing a shotgun sequence assembly, or looking at polymorphisms, but then you're likely to have no more than 10 copies, and i can't imagine you'd have more than 100 copies. For the human genome, 100 copies would be 75 GB, which is still only about four times the size of Team Fortress 2.
- rpedela 14y ago
  A single genome, true. But what if you wanted to compare 100,000 genomes of people with heart disease to find common genes for heart disease? Now you are at ~72 TB which I would consider big data. As far as I am aware, enough human genomes have not been sequenced to do that. But it is coming (I hope).
  - twic 14y ago
    Well, you probably wouldn't do it that way, for starters. There is a mature toolkit of techniques for narrowing down the interesting bits of the genome, so you have less sequence to look at. But yes, the day will come when it's feasible to simply sequence and compare large numbers of genomes. But by the time sequencing technology has got that much better, so will computers have!
  - comrade_ogilvy 14y ago
    If you were doing blind correlations between all the raw data at once, it would be 72 TB. There are ~30k genes (encoded protein sequences). 100,000 human individuals may or may not have that many deltas per gene. The data is naturally structured. There is no reason to believe that looking at 72TB at once is useful. You would start by looking for correlations with particular genes. (Even correlations with random genes is merely 30,000 simple problems.) This kind of work can be done on a single multicore computer with a lot diskspace and memory, if you have the right tools.
deleted 14y ago
[deleted]

by rpedela 14y ago

twic 14y ago

Even genomics isn't really that big. The human genome is three billion base pairs; that's 750 MB if packed as tightly as can be. The largest genome (that of _Paris japonica_) is 150 billion base pairs, 37.5 GB. You might be storing multiples of that amount of data if you are doing a shotgun sequence assembly, or looking at polymorphisms, but then you're likely to have no more than 10 copies, and i can't imagine you'd have more than 100 copies. For the human genome, 100 copies would be 75 GB, which is still only about four times the size of Team Fortress 2.

rpedela 14y ago
A single genome, true. But what if you wanted to compare 100,000 genomes of people with heart disease to find common genes for heart disease? Now you are at ~72 TB which I would consider big data. As far as I am aware, enough human genomes have not been sequenced to do that. But it is coming (I hope).
- twic 14y ago
  Well, you probably wouldn't do it that way, for starters. There is a mature toolkit of techniques for narrowing down the interesting bits of the genome, so you have less sequence to look at. But yes, the day will come when it's feasible to simply sequence and compare large numbers of genomes. But by the time sequencing technology has got that much better, so will computers have!
- comrade_ogilvy 14y ago
  If you were doing blind correlations between all the raw data at once, it would be 72 TB. There are ~30k genes (encoded protein sequences). 100,000 human individuals may or may not have that many deltas per gene. The data is naturally structured. There is no reason to believe that looking at 72TB at once is useful. You would start by looking for correlations with particular genes. (Even correlations with random genes is merely 30,000 simple problems.) This kind of work can be done on a single multicore computer with a lot diskspace and memory, if you have the right tools.

deleted 14y ago

[deleted]