4 ms·

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on

The less than a terabyte datasets being common had me awestruck. I, singular post-doctoral scientist noobermin[0], have processed terabytes of data at a time on HPC systems. Sure, a lot of it was garbage and I had to wade through it, but no one paid me millions to do it, I just did it to publish the papers. Sure, I needed the system which cost someone a lot of money, I suppose. But, I considered myself a small fry compared to some of the things others did on the system, particularly, hyrdrodynamics modellers. Moreover, I know I can probably process 100GB datasets on my own home PC, which isn't too impressive, it would just take longer (say a day or so instead of a hour or a few minutes). And this is with idk, python scripts using MPI. Yes, MPI because I'm a computational scientist and that's what HPC systems use, nothing fancy and likely the "legacy systems" he railed against in his pitches, but it worked. I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part. My mentors kept pushing me to mention the sheer size of the datasets I process in talks because it sounds impressive, and I do do so, but I always knew it didn't matter because the interpretation and analysis is the hard part, not just the "sheer size." [0] not going to use my real name

wongarsu 4y ago
> I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part Especially now that 1TB datasets fit in memory on off-the-shelf servers and 100GB fits in memory on consumer hardware. You need a lot of data to run into real technical challenges that can't be solved by throwing a couple hundreed dollars a month (amortized cost) at hardware. And often you can get by with much, much less than even that.
- hackernewds 4y ago
  Not to mention if you're physically storing it (not in the cloud) 1TB can fit on your pinky finger
krab 4y ago
That's a different category of big data. I worked for a big pharma and they were building their big data department with Spark and friends. I was quite surprised that their biggest dataset had something like 200 GB. At the same time, though, there was a lot of DNA sequencing data, we were designing CRISPR probes etc. But Spark and Hadoop aren't really that helpful in this area, so the Big Data team wasn't involved in those.
- noobermin 4y ago
  I think it does depend on the problem. Genetic stuff always seemed not easily parallelizable like my field (physics simulation) is. That said, the culture here is that MPI works and thus cray still builds computers that work better with it, so we use MPI so it works...etc etc.
  - krab 4y ago
    > Genetic stuff always seemed not easily parallelizable Not really. Processing one sample may take a few hours but if you have hundreds or more samples, it's an obvious axis for independent parallelization. The cool kids use Nextflow or CWL these days. It's something like `make` - it remembers what you've already computed and what the dependencies are - but it uses a batch engine like SGE/Condor/AWS Batch to actually execute the jobs.

wongarsu 4y ago

> I'm just awestruck, I could tell anyone that "large data" isn't really a bottleneck, but making sense of it is the very difficult part Especially now that 1TB datasets fit in memory on off-the-shelf servers and 100GB fits in memory on consumer hardware. You need a lot of data to run into real technical challenges that can't be solved by throwing a couple hundreed dollars a month (amortized cost) at hardware. And often you can get by with much, much less than even that.

hackernewds 4y ago
Not to mention if you're physically storing it (not in the cloud) 1TB can fit on your pinky finger

krab 4y ago

That's a different category of big data. I worked for a big pharma and they were building their big data department with Spark and friends. I was quite surprised that their biggest dataset had something like 200 GB. At the same time, though, there was a lot of DNA sequencing data, we were designing CRISPR probes etc. But Spark and Hadoop aren't really that helpful in this area, so the Big Data team wasn't involved in those.

noobermin 4y ago
I think it does depend on the problem. Genetic stuff always seemed not easily parallelizable like my field (physics simulation) is. That said, the culture here is that MPI works and thus cray still builds computers that work better with it, so we use MPI so it works...etc etc.
- krab 4y ago
  > Genetic stuff always seemed not easily parallelizable Not really. Processing one sample may take a few hours but if you have hundreds or more samples, it's an obvious axis for independent parallelization. The cool kids use Nextflow or CWL these days. It's something like `make` - it remembers what you've already computed and what the dependencies are - but it uses a batch engine like SGE/Condor/AWS Batch to actually execute the jobs.