3 ms·

The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage

The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry. There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.

fnord123 10y ago
It depends on the problem you want to solve. Working hard on detail oriented work and not giving up until you're successful is always a transferrable experience. ut a lot of the data analysis systems at the moment seem to want an SQL interface. It's a good idea because it's a stable language that lots of people already know. But your work with MPI won't really help you directly. >There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases. It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it. Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.

3 ms·

The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage

by conjectures 10y ago

fnord123 10y ago
It depends on the problem you want to solve. Working hard on detail oriented work and not giving up until you're successful is always a transferrable experience. ut a lot of the data analysis systems at the moment seem to want an SQL interface. It's a good idea because it's a stable language that lots of people already know. But your work with MPI won't really help you directly. >There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases. It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it. Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.