3 ms·

I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/

by batbomb 13y ago

I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world. If processing latencies don't matter much, it's an easier more flexible system to use.

evanb 13y ago
It is built into QDP++, which is the low-level code underlying lattice QCD code bases like Chroma and MILC, though! Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.
virtuabhi 13y ago
Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc. I do hope physics people consider MR systems in addition to the existing ones they use.
- apw 13y ago
  MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC. It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.
  - seanmcdirmid 13y ago
    MPI can't schedule to optimize IO/data shuffling like MapReduce, this is the price for being more general. The biggest competition to MPI in the HPC space is actually CUDA and GPUs. In either case, the data pipeline is exposed to the compiler or run time for optimization, where as MPI everything is just messaging.
mileswu 13y ago
At least on the experimental LHC side, we process/analyse each event independently from every other event, so it's an embarrassingly parallel workload. All we do is split our input dataset up into N files, run N jobs, combine the N outputs. Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so many users (several thousand physicists on LHC experiments) that's why we have hundreds of GRID sites across the world. The scheduler sends your jobs to sites where the data is located. The output can then be transferred back via various academic/research internet networks. HEP also tends to invent many of its own 'large-scale computing' solutions. For example most sites tend to use Condor[1] as the batch system, dcache[2] as the distributed storage system, XRootD[3] as the file access protocol, GridFTP[4] as the file transfer protocol. I know there are some sites that use Lustre but it's pretty uncommon. [1] http://research.cs.wisc.edu/htcondor/ http://research.cs.wisc.edu/htcondor/ [2] http://www.dcache.org/ http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ http://xrootd.slac.stanford.edu/ [4] http://www.globus.org/toolkit/docs/latest-stable/gridftp/ http://www.globus.org/toolkit/docs/latest-stable/gridftp/
- memracom 13y ago
  I remember when HEP invented that WWW technology stuff which turned out to be rather popular outside of HEP as well.

by batbomb 13y ago

evanb 13y ago

It is built into QDP++, which is the low-level code underlying lattice QCD code bases like Chroma and MILC, though! Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.

virtuabhi 13y ago

Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc. I do hope physics people consider MR systems in addition to the existing ones they use.

apw 13y ago
MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC. It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.
- seanmcdirmid 13y ago
  MPI can't schedule to optimize IO/data shuffling like MapReduce, this is the price for being more general. The biggest competition to MPI in the HPC space is actually CUDA and GPUs. In either case, the data pipeline is exposed to the compiler or run time for optimization, where as MPI everything is just messaging.

mileswu 13y ago

At least on the experimental LHC side, we process/analyse each event independently from every other event, so it's an embarrassingly parallel workload. All we do is split our input dataset up into N files, run N jobs, combine the N outputs. Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so many users (several thousand physicists on LHC experiments) that's why we have hundreds of GRID sites across the world. The scheduler sends your jobs to sites where the data is located. The output can then be transferred back via various academic/research internet networks. HEP also tends to invent many of its own 'large-scale computing' solutions. For example most sites tend to use Condor[1] as the batch system, dcache[2] as the distributed storage system, XRootD[3] as the file access protocol, GridFTP[4] as the file transfer protocol. I know there are some sites that use Lustre but it's pretty uncommon. [1] http://research.cs.wisc.edu/htcondor/ http://research.cs.wisc.edu/htcondor/ [2] http://www.dcache.org/ http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/ http://xrootd.slac.stanford.edu/ [4] http://www.globus.org/toolkit/docs/latest-stable/gridftp/ http://www.globus.org/toolkit/docs/latest-stable/gridftp/

memracom 13y ago
I remember when HEP invented that WWW technology stuff which turned out to be rather popular outside of HEP as well.