4 ms·
Different kinds of applications. Think of it like public transportation versus owning a car. For highly parallelizable and distributable applications (anything
by CmdrSprinkles 10y ago
Different kinds of applications. Think of it like public transportation versus owning a car.
For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle.
Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving.
And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations.
Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
- dekhn 10y agoNote that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale above 128 nodes on typical supercomputers due to the communication requirements. I switched to running N independent simulations and pooled the stats, getting far more high quality results. I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches. Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
- seanmcdirmid 10y agoDoes anyone have any experience training a DNN on a supercomputer?
- dekhn 10y agoThat's a good question. There are a small number of people who have experience with it. The number is small because supercomputers don't really provide any speedup advantages, although this is mainly because the code developers of distributed DNNs have been using TCP/IP networks with standard RPC instead of fast interconnect. I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI. There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.