3 ms·
Note that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale ab
by dekhn 10y ago
Note that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale above 128 nodes on typical supercomputers due to the communication requirements. I switched to running N independent simulations and pooled the stats, getting far more high quality results.
I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches.
Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
- seanmcdirmid 10y agoDoes anyone have any experience training a DNN on a supercomputer?
- dekhn 10y agoThat's a good question. There are a small number of people who have experience with it. The number is small because supercomputers don't really provide any speedup advantages, although this is mainly because the code developers of distributed DNNs have been using TCP/IP networks with standard RPC instead of fast interconnect. I think this will probably change: I think DNNs are going to run on supercomputer-like hardware, but not something that is a traditional supercomputer with MPI. There is one paper showing a port of TensorFlow to MPI (WHYYY?) and the results aren't very good.