4 ms·
petaflops really aren't about just summing up the flops of many independent nodes (at least within a supercomputer context, which this one is). It's about sync
by dekhn 8y ago
petaflops really aren't about just summing up the flops of many independent nodes (at least within a supercomputer context, which this one is).
It's about synchronized petaflops. DM and Google's internal networks aren't like the MPI-Infiniband networks- as published, they are significantly lower bandwidth and higher latency.
A lot has changed in the past few years as people are starting to figure out what are the appropriate models for large-scale synchronized flops-computing. In the TF world, that includes things like Collective AllReduce (now in TF< originally implemented in projects like Horovod).
When I worked in the supercomputer world, I couldn't get my code to do synchronized scaling beyond 128 nodes, so I couldn't really get time to run my codes there (so I left for Google, and ran exacycle, which provided more unsynchronized CPU flops than anybody for a while in the mid 2010's).