3 ms·

The "High Performance Conjugate Gradients" benchmark was proposed a couple years ago as an alternative metric for ranking supercomputers. Its proponents claim i

• jedbrown 11y ago
  HPCG basically measures STREAM and has many technical flaws making it scale-dependent and difficulty to adjudicate. As codeveloper of a different benchmark, I'll just cite this paper from a third party. https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceModelHPCG-2014.pdf https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceMo... The reality is that there are many dimensions to supercomputing performance and it's impossible for one number to capture the utility of the machine. Our HPGMG benchmark (https://hpgmg.org https://hpgmg.org) attempts to strike a balance and give useful supplementary information. I do think it's better than any other single benchmark for evaluating today's machines and will also prove to be more durable over time.
  • dekhn 11y ago
    How would you use a benchmark like this to predict the performance of a well-designed asynchronous parallel conjugate gradient solver, like most modern deep learning neural networks that run on Internet HPC machines?
    • jedbrown 11y ago
      CG isn't truly asynchronous due to its reductions. It can be pipelined in various ways (we have several implementations in PETSc), but performance requires a quality implementation of asynchronous reduction (e.g., MPI_Iallreduce) which the vendors have been slow about developing (I've been working with some on fixing this and Cray has made recent progress). With respect to deep learning and other applications using CG or related algorithms, the bottlenecks depend on the scale, and ability to expose locality, and operator/preconditioner representation. If there is no locality, then matrix-vector products require all-to-all communication which tend to dwarf the cost of the reductions in CG. Even with locality in the matrix-vector product, preconditioners often need to communicate globally in a scalable way similar to HPGMG. Operators need not be represented as a table of numbers or a sparse matrix format, but could use a tensor product, fast transform, or other information to compute the action using less storage. If they are represented explicitly (sparse or dense), then matrix-vector product performance (thus CG as a whole) is dominated by memory bandwidth for problem sizes that do not fit in cache. HPGMG tries to strike a balance between memory bandwidth demands and compute using a matrix-free representation. HPGMG also reports dynamic range expressed as Performance versus Time-to-solution as the problem size is varied, which allows applications to see performance barriers that might be relevant to them (e.g., see how Titan cannot do a solve in less than 200 ms while Edison can do 50 ms, and how that relates to climate simulation performance targets; see slide 7 of https://jedbrown.org/files/20150624-Versatility.pdf https://jedbrown.org/files/20150624-Versatility.pdf).
      • dekhn 11y ago
        You might want to read up on Sibyl.
        • jedbrown 11y ago
          The information I've seen about Sibyl (the Google ML system, not the genomics package (http://sybil.sourceforge.net/documentation.html) http://sybil.sourceforge.net/documentation.html)) says it is basically doing logistic regression using a parallel algorithm (Collins, Schapire, Singer) with a transpose on each iteration. Without knowing more about the problem sizes and data sparsity/irregularity, I expect the transpose to be a significant expense. I'd be happy to read more if you have access to further technical information, but it's not clear how this comment relates to your previous question about CG and deep learning. As it relates to HPGMG, I think my previous response covers the important performance dimensions. I'd be happy to discuss further over email.
          • dekhn 11y ago
            Both logistic regression and deep learning are basically just big conjugate gradient minimizers. What I meant by asynchronous is that not all terms in a gradient are required to be summed in the same step. The transpose step in Sibyl is implemented in the Shuffle and Reduce phases. The filesystem is used to hold the temporary data. Nevertheless even for large systems, very few steps are required, and step times are reasonable, even compared to modern supercomputers. This is a tribute primarily to the design of sibyl and the implementation of MapReduce at Google. This is all explained in online versions of the Sibyl presentation. I really wish more people from DOE who write modern solvers would pay attention to this stuff.
      • manjunaths 11y ago
        Is it possible to calculate the theoretical performance of a cluster under HPGMG and then do a practical run and come with an efficiency number like in HPL ? One of the biggest reasons for use of HPL is that many sizing considerations can be based off of the theoretical calculations. But anyway this is very interesting. I definitely need to check this out.
  • ajdecon 11y ago
    Interesting! I'll have to check this out, thanks!
• bhouston 11y ago
  It isn't that different, 7 of the top 10 of the top500 are in the top 10 of this HPCG benchmark. Most barely changed positions.