5 ms·

This isn't only about the FLOPS, the big trend of these countries* ordering new supercomputers by 2020/2025 is very focused on power. Current supercomputers con

by vmarsy 11y ago

This isn't only about the FLOPS, the big trend of these countries* ordering new supercomputers by 2020/2025 is very focused on power. Current supercomputers consume a lot. Also the FLOPS measurement is a bit broken: It focuses on dense linear algebra problem, for which GPU or other accelerators boost the results easily. If all you plan to do is running simulations that are easily parallelized on GPU it is fine, for other types of programs it is hard to tell which is the fastest supercomputer. * France is also ordering a would -be top 10 supercomputer : http://www.hpcwire.com/off-the-wire/the-cea-agency-and-atos-team-to-deliver-exaflop-supercomputer-by-2020/ http://www.hpcwire.com/off-the-wire/the-cea-agency-and-atos-...

amelius 11y ago
> Also the FLOPS measurement is a bit broken Does anybody have a better measurement? Perhaps it could be the size of the matrix that can be inverted on it in an hour of time, with IEEE double precision floats, using some standard algorithm.
- vmarsy 11y ago
  It depends, I meant broken if you plan to compute other types of problems. For instance some big graph problem instead of involving linear algebra. Then you would favor the benchmarks from http://www.graph500.org/ http://www.graph500.org/. But what if you want to optimize for programs that are communication intensive, or memory intensive? Should the FLOPS of a very specific linear algebra suite be used as the metric of best computers?
- ajdecon 11y ago
  The "High Performance Conjugate Gradients" benchmark was proposed a couple years ago as an alternative metric for ranking supercomputers. Its proponents claim its behavior is more similar to real applications (irregular access patterns, lower ratio of computation to memory access, etc), compared to linear algebra problems like the "High Performance Linpack" benchmark currently used by the Top500. The different performance numbers for top systems on HPCG vs HPL are pretty striking: http://www.hpcg-benchmark.org/custom/index.html?lid=155&slid=279 http://www.hpcg-benchmark.org/custom/index.html?lid=155&slid... Original proposal to use HPCG as an alternative to HPL for supercomputer rankings: http://www.sandia.gov/~maherou/docs/HPCG-Benchmark.pdf http://www.sandia.gov/~maherou/docs/HPCG-Benchmark.pdf
  - jedbrown 11y ago
    HPCG basically measures STREAM and has many technical flaws making it scale-dependent and difficulty to adjudicate. As codeveloper of a different benchmark, I'll just cite this paper from a third party. https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceModelHPCG-2014.pdf https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceMo... The reality is that there are many dimensions to supercomputing performance and it's impossible for one number to capture the utility of the machine. Our HPGMG benchmark (https://hpgmg.org https://hpgmg.org) attempts to strike a balance and give useful supplementary information. I do think it's better than any other single benchmark for evaluating today's machines and will also prove to be more durable over time.
    - dekhn 11y ago
      How would you use a benchmark like this to predict the performance of a well-designed asynchronous parallel conjugate gradient solver, like most modern deep learning neural networks that run on Internet HPC machines?
      - jedbrown 11y ago
        CG isn't truly asynchronous due to its reductions. It can be pipelined in various ways (we have several implementations in PETSc), but performance requires a quality implementation of asynchronous reduction (e.g., MPI_Iallreduce) which the vendors have been slow about developing (I've been working with some on fixing this and Cray has made recent progress). With respect to deep learning and other applications using CG or related algorithms, the bottlenecks depend on the scale, and ability to expose locality, and operator/preconditioner representation. If there is no locality, then matrix-vector products require all-to-all communication which tend to dwarf the cost of the reductions in CG. Even with locality in the matrix-vector product, preconditioners often need to communicate globally in a scalable way similar to HPGMG. Operators need not be represented as a table of numbers or a sparse matrix format, but could use a tensor product, fast transform, or other information to compute the action using less storage. If they are represented explicitly (sparse or dense), then matrix-vector product performance (thus CG as a whole) is dominated by memory bandwidth for problem sizes that do not fit in cache. HPGMG tries to strike a balance between memory bandwidth demands and compute using a matrix-free representation. HPGMG also reports dynamic range expressed as Performance versus Time-to-solution as the problem size is varied, which allows applications to see performance barriers that might be relevant to them (e.g., see how Titan cannot do a solve in less than 200 ms while Edison can do 50 ms, and how that relates to climate simulation performance targets; see slide 7 of https://jedbrown.org/files/20150624-Versatility.pdf https://jedbrown.org/files/20150624-Versatility.pdf).
jedbrown 11y ago
Power is over-emphasized in HPC circles. According to FOIA reports (and you can back out similar from public budget information), less than 10% of the budget is going to energy. It is frequently used as an excuse to build machines that are inappropriate for the science (not just "hard to program", but actually inappropriate in the sense that even with infinite programming effort, they deliver less scientific value than a more conventional architecture). There is some value in making scientists uncomfortable so that they think of creative algorithmic solutions that may pay off as the inevitabilities of semiconductor physics become more apparent, but seeing the number of applications that are within small constant factors of proven barriers and the willingness to compromise quality of solution and/or run scientifically irrelevant configurations to demonstrate "speedup", I think it has gone too far.

by vmarsy 11y ago

amelius 11y ago

> Also the FLOPS measurement is a bit broken Does anybody have a better measurement? Perhaps it could be the size of the matrix that can be inverted on it in an hour of time, with IEEE double precision floats, using some standard algorithm.

vmarsy 11y ago
It depends, I meant broken if you plan to compute other types of problems. For instance some big graph problem instead of involving linear algebra. Then you would favor the benchmarks from http://www.graph500.org/ http://www.graph500.org/. But what if you want to optimize for programs that are communication intensive, or memory intensive? Should the FLOPS of a very specific linear algebra suite be used as the metric of best computers?
ajdecon 11y ago
The "High Performance Conjugate Gradients" benchmark was proposed a couple years ago as an alternative metric for ranking supercomputers. Its proponents claim its behavior is more similar to real applications (irregular access patterns, lower ratio of computation to memory access, etc), compared to linear algebra problems like the "High Performance Linpack" benchmark currently used by the Top500. The different performance numbers for top systems on HPCG vs HPL are pretty striking: http://www.hpcg-benchmark.org/custom/index.html?lid=155&slid=279 http://www.hpcg-benchmark.org/custom/index.html?lid=155&slid... Original proposal to use HPCG as an alternative to HPL for supercomputer rankings: http://www.sandia.gov/~maherou/docs/HPCG-Benchmark.pdf http://www.sandia.gov/~maherou/docs/HPCG-Benchmark.pdf
- jedbrown 11y ago
  HPCG basically measures STREAM and has many technical flaws making it scale-dependent and difficulty to adjudicate. As codeveloper of a different benchmark, I'll just cite this paper from a third party. https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceModelHPCG-2014.pdf https://hpgmg.org/static/MarjanovicGraciaGlass-PerformanceMo... The reality is that there are many dimensions to supercomputing performance and it's impossible for one number to capture the utility of the machine. Our HPGMG benchmark (https://hpgmg.org https://hpgmg.org) attempts to strike a balance and give useful supplementary information. I do think it's better than any other single benchmark for evaluating today's machines and will also prove to be more durable over time.
  - dekhn 11y ago
    How would you use a benchmark like this to predict the performance of a well-designed asynchronous parallel conjugate gradient solver, like most modern deep learning neural networks that run on Internet HPC machines?
    - jedbrown 11y ago
      CG isn't truly asynchronous due to its reductions. It can be pipelined in various ways (we have several implementations in PETSc), but performance requires a quality implementation of asynchronous reduction (e.g., MPI_Iallreduce) which the vendors have been slow about developing (I've been working with some on fixing this and Cray has made recent progress). With respect to deep learning and other applications using CG or related algorithms, the bottlenecks depend on the scale, and ability to expose locality, and operator/preconditioner representation. If there is no locality, then matrix-vector products require all-to-all communication which tend to dwarf the cost of the reductions in CG. Even with locality in the matrix-vector product, preconditioners often need to communicate globally in a scalable way similar to HPGMG. Operators need not be represented as a table of numbers or a sparse matrix format, but could use a tensor product, fast transform, or other information to compute the action using less storage. If they are represented explicitly (sparse or dense), then matrix-vector product performance (thus CG as a whole) is dominated by memory bandwidth for problem sizes that do not fit in cache. HPGMG tries to strike a balance between memory bandwidth demands and compute using a matrix-free representation. HPGMG also reports dynamic range expressed as Performance versus Time-to-solution as the problem size is varied, which allows applications to see performance barriers that might be relevant to them (e.g., see how Titan cannot do a solve in less than 200 ms while Edison can do 50 ms, and how that relates to climate simulation performance targets; see slide 7 of https://jedbrown.org/files/20150624-Versatility.pdf https://jedbrown.org/files/20150624-Versatility.pdf).

jedbrown 11y ago

Power is over-emphasized in HPC circles. According to FOIA reports (and you can back out similar from public budget information), less than 10% of the budget is going to energy. It is frequently used as an excuse to build machines that are inappropriate for the science (not just "hard to program", but actually inappropriate in the sense that even with infinite programming effort, they deliver less scientific value than a more conventional architecture). There is some value in making scientists uncomfortable so that they think of creative algorithmic solutions that may pay off as the inevitabilities of semiconductor physics become more apparent, but seeing the number of applications that are within small constant factors of proven barriers and the willingness to compromise quality of solution and/or run scientifically irrelevant configurations to demonstrate "speedup", I think it has gone too far.