7 ms·

> Never rely on benchmark tests unless the benchmarking code is known to be open source and compiled without using any Intel tools. A serious question - how ma

> Never rely on benchmark tests unless the benchmarking code is known to be open source and compiled without using any Intel tools. A serious question - how many common benchmark packages are compiled by ICC or uses Intel MKL? I hope the number is limited, otherwise all the benchmarks published by mainstream PC reviewers are potentially biased. If there's a serious ICC-in-benchmark problem, then only Phoronix's Linux benchmarks are trustworthy - the majority of benchmarks on Phoronix uses free and open source compilers and testsuits, with known versions, build parameters and optimization levels. Thanks Michael Larabel for his service for the community.

wtallis 6y ago
This concern really only applies to synthetic benchmarks (stuff like SPEC CPU). If you're testing a commercially available application or game as delivered to consumers, this issue does not invalidate the benchmark, it just makes the software vendor a bit of an Intel stooge.
- sudosysgen 6y ago
  Well, it depends. If you are a big enough outlet or a hyperscaler, you should care about raw-performance because if a sufficient amount of people make the same choice as you, the software vendor will adapt and you will end up with the competitive advantage.
- tagrun 6y ago
  Of course it doesn't matter for home consumers because real target audience for MKL isn't desktop user applications or games. Try using VASP or COMSOL or Mathematica on an AMD CPU. Both MKL and CUDA are major "issues" in HPC that limit decisions in purchasing clusters.
  - gnufx 6y ago
    I don't know about VASP because it's proprietary, but I have compared other DFT code on 64-core Bulldozer with 12-core Sandybridge nodes. I don't remember the numbers, but the high core count was rather effective in reducing the communication costs with all free software. https://www.archer2.ac.uk/ https://www.archer2.ac.uk/ will run a lot of that sort of thing. I think at least cp2k and CASTEP are included in the benchmarks, but they're listed somewhere.
    - tagrun 6y ago
      When you're actually a part of conversations on purchasing a cluster (which I gather from the way you talk, you haven't been a part of) which cost $400k ~ $1M for a smallish/mid-sized, arguments like "I don't remember the numbers" "they're listed somewhere", "there's this some other random DFT code" aren't effective. The hard fact is (which I hate), you get results faster with MKL on Intel than any other alternatives. This is more so with the proprietary software that are golden standards. > I have compared other DFT code on 64-core Bulldozer with 12-core Sandybridge nodes And what's that comparison supposed to tell us, aside from the obvious fact that MPI introduces latency? That's just about the number of cores, not the performance of a each core. You need to compare 64-core AMD node against a 64-core Intel node.
      - gnufx 6y ago
        I don't remember, because the measurements on the £1M purchase was maybe five years ago, but they taught a useful lesson. I didn't see figures in what I was responding to. If I'd had more influence on the purchase, as opposed to observing the process, we wouldn't have ended up with a pure sandybridge system, which was a mistake. Anyhow, my all-free-software version of cp2k was faster on it than an all-Intel version on slightly faster CPUs on an otherwise equivalent cluster. I measured and paid attention to the MPI, which benefited everything using alltoallv. The large core-count AMD boxes were simply a better bet for the range of work on a university HPC system. It's not as if most codes topped out an arithmetic intensity and there was a serious problem with serial performance, even if MKL had been significantly better than the free libraries, which it wasn't. For a recent exercise, spending rather more money on AMD CPUs for the UK Tier 1 system, look at the Archer2 reference and benchmarking for it. It's expected to run large amounts of VASP-like code; www.archer.ac.uk publishes usage of the current system. Circumstances differ, and I'm pointing out contrary experience, understanding the measurements and what determined them.
- formerly_proven 6y ago
  The go-to Intel-crushing benchmark these days is Cinebench R20, which runs on Intel's Embree raytracer.
  - segfaultbuserr 6y ago
    So Cinebench R20 is a suspect?! That's not good...
  - gruez 6y ago
    But is that compiled by ICC?
bitL 6y ago
Most of Python math libraries? So a majority of Machine Learning folks is affected.
- roseway4 6y ago
  Most Python math and ML libraries are compiled using GCC or LLVM and are linked against against cBLAS or OpenBLAS[1]. The latter is highly performant on both Intel and AMD (and other platforms). Some libraries are optionally compiled against the MKL, in particular those that are distributed with the Anaconda Python distribution. [1] https://numpy.org/doc/stable/reference/routines.linalg.html?highlight=blas https://numpy.org/doc/stable/reference/routines.linalg.html?...
  - bitL 6y ago
    I'd have to disagree, whenever I install new pip/conda packages, an MKL version is downloaded by default (I've never seen any *BLAS version by default in my life).
    - ddavis 6y ago
      NumPy binary wheels on PyPI (i.e. from pip) are built with OpenBLAS. NumPy from the official Anaconda, Inc. conda channel defaults to MKL. The conda-forge channel defaults to OpenBLAS. Explained in the NumPy docs here: https://numpy.org/install/ https://numpy.org/install/
  - erwincoumans 6y ago
    PyTorch uses MKL https://www.google.com/amp/s/amp.reddit.com/r/MachineLearning/comments/iap6yo/discussion_pytorch_favors_intel_against_amds/ https://www.google.com/amp/s/amp.reddit.com/r/MachineLearnin... and the workaround for AMD has been disabled by Intel
    - celrod 6y ago
      I think MKL actually fixed Zen performance. That is, the workaround no longer makes any difference because it is no longer needed. Small matrix multiply benchmarks on a Zen2 (Ryzen 7 4700U), featuring MKL 2020.1.216+0, OpenBLAS, and Eigen: https://gist.github.com/stillyslalom/bd916e3d26b4531364676ac09d8469ad#gistcomment-3403272 https://gist.github.com/stillyslalom/bd916e3d26b4531364676ac... MKL's benchmark performance requires AVX + FMA. 3.5 GHz * (4 add + 4 multiply) * 2 fma/cycle = 56 peak GFLOPS. To exceed 50 GFLOPS without them would imply the CPU ran at 12.5 GHz. OpenBLAS, on the other hand, actually performed poorly because it was limited to SSE thanks to a bug preventing the CPU from being recognized as Zen2.
tagrun 6y ago
MKL is a real problem in HPC, and it does play a significant role in decisions when purchasing clusters. In addition, software like Mathematica/Matlab which use only MKL can affect decisions even for office workstations.

wtallis 6y ago

This concern really only applies to synthetic benchmarks (stuff like SPEC CPU). If you're testing a commercially available application or game as delivered to consumers, this issue does not invalidate the benchmark, it just makes the software vendor a bit of an Intel stooge.

sudosysgen 6y ago
Well, it depends. If you are a big enough outlet or a hyperscaler, you should care about raw-performance because if a sufficient amount of people make the same choice as you, the software vendor will adapt and you will end up with the competitive advantage.
tagrun 6y ago
Of course it doesn't matter for home consumers because real target audience for MKL isn't desktop user applications or games. Try using VASP or COMSOL or Mathematica on an AMD CPU. Both MKL and CUDA are major "issues" in HPC that limit decisions in purchasing clusters.
- gnufx 6y ago
  I don't know about VASP because it's proprietary, but I have compared other DFT code on 64-core Bulldozer with 12-core Sandybridge nodes. I don't remember the numbers, but the high core count was rather effective in reducing the communication costs with all free software. https://www.archer2.ac.uk/ https://www.archer2.ac.uk/ will run a lot of that sort of thing. I think at least cp2k and CASTEP are included in the benchmarks, but they're listed somewhere.
  - tagrun 6y ago
    When you're actually a part of conversations on purchasing a cluster (which I gather from the way you talk, you haven't been a part of) which cost $400k ~ $1M for a smallish/mid-sized, arguments like "I don't remember the numbers" "they're listed somewhere", "there's this some other random DFT code" aren't effective. The hard fact is (which I hate), you get results faster with MKL on Intel than any other alternatives. This is more so with the proprietary software that are golden standards. > I have compared other DFT code on 64-core Bulldozer with 12-core Sandybridge nodes And what's that comparison supposed to tell us, aside from the obvious fact that MPI introduces latency? That's just about the number of cores, not the performance of a each core. You need to compare 64-core AMD node against a 64-core Intel node.
    - gnufx 6y ago
      I don't remember, because the measurements on the £1M purchase was maybe five years ago, but they taught a useful lesson. I didn't see figures in what I was responding to. If I'd had more influence on the purchase, as opposed to observing the process, we wouldn't have ended up with a pure sandybridge system, which was a mistake. Anyhow, my all-free-software version of cp2k was faster on it than an all-Intel version on slightly faster CPUs on an otherwise equivalent cluster. I measured and paid attention to the MPI, which benefited everything using alltoallv. The large core-count AMD boxes were simply a better bet for the range of work on a university HPC system. It's not as if most codes topped out an arithmetic intensity and there was a serious problem with serial performance, even if MKL had been significantly better than the free libraries, which it wasn't. For a recent exercise, spending rather more money on AMD CPUs for the UK Tier 1 system, look at the Archer2 reference and benchmarking for it. It's expected to run large amounts of VASP-like code; www.archer.ac.uk publishes usage of the current system. Circumstances differ, and I'm pointing out contrary experience, understanding the measurements and what determined them.
formerly_proven 6y ago
The go-to Intel-crushing benchmark these days is Cinebench R20, which runs on Intel's Embree raytracer.
- segfaultbuserr 6y ago
  So Cinebench R20 is a suspect?! That's not good...
- gruez 6y ago
  But is that compiled by ICC?

bitL 6y ago

Most of Python math libraries? So a majority of Machine Learning folks is affected.

roseway4 6y ago
Most Python math and ML libraries are compiled using GCC or LLVM and are linked against against cBLAS or OpenBLAS[1]. The latter is highly performant on both Intel and AMD (and other platforms). Some libraries are optionally compiled against the MKL, in particular those that are distributed with the Anaconda Python distribution. [1] https://numpy.org/doc/stable/reference/routines.linalg.html?highlight=blas https://numpy.org/doc/stable/reference/routines.linalg.html?...
- bitL 6y ago
  I'd have to disagree, whenever I install new pip/conda packages, an MKL version is downloaded by default (I've never seen any *BLAS version by default in my life).
  - ddavis 6y ago
    NumPy binary wheels on PyPI (i.e. from pip) are built with OpenBLAS. NumPy from the official Anaconda, Inc. conda channel defaults to MKL. The conda-forge channel defaults to OpenBLAS. Explained in the NumPy docs here: https://numpy.org/install/ https://numpy.org/install/
- erwincoumans 6y ago
  PyTorch uses MKL https://www.google.com/amp/s/amp.reddit.com/r/MachineLearning/comments/iap6yo/discussion_pytorch_favors_intel_against_amds/ https://www.google.com/amp/s/amp.reddit.com/r/MachineLearnin... and the workaround for AMD has been disabled by Intel
  - celrod 6y ago
    I think MKL actually fixed Zen performance. That is, the workaround no longer makes any difference because it is no longer needed. Small matrix multiply benchmarks on a Zen2 (Ryzen 7 4700U), featuring MKL 2020.1.216+0, OpenBLAS, and Eigen: https://gist.github.com/stillyslalom/bd916e3d26b4531364676ac09d8469ad#gistcomment-3403272 https://gist.github.com/stillyslalom/bd916e3d26b4531364676ac... MKL's benchmark performance requires AVX + FMA. 3.5 GHz * (4 add + 4 multiply) * 2 fma/cycle = 56 peak GFLOPS. To exceed 50 GFLOPS without them would imply the CPU ran at 12.5 GHz. OpenBLAS, on the other hand, actually performed poorly because it was limited to SSE thanks to a bug preventing the CPU from being recognized as Zen2.

tagrun 6y ago

MKL is a real problem in HPC, and it does play a significant role in decisions when purchasing clusters. In addition, software like Mathematica/Matlab which use only MKL can affect decisions even for office workstations.