3 ms·

Has anyone seen similar benchmarks, but with BLIS? OpenBLAS is interesting, but BLIS would possibly perform better.

edraferi 7y ago
The article says they’d like to include BLIS in the benchmark, but nobody has written a good Numpy Conda package with BLIS hooks yet
- fock 7y ago
  because all of the people "benchmarking" cpus nowadays are just clueless self-advertisers...
- lhl 7y ago
  The article was specific to AMD's BLIS fork, but just as an FYI, using standard BLIS w/ numpy (which I believe would still outperform OpenBLAS) is pretty straightforward: conda create -c conda-forge -n numpy-blis numpy "blas=*=blis"
  - wjnc 7y ago
    Are there any reads you can recommend on how to recover from (possibly) failed experiments like these under pip / conda / linux in general? I've been doing R and Linux for two decades, but still am perfectly capable of totally breaking my python/jupyter-workflow by messing up dependencies without knowing how to recover. Would love to learn to remedy that gap and reducing the risk.
    - paol 7y ago
      When dealing with purely python packages: always always work in python virtual envs. Conduct these kinds of experiments in separate venvs. If worse comes to worse, a reset consists simply in deleting the venv directory and installing from scratch. If an experiment involves system packages it's a little trickier. You can learn about Docker (which does for the whole system what venv does for python packages), but in most cases you may get away with just taking note of what packages you have installed/uninstalled, so you can revert the process later if needed.
    - fock 7y ago
      never work as root, just undo all changes you did for it to work
    - lhl 7y ago
      You can actually see in my pasted command that I created a new venv for the blis install. It's as simple as "activate" or "deactivate" to switch around. That's basically all there is to it, but here are some docs for you to reference from the top page of search: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html https://docs.conda.io/projects/conda/en/latest/user-guide/ta... https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/ https://uoa-eresearch.github.io/eresearch-cookbook/recipe/20...
- gnufx 7y ago
  Why would you need a Conda package to measure performance or BLIS or another implementation? Assuming an ELF system, either preload an alternative library, or make a trivial shim for libblas.so3, as the Fedora blis package does. See the somewhat old https://loveshack.fedorapeople.org/blas-subversion.html https://loveshack.fedorapeople.org/blas-subversion.html for example.
lhl 7y ago
Based on Zen1 Epyc performance it would seem that BLIS might be significantly more performant even without AMD-specific optimizations (I know AMD has their own fork, but check out the 64 core results for "vanilla" BLIS: https://github.com/flame/blis/blob/master/docs/Performance.md#epyc https://github.com/flame/blis/blob/master/docs/Performance.m...)
- shaklee3 7y ago
  Something is still weird with the gemm benchmarks. That link you sent has BLIS winning all around, but the absolute value of the BLIS is poor. MKL gets around 2TFLOPS on xeons, and this blis benchmark is only hitting around 30-50GFLOPS. Are AMDs hitting over 1TFLOPS anywhere?
  - shaklee3 7y ago
    To illustrate this, let's assume the GEMM used avx512 perfectly. Roughly that's 2.5e9 (clock) * 16 (ops per cycle) * 2 (FMA per cycle) * 64 (cores). That puts Intel's peak performance over 2TFLOPs, which they're showing. That's about 80x higher than AMD on BLIS...
  - lhl 7y ago
    Everything on that benchmark page is listed as GFLOPS/core so at 30GFLOPS that Epyc 7551 should be pushing around 2 TFLOPS. You can take a look at reproducing them if you want if you have a similar system: https://github.com/flame/blis/blob/master/docs/Performance.md#reproduction https://github.com/flame/blis/blob/master/docs/Performance.m... I ran into some compile issues, but I care more about numpy so I gave this script a try: https://markus-beuckelmann.de/blog/boosting-numpy-blas.html https://markus-beuckelmann.de/blog/boosting-numpy-blas.html with some quick/dirty results on my Ryzen 3700X (8C16T) workstation (BLIS doesn't seem to perform very well on this small test). You can see performance basically double on MKL when MKL_DEBUG_CPU_TYPE=5 is used. I'll probably continue to stick w/ OpenBLAS for now: ## blis 0.6.0 h516909a_0 # conda activate numpy-blis # conda run python bench.py Dotted two 4096x4096 matrices in 2.30 s. Dotted two vectors of length 524288 in 0.08 ms. SVD of a 2048x1024 matrix in 0.94 s. Cholesky decomposition of a 2048x2048 matrix in 0.24 s. Eigendecomposition of a 2048x2048 matrix in 6.36 s. ## libopenblas 0.3.7 h5ec1e0e_4 # conda activate numpy-openblas # conda run python bench.py Dotted two 4096x4096 matrices in 0.41 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.55 s. Cholesky decomposition of a 2048x2048 matrix in 0.14 s. Eigendecomposition of a 2048x2048 matrix in 5.53 s. ## mkl 2019.4 243 # conda activate numpy-mkl # conda run python bench.py Dotted two 4096x4096 matrices in 1.53 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.51 s. Cholesky decomposition of a 2048x2048 matrix in 0.29 s. Eigendecomposition of a 2048x2048 matrix in 4.79 s. # export MKL_DEBUG_CPU_TYPE=5 # conda run python bench.py Dotted two 4096x4096 matrices in 0.33 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.29 s. Cholesky decomposition of a 2048x2048 matrix in 0.15 s. Eigendecomposition of a 2048x2048 matrix in 3.29 s.

The article says they’d like to include BLIS in the benchmark, but nobody has written a good Numpy Conda package with BLIS hooks yet

fock 7y ago
because all of the people "benchmarking" cpus nowadays are just clueless self-advertisers...
lhl 7y ago
The article was specific to AMD's BLIS fork, but just as an FYI, using standard BLIS w/ numpy (which I believe would still outperform OpenBLAS) is pretty straightforward: conda create -c conda-forge -n numpy-blis numpy "blas=*=blis"
- wjnc 7y ago
  Are there any reads you can recommend on how to recover from (possibly) failed experiments like these under pip / conda / linux in general? I've been doing R and Linux for two decades, but still am perfectly capable of totally breaking my python/jupyter-workflow by messing up dependencies without knowing how to recover. Would love to learn to remedy that gap and reducing the risk.
  - paol 7y ago
    When dealing with purely python packages: always always work in python virtual envs. Conduct these kinds of experiments in separate venvs. If worse comes to worse, a reset consists simply in deleting the venv directory and installing from scratch. If an experiment involves system packages it's a little trickier. You can learn about Docker (which does for the whole system what venv does for python packages), but in most cases you may get away with just taking note of what packages you have installed/uninstalled, so you can revert the process later if needed.
  - fock 7y ago
    never work as root, just undo all changes you did for it to work
  - lhl 7y ago
    You can actually see in my pasted command that I created a new venv for the blis install. It's as simple as "activate" or "deactivate" to switch around. That's basically all there is to it, but here are some docs for you to reference from the top page of search: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html https://docs.conda.io/projects/conda/en/latest/user-guide/ta... https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/ https://uoa-eresearch.github.io/eresearch-cookbook/recipe/20...
gnufx 7y ago
Why would you need a Conda package to measure performance or BLIS or another implementation? Assuming an ELF system, either preload an alternative library, or make a trivial shim for libblas.so3, as the Fedora blis package does. See the somewhat old https://loveshack.fedorapeople.org/blas-subversion.html https://loveshack.fedorapeople.org/blas-subversion.html for example.

lhl 7y ago

Based on Zen1 Epyc performance it would seem that BLIS might be significantly more performant even without AMD-specific optimizations (I know AMD has their own fork, but check out the 64 core results for "vanilla" BLIS: https://github.com/flame/blis/blob/master/docs/Performance.md#epyc https://github.com/flame/blis/blob/master/docs/Performance.m...)

shaklee3 7y ago
Something is still weird with the gemm benchmarks. That link you sent has BLIS winning all around, but the absolute value of the BLIS is poor. MKL gets around 2TFLOPS on xeons, and this blis benchmark is only hitting around 30-50GFLOPS. Are AMDs hitting over 1TFLOPS anywhere?
- shaklee3 7y ago
  To illustrate this, let's assume the GEMM used avx512 perfectly. Roughly that's 2.5e9 (clock) * 16 (ops per cycle) * 2 (FMA per cycle) * 64 (cores). That puts Intel's peak performance over 2TFLOPs, which they're showing. That's about 80x higher than AMD on BLIS...
- lhl 7y ago
  Everything on that benchmark page is listed as GFLOPS/core so at 30GFLOPS that Epyc 7551 should be pushing around 2 TFLOPS. You can take a look at reproducing them if you want if you have a similar system: https://github.com/flame/blis/blob/master/docs/Performance.md#reproduction https://github.com/flame/blis/blob/master/docs/Performance.m... I ran into some compile issues, but I care more about numpy so I gave this script a try: https://markus-beuckelmann.de/blog/boosting-numpy-blas.html https://markus-beuckelmann.de/blog/boosting-numpy-blas.html with some quick/dirty results on my Ryzen 3700X (8C16T) workstation (BLIS doesn't seem to perform very well on this small test). You can see performance basically double on MKL when MKL_DEBUG_CPU_TYPE=5 is used. I'll probably continue to stick w/ OpenBLAS for now: ## blis 0.6.0 h516909a_0 # conda activate numpy-blis # conda run python bench.py Dotted two 4096x4096 matrices in 2.30 s. Dotted two vectors of length 524288 in 0.08 ms. SVD of a 2048x1024 matrix in 0.94 s. Cholesky decomposition of a 2048x2048 matrix in 0.24 s. Eigendecomposition of a 2048x2048 matrix in 6.36 s. ## libopenblas 0.3.7 h5ec1e0e_4 # conda activate numpy-openblas # conda run python bench.py Dotted two 4096x4096 matrices in 0.41 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.55 s. Cholesky decomposition of a 2048x2048 matrix in 0.14 s. Eigendecomposition of a 2048x2048 matrix in 5.53 s. ## mkl 2019.4 243 # conda activate numpy-mkl # conda run python bench.py Dotted two 4096x4096 matrices in 1.53 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.51 s. Cholesky decomposition of a 2048x2048 matrix in 0.29 s. Eigendecomposition of a 2048x2048 matrix in 4.79 s. # export MKL_DEBUG_CPU_TYPE=5 # conda run python bench.py Dotted two 4096x4096 matrices in 0.33 s. Dotted two vectors of length 524288 in 0.02 ms. SVD of a 2048x1024 matrix in 0.29 s. Cholesky decomposition of a 2048x2048 matrix in 0.15 s. Eigendecomposition of a 2048x2048 matrix in 3.29 s.