7 ms·

R, OpenMP, MKL, Disaster

103 points by yomritoyj 5y ago

deng 5y ago
> As a good citizen he wanted to file a documentation bug to have this behaviour documented. But R’s bug tracker seems not to be open to the public. So the story has to be recorded here for Google to find. Huh? The bug tracker is here: https://bugs.r-project.org/ https://bugs.r-project.org/ Yes, for filing a bug you need to request an account because they apparently were overwhelmed with spam, as documented here: https://www.r-project.org/bugs.html https://www.r-project.org/bugs.html
- dash2 5y ago
  The first place to report this would be on the R-devel mailing list.
kettleballroll 5y ago
In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then). I benched this extensively for all my R usecases and for several systems that i cared about back then. So there is usually no need to use MKL in the first place.
- danieldk 5y ago
  OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it with the right flags (in a multi-threaded application: single-threaded, but with locking). it's worth noting that OpenBLAS is as fast as MKL This depends highly on the application. E.g. MKL provides batch GEMM, which is used by libraries like PyTorch. So if you use PyTorch for machine learning, performance is still much better with MKL. Of course, that is if you do not have an AMD CPU. If you have an AMD CPU, you have to override Intel CPU detection if you do not want abysmal performance: https://danieldk.eu/Posts/2020-08-31-MKL-Zen.html https://danieldk.eu/Posts/2020-08-31-MKL-Zen.html https://www.agner.org/optimize/blog/read.php?i=49 https://www.agner.org/optimize/blog/read.php?i=49 The BLAS/LAPACK ecosystem is a mess. I wish that Intel would just open source MKL and properly support AMD CPUs.
  - jabl 5y ago
    > OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Can you explain what you mean by this? Are you saying there's a correctness issue here? I only recall running into issues with MPI, where you (typically) run one MPI rank (process) per CPU core. Then if you combine that with a multi-threaded BLAS library you'll suddenly have N^2 BLAS threads fighting over the CPU's and performance goes down the drain. The solution to this is, like you say, to use a single-threaded OpenBLAS, or then the OpenMP OpenBLAS and set OMP_NUM_THREADS=1 I guess with threads you'll have the same issue if you launch N cpu-bound threads and all those call BLAS, resulting in the same N^2 issue as you see with MPI.
    - danieldk 5y ago
      Can you explain what you mean by this? There is a nice description of this: https://github.com/xianyi/OpenBLAS/issues/2543 https://github.com/xianyi/OpenBLAS/issues/2543 At a previous employer, we have seen various issues, including crashes, non-determinisms, etc. Usually, these issues would go away when switching to MKL.
      - syockit 5y ago
        One of the more painful issues is hanging (lockup) at full CPU usage. At my workplace, initially we introduced a timeout to workaround the hang while trying to determine the cause of the hang. It happened within multithread R code. Various build flags for OpenBLAS have been tried to no avail. Setting OPENBLAS_NUM_THREADS=1 surely makes the problem go away, at the expense of performance. That R code has since been ported to Python, but we faced the same issue again when using ThreadPoolExecutor, so we had to change it into ProcessPoolExecutor instead.
  - kergonath 5y ago
    > OpenBLAS is incompatible with application threads. I’ve never had any issue when using it in OpenMP codes (either compiling it myself or using the libopenblas_omp.so present in some distros), what do you mean by “burn in a fire”?
- jabl 5y ago
  > i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then Also in a previous life, I recall running into distro openblas packages that were not compiled with DYNAMIC_ARCH=1 (which enables the openblas runtime cpu target architecture selection, similar to e.g. MKL) but were instead compiled with some lowest common denominator x86_64 arch. I filed some bug(s?), and IIRC this problem has subsequently been fixed.
keewee7 5y ago
As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.
- WhompingWindows 5y ago
  Wow, tell us how you really feel. How much have you used R and Python? Maybe those data scientists would prefer if you didn't viscerally hate the main data/statistics language and didn't call it useless for things beyond a narrow use-case. It may lead to better outcomes if people hated things less and tried to understand the valid use-cases, for instance the reams and reams of statistics that can be done on R where Python may lag behind, since R is the lingua franca of statistics and research.
- project2501a 5y ago
  it is not much as R, as the (relative) unwillingness to break compatibility or enforce global standards. Why do all string functions do not accept UTF-8?
- kgwgk 5y ago
  It could be worse. They could learn python and still prefer to use R!
- otabdeveloper4 5y ago
  R is way more powerful and flexible for data science stuff. (Going from Python to R is almost like going from Excel to Python.)
- gnufx 5y ago
  I've never seen anything for Python that allows you to a linear algebra-based code and run it at maybe petascale with trivial modifications. There's an R example somewhere under https://pbdr.org/publications.html https://pbdr.org/publications.html
- civilized 5y ago
  Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever. I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's always because the people involved don't understand computers, don't read documentation, don't debug, don't do root cause analysis, and want to quickly pass off responsibility for their laziness and incompetence. Meanwhile I and my team are happily chugging away, producing millions of dollars of reliable value for my company in R year after year. Python lags far behind R in wide swaths of data science. Pandas is inferior to both dplyr and data.table, and R's modeling capabilities blow Python's out of the water in breadth and depth. You only use Python when you have to, e.g. for unstructured data and deep learning type stuff. If your colleagues make you deal with their bad R code, that's too bad, but don't blame the language. It's designed to be easy to use, so a lot of bad coders use it. Go train your bad coders or hire better ones.
kzrdude 5y ago
Systems that fail without producing an error or warning are worrying and unsettling to me.
gnufx 5y ago
It's perhaps worth saying that if you must mix OpenMP libraries built against the LLVM (Kuck/Intel) runtime, and GNU GOMP on GNU/Linux: Ensure libomp is built with GOMP compatibility, however that's configured, make a shim from the result, like gcc -shared -Wl,-soname=libgomp.so.1 -o libgomp.so.1 empty.c -lomp5 where empty.c is an empty file, and put the result on LD_LIBRARY_PATH ahead of the real libgomp. Alternatively, preload the compatible libomp5. On Debian 11 there's already a libgomp in the llvm packaging. Dynamic linking assumed, as is right and fitting.
boulos 5y ago
I don't have MKL to try this out, but I'd check that the MKL threading choice actually didn't break the initialization to 1.0 loop. That is, instead of checking after doing the x[i] *= SCALE bit with cblas, I would check both before and after the scaling.
lstmemery 5y ago
I had a similar problem in a prediction pipeline a few years back. If I remember correctly, someone updated a R package to the next minor version. The package was to read an obscure file format. The fix installed a new C++ library. That C++ library somehow interacted with a second R package (using a specialized type of linear model) when compiled at source and all the results coming out of our package were subtly wrong but only with large files. It turns out the way the second R package would determine the required precision of floats in sparse arrays was based on the compiled linear algebra libraries available. It took a week for us to debug and ultimately it was easier for us to just rewrite the whole thing in Python. Renv has made things easier but I don't think packrat/renv allows you to lock C/C++ libraries as well as R ones.
teleforce 5y ago
In this excellent article Patrick Li, the author of a new optionally type language Stanza and co-founder of JITX (YC S18), provided a compelling reason to design a new language [1]. TL;DR, a powerful language like Ruby enabled the creation of powerful RoR library and framework that help spawned unicorn size startups like Github and Twitter, that's otherwise not feasible. I want to add another dimension to this argument, what if we can maintain an existing language eco-system (library, community, etc) but modernize the engine that's running and compilation of the R language. This new engine can avoid the dreaded global locking limitation, provides native multi-thread applications and seamless interface with non-native R libraries in C/C++. Interestingly someone has tried this, with a sponsorship from Oracle no less, and presented this futile effort in the last year's R flagship conference keynote [2]. IMHO he will be more successful in his endeavour if using D language in his previous endeavors. What so special about D language you may ask? I would refer to the fact most of the languages do not provide Ruby on Rails (RoR) like tool except D but that for another story (see ref [1]). There's also the fact that D has a working alternative library to OpenBLAS and MKL, and it's even faster than both them five years back [3]! D also supports open method as an alternative for multiple dispatches that is much touted by Julia language community. D is also bringing native support for borrow checker feature that's always mentioned in the same sentence as Rust language. In addition D also has second to none FFI support for C and C++ language. Heck the latest D compiler has standard C compiler built-in. I can go on furthermore but I think you've probably already got the pictures. My not so humble proposal to R and D language community is to compile R on top of D language. Essentially you a have dynamic language of R that is compiled at runtime (CTFE) on top of static D language. This approach is becoming more popular now as posted recently for the new Val and Valet language combination [4]. Just think of CTFE as the new JVM, but provides truly static and native compilation for R. [1] What makes a programming language productive? “Stop designing languages. Write libraries instead.”: https://jaxenter.com/stop-designing-languages-write-libraries-instead-126738.html https://jaxenter.com/stop-designing-languages-write-librarie... [2] Why R? 2020 Keynote - Jan Vitek - How I Learned to Love Failing at Compiling R: https://www.youtube.com/watch?v=VdD0nHbcyk4 https://www.youtube.com/watch?v=VdD0nHbcyk4 [3] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/... [4] Show HN: Val - A powerful static and dynamic programming language(val-lang.org): https://news.ycombinator.com/item?id=28683171 https://news.ycombinator.com/item?id=28683171

103 points by yomritoyj 5y ago

deng 5y ago

> As a good citizen he wanted to file a documentation bug to have this behaviour documented. But R’s bug tracker seems not to be open to the public. So the story has to be recorded here for Google to find. Huh? The bug tracker is here: https://bugs.r-project.org/ https://bugs.r-project.org/ Yes, for filing a bug you need to request an account because they apparently were overwhelmed with spam, as documented here: https://www.r-project.org/bugs.html https://www.r-project.org/bugs.html

dash2 5y ago
The first place to report this would be on the R-devel mailing list.

kettleballroll 5y ago

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then). I benched this extensively for all my R usecases and for several systems that i cared about back then. So there is usually no need to use MKL in the first place.

danieldk 5y ago
OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it with the right flags (in a multi-threaded application: single-threaded, but with locking). it's worth noting that OpenBLAS is as fast as MKL This depends highly on the application. E.g. MKL provides batch GEMM, which is used by libraries like PyTorch. So if you use PyTorch for machine learning, performance is still much better with MKL. Of course, that is if you do not have an AMD CPU. If you have an AMD CPU, you have to override Intel CPU detection if you do not want abysmal performance: https://danieldk.eu/Posts/2020-08-31-MKL-Zen.html https://danieldk.eu/Posts/2020-08-31-MKL-Zen.html https://www.agner.org/optimize/blog/read.php?i=49 https://www.agner.org/optimize/blog/read.php?i=49 The BLAS/LAPACK ecosystem is a mess. I wish that Intel would just open source MKL and properly support AMD CPUs.
- jabl 5y ago
  > OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Can you explain what you mean by this? Are you saying there's a correctness issue here? I only recall running into issues with MPI, where you (typically) run one MPI rank (process) per CPU core. Then if you combine that with a multi-threaded BLAS library you'll suddenly have N^2 BLAS threads fighting over the CPU's and performance goes down the drain. The solution to this is, like you say, to use a single-threaded OpenBLAS, or then the OpenMP OpenBLAS and set OMP_NUM_THREADS=1 I guess with threads you'll have the same issue if you launch N cpu-bound threads and all those call BLAS, resulting in the same N^2 issue as you see with MPI.
  - danieldk 5y ago
    Can you explain what you mean by this? There is a nice description of this: https://github.com/xianyi/OpenBLAS/issues/2543 https://github.com/xianyi/OpenBLAS/issues/2543 At a previous employer, we have seen various issues, including crashes, non-determinisms, etc. Usually, these issues would go away when switching to MKL.
    - syockit 5y ago
      One of the more painful issues is hanging (lockup) at full CPU usage. At my workplace, initially we introduced a timeout to workaround the hang while trying to determine the cause of the hang. It happened within multithread R code. Various build flags for OpenBLAS have been tried to no avail. Setting OPENBLAS_NUM_THREADS=1 surely makes the problem go away, at the expense of performance. That R code has since been ported to Python, but we faced the same issue again when using ThreadPoolExecutor, so we had to change it into ProcessPoolExecutor instead.
- kergonath 5y ago
  > OpenBLAS is incompatible with application threads. I’ve never had any issue when using it in OpenMP codes (either compiling it myself or using the libopenblas_omp.so present in some distros), what do you mean by “burn in a fire”?
jabl 5y ago
> i would expect that system provided ones with system detection would be as good, but that wasn't always the case back then Also in a previous life, I recall running into distro openblas packages that were not compiled with DYNAMIC_ARCH=1 (which enables the openblas runtime cpu target architecture selection, similar to e.g. MKL) but were instead compiled with some lowest common denominator x86_64 arch. I filed some bug(s?), and IIRC this problem has subsequently been fixed.

keewee7 5y ago

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

WhompingWindows 5y ago
Wow, tell us how you really feel. How much have you used R and Python? Maybe those data scientists would prefer if you didn't viscerally hate the main data/statistics language and didn't call it useless for things beyond a narrow use-case. It may lead to better outcomes if people hated things less and tried to understand the valid use-cases, for instance the reams and reams of statistics that can be done on R where Python may lag behind, since R is the lingua franca of statistics and research.
project2501a 5y ago
it is not much as R, as the (relative) unwillingness to break compatibility or enforce global standards. Why do all string functions do not accept UTF-8?
kgwgk 5y ago
It could be worse. They could learn python and still prefer to use R!
otabdeveloper4 5y ago
R is way more powerful and flexible for data science stuff. (Going from Python to R is almost like going from Excel to Python.)
gnufx 5y ago
I've never seen anything for Python that allows you to a linear algebra-based code and run it at maybe petascale with trivial modifications. There's an R example somewhere under https://pbdr.org/publications.html https://pbdr.org/publications.html
civilized 5y ago
Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever. I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's always because the people involved don't understand computers, don't read documentation, don't debug, don't do root cause analysis, and want to quickly pass off responsibility for their laziness and incompetence. Meanwhile I and my team are happily chugging away, producing millions of dollars of reliable value for my company in R year after year. Python lags far behind R in wide swaths of data science. Pandas is inferior to both dplyr and data.table, and R's modeling capabilities blow Python's out of the water in breadth and depth. You only use Python when you have to, e.g. for unstructured data and deep learning type stuff. If your colleagues make you deal with their bad R code, that's too bad, but don't blame the language. It's designed to be easy to use, so a lot of bad coders use it. Go train your bad coders or hire better ones.

kzrdude 5y ago

Systems that fail without producing an error or warning are worrying and unsettling to me.

gnufx 5y ago

It's perhaps worth saying that if you must mix OpenMP libraries built against the LLVM (Kuck/Intel) runtime, and GNU GOMP on GNU/Linux: Ensure libomp is built with GOMP compatibility, however that's configured, make a shim from the result, like gcc -shared -Wl,-soname=libgomp.so.1 -o libgomp.so.1 empty.c -lomp5 where empty.c is an empty file, and put the result on LD_LIBRARY_PATH ahead of the real libgomp. Alternatively, preload the compatible libomp5. On Debian 11 there's already a libgomp in the llvm packaging. Dynamic linking assumed, as is right and fitting.

boulos 5y ago

I don't have MKL to try this out, but I'd check that the MKL threading choice actually didn't break the initialization to 1.0 loop. That is, instead of checking after doing the x[i] *= SCALE bit with cblas, I would check both before and after the scaling.

lstmemery 5y ago

I had a similar problem in a prediction pipeline a few years back. If I remember correctly, someone updated a R package to the next minor version. The package was to read an obscure file format. The fix installed a new C++ library. That C++ library somehow interacted with a second R package (using a specialized type of linear model) when compiled at source and all the results coming out of our package were subtly wrong but only with large files. It turns out the way the second R package would determine the required precision of floats in sparse arrays was based on the compiled linear algebra libraries available. It took a week for us to debug and ultimately it was easier for us to just rewrite the whole thing in Python. Renv has made things easier but I don't think packrat/renv allows you to lock C/C++ libraries as well as R ones.

teleforce 5y ago

In this excellent article Patrick Li, the author of a new optionally type language Stanza and co-founder of JITX (YC S18), provided a compelling reason to design a new language [1]. TL;DR, a powerful language like Ruby enabled the creation of powerful RoR library and framework that help spawned unicorn size startups like Github and Twitter, that's otherwise not feasible. I want to add another dimension to this argument, what if we can maintain an existing language eco-system (library, community, etc) but modernize the engine that's running and compilation of the R language. This new engine can avoid the dreaded global locking limitation, provides native multi-thread applications and seamless interface with non-native R libraries in C/C++. Interestingly someone has tried this, with a sponsorship from Oracle no less, and presented this futile effort in the last year's R flagship conference keynote [2]. IMHO he will be more successful in his endeavour if using D language in his previous endeavors. What so special about D language you may ask? I would refer to the fact most of the languages do not provide Ruby on Rails (RoR) like tool except D but that for another story (see ref [1]). There's also the fact that D has a working alternative library to OpenBLAS and MKL, and it's even faster than both them five years back [3]! D also supports open method as an alternative for multiple dispatches that is much touted by Julia language community. D is also bringing native support for borrow checker feature that's always mentioned in the same sentence as Rust language. In addition D also has second to none FFI support for C and C++ language. Heck the latest D compiler has standard C compiler built-in. I can go on furthermore but I think you've probably already got the pictures. My not so humble proposal to R and D language community is to compile R on top of D language. Essentially you a have dynamic language of R that is compiled at runtime (CTFE) on top of static D language. This approach is becoming more popular now as posted recently for the new Val and Valet language combination [4]. Just think of CTFE as the new JVM, but provides truly static and native compilation for R. [1] What makes a programming language productive? “Stop designing languages. Write libraries instead.”: https://jaxenter.com/stop-designing-languages-write-libraries-instead-126738.html https://jaxenter.com/stop-designing-languages-write-librarie... [2] Why R? 2020 Keynote - Jan Vitek - How I Learned to Love Failing at Compiling R: https://www.youtube.com/watch?v=VdD0nHbcyk4 https://www.youtube.com/watch?v=VdD0nHbcyk4 [3] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen: http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/... [4] Show HN: Val - A powerful static and dynamic programming language(val-lang.org): https://news.ycombinator.com/item?id=28683171 https://news.ycombinator.com/item?id=28683171