6 ms·

Neanderthal vs. ND4J – Native Performance, Java and CPU

102 points by dragandj 8y ago

dragandj 8y ago
Author here. I'm open for discussion. The source is in the text, but conveniently accessible here: https://github.com/uncomplicate/neanderthal/blob/master/examples/benchmarks/src/benchmarks/neanderthal_vs_nd4j.clj https://github.com/uncomplicate/neanderthal/blob/master/exam...
- gok 8y ago
  How about running these benchmarks through a time profiler so you can see where the time is actually going?
  - dragandj 8y ago
    Sure! What exactly are you interested in?
    - gok 8y ago
      Your hypothesis is that MKL is taking the same amount of time and FFI overhead is the problem. A profiler could confirm your guess about MKL and tell you which part of the overhead is taking the time.
      - dragandj 8y ago
        I have to leave some exercises to the reader though ;)
- thom 8y ago
  Huge thanks for Neanderthal and Bayadera! There is stuff that is basically impossible to run in Stan that becomes feasible on a laptop with a GPU with Bayadera, and the fact that I can do all this in Clojure without having to migrate to Python or worse R has been a great boon.
  - dragandj 8y ago
    Thanks for the thumbs up!
- blueyes 8y ago
  TL/DR In Nd4j, historically, mmuli returns an F-ordered array. If you specify C-ordered result, you'll have F->C conversion underneath, and C -> F assign takes time. ND4J returns the `mmuli` result in F order, and DL$J is aware of that. We design our algorithms around this. It is possible to get the `mmuli` result in C order, but it'll cause an additional conversion operation call, which is expensive. That's exactly what Dragan did. So on one hand - he's right: in the CCC case, Neanderthal is faster, because he doesn't have to do the F->C conversion of the result. We make mmuli always return F because of Cuda. The cuBLAS implementation has no option for C ordered output. `mmuli` CCC always introduces a dup for us. We expect the gemm result to always be F, so if result is not F, each operation allocates tempResult array, F ordered, and does result.assign(tempResult).
  - dragandj 8y ago
    Neanderthal handles C order in cuBLAS without problems, though. Thanks for the explanation. I used the code that ND4J guys provided, assuming that they'd do the right thing. Also read my response to crockpotveggies: provide the ND4J code that you think is optimal, and ND4J and Neanderthal results on your machine, and I'll be happy to write a follow up.
    - blueyes 8y ago
      That's great, but that's not the essential point for us. with cuBLAS, there is no room for an output layout argument. transpose is the only option. cublasStatus_t cublasSgemm(cublasHandle_t handle, cublasOperation_t transa, cublasOperation_t transb, int m, int n, int k, const float alpha, const float A, int lda, const float B, int ldb, const float beta, float *C, int ldc)
- treo 8y ago
  Awesome! I didn't expect you to write that blog post so soon. It will be interesting to find out where the differences (beyond the 'f' ordering thing) are that make Neanderthal so fast, since when I originally started the comparison, both basically boiled down to calling MKL through JNI. For completeness sake, could you also provide some information about your machine specs and operating system (and if on linux your glibc and kernel version)?
- wmeddie 8y ago
  Is there a guide for getting neanderthal running with MKL on Windows? I tried my best but I'm not smart enough to get passed the require statement.
  - dragandj 8y ago
    Yes: https://neanderthal.uncomplicate.org/articles/getting_started.html#the-native-library-used-by-neanderthals-native-engine https://neanderthal.uncomplicate.org/articles/getting_starte... And just to be sure, choose option 2 under that heading (add the folder with appropriate dlls to PATH)
vonnik 8y ago
For anyone interested in looking at or running ND4J benchmarks, these links may be useful. https://github.com/treo/benchmarking_nd4j/tree/master/src/main/java/com/example/neanderthal https://github.com/treo/benchmarking_nd4j/tree/master/src/ma... deeplearning4j.org/native http://deeplearning4j.org/workspaces http://deeplearning4j.org/workspaces We have our own garbage collection as well as native config and off heap memory management. We use this https://github.com/deeplearning4j/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-backend-impls/nd4j-native-platform/pom.xml#L22 https://github.com/deeplearning4j/deeplearning4j/blob/master... to specify use of MKL.
crockpotveggies 8y ago
DL4J contributor here. I spoke with our team so the differences in performance are likely explained by array ordering. ND4J, for good reason, requires F ordering because of limitations in cuBLAS. While I haven't had the opportunity to closely examine the Neanderthal comparison (I'm also not a clojure user), the likely explanation is that there's an implicit ordering impacting this. Deeplearning4j is written around F ordering and ND4J supports this. Admittedly, our ordering API is not obvious to the average user. Here's an example test you can run yourself that demonstrates ordering: https://gist.github.com/raver119/92b615704ca1bf169aa23a6a6e7d9880 https://gist.github.com/raver119/92b615704ca1bf169aa23a6a6e7... o.n.i.TensorFlowImportTest - Orders: CCC; Time: 11532 ns; o.n.i.TensorFlowImportTest - Orders: CCF; Time: 2101 ns; o.n.i.TensorFlowImportTest - Orders: CFC; Time: 10202 ns; o.n.i.TensorFlowImportTest - Orders: CFF; Time: 1960 ns; o.n.i.TensorFlowImportTest - Orders: FCC; Time: 10744 ns; o.n.i.TensorFlowImportTest - Orders: FCF; Time: 1717 ns; o.n.i.TensorFlowImportTest - Orders: FFC; Time: 10097 ns; o.n.i.TensorFlowImportTest - Orders: FFF; Time: 1716 ns; We also profiled the above test and confirmed that F -> C ordering adds significant overhead. I can share screenshots if anyone is interested.
- dragandj 8y ago
  While this is a good explanation, keep in mind that: 1. This is the benchmark that you provided, so while this might be not obvious to the average user, it also seems to not be obvious to the above average user that wrote the benchmark. I was just using what you proposed, assuming that you used the right thing in your library. 2. It still does not explain how you got better performance with ND4J with the same non-optimal call, which was what started the discussion, and inspired this post. 3. Neanderthal supports both Row and Column oriented order with cuBLAS with the same performance, and won't have those problems that you mention for ND4J. I'm, of course, interested in following up on this. Please decide on what cases you'd like to compare, post the (optimal) code, and the ND4J and Neanderthal numbers that you get, and I'll respond with my comments.
  - treo 8y ago
    It looks like while converting from my benchmarking code you've dropped the 'f' when creating the resulting array. https://github.com/treo/benchmarking_nd4j/blob/master/src/main/java/com/example/neanderthal/NeanderthalComparision_1024x1024.java#L17 https://github.com/treo/benchmarking_nd4j/blob/master/src/ma... The difference is rather huge with the newer versions of nd4j. While the numbers in the following gists do not contain the measurements I took for neanderthal, they do contain the numbers that I got for ND4J. Without f ordering: https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff908 https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff90... With f ordering: https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b04 https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b0... As you can see something happened in the time between the 0.4 release (I took that as the comparison point since that was when I ran my own benchmarks the last time) and the 0.9.1 release that introduced additional overhead. Originally I planned to create my own write-up on this, but I wanted to first to find out what happened there. Given that ND4J is mainly used inside of DL4J and the matrix sizes it is used with usually are rather large, the performance overhead difference that I've observed there for tiny multiplications isn't necessarily that bad, as the newer version performs much better on larger matrices.
shoyer 8y ago
> I am almost sure that both would be faster than Numpy; that would be a good comparison. This seems a little unfair :). I’m pretty sure that if you’re using NumPy linked against MKL, it would be exactly as fast as Neanderthal running on MKL. Matrix multiplication benchmarks themselves just aren’t that interesting when all they are doing is testing an underlying library.
zz34 8y ago
The DL4J folks are a joke. They make a ton of outrageous claims then and they tell you you're getting it all wrong when you try to call them on it. I know they need to justify their nonsense so SkyMind can sucker someone into an acquisition, but they're not fooling anyone. At best they can claim it's as fast as what one person built in their spare time....

102 points by dragandj 8y ago

Author here. I'm open for discussion. The source is in the text, but conveniently accessible here: https://github.com/uncomplicate/neanderthal/blob/master/examples/benchmarks/src/benchmarks/neanderthal_vs_nd4j.clj https://github.com/uncomplicate/neanderthal/blob/master/exam...

gok 8y ago
How about running these benchmarks through a time profiler so you can see where the time is actually going?
- dragandj 8y ago
  Sure! What exactly are you interested in?
  - gok 8y ago
    Your hypothesis is that MKL is taking the same amount of time and FFI overhead is the problem. A profiler could confirm your guess about MKL and tell you which part of the overhead is taking the time.
    - dragandj 8y ago
      I have to leave some exercises to the reader though ;)
thom 8y ago
Huge thanks for Neanderthal and Bayadera! There is stuff that is basically impossible to run in Stan that becomes feasible on a laptop with a GPU with Bayadera, and the fact that I can do all this in Clojure without having to migrate to Python or worse R has been a great boon.
- dragandj 8y ago
  Thanks for the thumbs up!
blueyes 8y ago
TL/DR In Nd4j, historically, mmuli returns an F-ordered array. If you specify C-ordered result, you'll have F->C conversion underneath, and C -> F assign takes time. ND4J returns the `mmuli` result in F order, and DL$J is aware of that. We design our algorithms around this. It is possible to get the `mmuli` result in C order, but it'll cause an additional conversion operation call, which is expensive. That's exactly what Dragan did. So on one hand - he's right: in the CCC case, Neanderthal is faster, because he doesn't have to do the F->C conversion of the result. We make mmuli always return F because of Cuda. The cuBLAS implementation has no option for C ordered output. `mmuli` CCC always introduces a dup for us. We expect the gemm result to always be F, so if result is not F, each operation allocates tempResult array, F ordered, and does result.assign(tempResult).
- dragandj 8y ago
  Neanderthal handles C order in cuBLAS without problems, though. Thanks for the explanation. I used the code that ND4J guys provided, assuming that they'd do the right thing. Also read my response to crockpotveggies: provide the ND4J code that you think is optimal, and ND4J and Neanderthal results on your machine, and I'll be happy to write a follow up.
  - blueyes 8y ago
    That's great, but that's not the essential point for us. with cuBLAS, there is no room for an output layout argument. transpose is the only option. cublasStatus_t cublasSgemm(cublasHandle_t handle, cublasOperation_t transa, cublasOperation_t transb, int m, int n, int k, const float alpha, const float A, int lda, const float B, int ldb, const float beta, float *C, int ldc)
treo 8y ago
Awesome! I didn't expect you to write that blog post so soon. It will be interesting to find out where the differences (beyond the 'f' ordering thing) are that make Neanderthal so fast, since when I originally started the comparison, both basically boiled down to calling MKL through JNI. For completeness sake, could you also provide some information about your machine specs and operating system (and if on linux your glibc and kernel version)?
wmeddie 8y ago
Is there a guide for getting neanderthal running with MKL on Windows? I tried my best but I'm not smart enough to get passed the require statement.
- dragandj 8y ago
  Yes: https://neanderthal.uncomplicate.org/articles/getting_started.html#the-native-library-used-by-neanderthals-native-engine https://neanderthal.uncomplicate.org/articles/getting_starte... And just to be sure, choose option 2 under that heading (add the folder with appropriate dlls to PATH)

vonnik 8y ago

For anyone interested in looking at or running ND4J benchmarks, these links may be useful. https://github.com/treo/benchmarking_nd4j/tree/master/src/main/java/com/example/neanderthal https://github.com/treo/benchmarking_nd4j/tree/master/src/ma... deeplearning4j.org/native http://deeplearning4j.org/workspaces http://deeplearning4j.org/workspaces We have our own garbage collection as well as native config and off heap memory management. We use this https://github.com/deeplearning4j/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-backend-impls/nd4j-native-platform/pom.xml#L22 https://github.com/deeplearning4j/deeplearning4j/blob/master... to specify use of MKL.

crockpotveggies 8y ago

DL4J contributor here. I spoke with our team so the differences in performance are likely explained by array ordering. ND4J, for good reason, requires F ordering because of limitations in cuBLAS. While I haven't had the opportunity to closely examine the Neanderthal comparison (I'm also not a clojure user), the likely explanation is that there's an implicit ordering impacting this. Deeplearning4j is written around F ordering and ND4J supports this. Admittedly, our ordering API is not obvious to the average user. Here's an example test you can run yourself that demonstrates ordering: https://gist.github.com/raver119/92b615704ca1bf169aa23a6a6e7d9880 https://gist.github.com/raver119/92b615704ca1bf169aa23a6a6e7... o.n.i.TensorFlowImportTest - Orders: CCC; Time: 11532 ns; o.n.i.TensorFlowImportTest - Orders: CCF; Time: 2101 ns; o.n.i.TensorFlowImportTest - Orders: CFC; Time: 10202 ns; o.n.i.TensorFlowImportTest - Orders: CFF; Time: 1960 ns; o.n.i.TensorFlowImportTest - Orders: FCC; Time: 10744 ns; o.n.i.TensorFlowImportTest - Orders: FCF; Time: 1717 ns; o.n.i.TensorFlowImportTest - Orders: FFC; Time: 10097 ns; o.n.i.TensorFlowImportTest - Orders: FFF; Time: 1716 ns; We also profiled the above test and confirmed that F -> C ordering adds significant overhead. I can share screenshots if anyone is interested.

dragandj 8y ago
While this is a good explanation, keep in mind that: 1. This is the benchmark that you provided, so while this might be not obvious to the average user, it also seems to not be obvious to the above average user that wrote the benchmark. I was just using what you proposed, assuming that you used the right thing in your library. 2. It still does not explain how you got better performance with ND4J with the same non-optimal call, which was what started the discussion, and inspired this post. 3. Neanderthal supports both Row and Column oriented order with cuBLAS with the same performance, and won't have those problems that you mention for ND4J. I'm, of course, interested in following up on this. Please decide on what cases you'd like to compare, post the (optimal) code, and the ND4J and Neanderthal numbers that you get, and I'll respond with my comments.
- treo 8y ago
  It looks like while converting from my benchmarking code you've dropped the 'f' when creating the resulting array. https://github.com/treo/benchmarking_nd4j/blob/master/src/main/java/com/example/neanderthal/NeanderthalComparision_1024x1024.java#L17 https://github.com/treo/benchmarking_nd4j/blob/master/src/ma... The difference is rather huge with the newer versions of nd4j. While the numbers in the following gists do not contain the measurements I took for neanderthal, they do contain the numbers that I got for ND4J. Without f ordering: https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff908 https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff90... With f ordering: https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b04 https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b0... As you can see something happened in the time between the 0.4 release (I took that as the comparison point since that was when I ran my own benchmarks the last time) and the 0.9.1 release that introduced additional overhead. Originally I planned to create my own write-up on this, but I wanted to first to find out what happened there. Given that ND4J is mainly used inside of DL4J and the matrix sizes it is used with usually are rather large, the performance overhead difference that I've observed there for tiny multiplications isn't necessarily that bad, as the newer version performs much better on larger matrices.

shoyer 8y ago

> I am almost sure that both would be faster than Numpy; that would be a good comparison. This seems a little unfair :). I’m pretty sure that if you’re using NumPy linked against MKL, it would be exactly as fast as Neanderthal running on MKL. Matrix multiplication benchmarks themselves just aren’t that interesting when all they are doing is testing an underlying library.

zz34 8y ago

The DL4J folks are a joke. They make a ton of outrageous claims then and they tell you you're getting it all wrong when you try to call them on it. I know they need to justify their nonsense so SkyMind can sucker someone into an acquisition, but they're not fooling anyone. At best they can claim it's as fast as what one person built in their spare time....