6 ms·

While this is a good explanation, keep in mind that: 1. This is the benchmark that you provided, so while this might be not obvious to the average user, it als

While this is a good explanation, keep in mind that: 1. This is the benchmark that you provided, so while this might be not obvious to the average user, it also seems to not be obvious to the above average user that wrote the benchmark. I was just using what you proposed, assuming that you used the right thing in your library. 2. It still does not explain how you got better performance with ND4J with the same non-optimal call, which was what started the discussion, and inspired this post. 3. Neanderthal supports both Row and Column oriented order with cuBLAS with the same performance, and won't have those problems that you mention for ND4J. I'm, of course, interested in following up on this. Please decide on what cases you'd like to compare, post the (optimal) code, and the ND4J and Neanderthal numbers that you get, and I'll respond with my comments.

treo 8y ago
It looks like while converting from my benchmarking code you've dropped the 'f' when creating the resulting array. https://github.com/treo/benchmarking_nd4j/blob/master/src/main/java/com/example/neanderthal/NeanderthalComparision_1024x1024.java#L17 https://github.com/treo/benchmarking_nd4j/blob/master/src/ma... The difference is rather huge with the newer versions of nd4j. While the numbers in the following gists do not contain the measurements I took for neanderthal, they do contain the numbers that I got for ND4J. Without f ordering: https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff908 https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff90... With f ordering: https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b04 https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b0... As you can see something happened in the time between the 0.4 release (I took that as the comparison point since that was when I ran my own benchmarks the last time) and the 0.9.1 release that introduced additional overhead. Originally I planned to create my own write-up on this, but I wanted to first to find out what happened there. Given that ND4J is mainly used inside of DL4J and the matrix sizes it is used with usually are rather large, the performance overhead difference that I've observed there for tiny multiplications isn't necessarily that bad, as the newer version performs much better on larger matrices.
- dragandj 8y ago
  You're right. In that particular case, ND4J comes to Neanderthal's speed. But only in that particular case; and even then ND4J is still not faster than Neanderthal. My initial quest was to find out whether ND4J can be faster than Neanderthal, and I still couldn't find a case where it is. Although, to my defense, the option in question here is very poorly documented. I've found the ND4J tutorial page where it's mentioned, and even after re-reading the sentence multiple times, I still do not connect its description to what it (seems to) actually do. It also does not mention that it affects computation speed. Anyway, I'm looking forward to reading your detailed analysis, and especially seeing your Neanderthal numbers.
  - treo 8y ago
    Do you have any pointer on how you've profiled Neanderthal during development? When I originally set out to compare ND4J and Neanderthal, I've ran into the issue that I bottomed out at: they basically both call MKL (or Openblas) for BLAS operations.
  - agibsonccc 8y ago
    Fair point we are fixing now: https://github.com/deeplearning4j/deeplearning4j-docs/issues/83 https://github.com/deeplearning4j/deeplearning4j-docs/issues... We will be sending out a doc for this by next week with these updates. Thanks a lot for playing ball here. Beyond that, can you clarify what you mean? Do you mean just the gemm op? For that, that's the only case that mattered for us. We will be documenting the what/how/why of this in our docs. Beyond that, I'm not convinced the libraries are directly comparable when it comes to the sheer scope of the libraries to each other. You're treating nd4j as a gemm library rather than a fully fledged numpy/tensorflow with hundreds of ops and support for things you would likely have no interest in building. A big reason I built nd4j was to solve the general use case of building a tensor library for deep learning, not just a gemm library. Beyond that - I'll give you props for what you built. There's always lessons to learn when comparing libraries and making sure the numbers match. Our target isn't you though, it's the likes of google,facebook, and co and tackling the scope of tasks they are. That being said - could we spend some time on docs? Heck yeah we should. At most we have java doc and examples. We tend to help people as much as we can when profiling. Could we manage it better? Yes for sure. That's partially why we moved dl4j to the eclipse foundation to get more 3rd party contributions and build a better governance setup. Will it take time for all of this to evolve? Oh yeah most definitely. No project is perfect and always has things it could improve on. Anyways - let's be clear here. You're a one man shop who built an amazingly fast library that scratches your own itch for a very specific set of use cases. We're a company and community tackling a wider breadth of tasks and trying to focus more on serving customers and adding odd things like different kinds of serialization, spark interop,.. etc. We benefit from doing these comparisons and it forces us to document things better that we normally don't pay attention to. This little exercise is good for us. As mentioned, we will document the limitations a bit better but we will make sure to cover other topics like allocation and the like as well as the blas interface. Positive change has come out of this and I'd like to thank you for the work you put in. We will make sure to re run some of the comaprisons on our side.

treo 8y ago

It looks like while converting from my benchmarking code you've dropped the 'f' when creating the resulting array. https://github.com/treo/benchmarking_nd4j/blob/master/src/main/java/com/example/neanderthal/NeanderthalComparision_1024x1024.java#L17 https://github.com/treo/benchmarking_nd4j/blob/master/src/ma... The difference is rather huge with the newer versions of nd4j. While the numbers in the following gists do not contain the measurements I took for neanderthal, they do contain the numbers that I got for ND4J. Without f ordering: https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff908 https://gist.github.com/treo/1fab39f213da26255cf4f75e383ff90... With f ordering: https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b04 https://gist.github.com/treo/94fe92c9417b5c8b24baa12924a35b0... As you can see something happened in the time between the 0.4 release (I took that as the comparison point since that was when I ran my own benchmarks the last time) and the 0.9.1 release that introduced additional overhead. Originally I planned to create my own write-up on this, but I wanted to first to find out what happened there. Given that ND4J is mainly used inside of DL4J and the matrix sizes it is used with usually are rather large, the performance overhead difference that I've observed there for tiny multiplications isn't necessarily that bad, as the newer version performs much better on larger matrices.

dragandj 8y ago
You're right. In that particular case, ND4J comes to Neanderthal's speed. But only in that particular case; and even then ND4J is still not faster than Neanderthal. My initial quest was to find out whether ND4J can be faster than Neanderthal, and I still couldn't find a case where it is. Although, to my defense, the option in question here is very poorly documented. I've found the ND4J tutorial page where it's mentioned, and even after re-reading the sentence multiple times, I still do not connect its description to what it (seems to) actually do. It also does not mention that it affects computation speed. Anyway, I'm looking forward to reading your detailed analysis, and especially seeing your Neanderthal numbers.
- treo 8y ago
  Do you have any pointer on how you've profiled Neanderthal during development? When I originally set out to compare ND4J and Neanderthal, I've ran into the issue that I bottomed out at: they basically both call MKL (or Openblas) for BLAS operations.
- agibsonccc 8y ago
  Fair point we are fixing now: https://github.com/deeplearning4j/deeplearning4j-docs/issues/83 https://github.com/deeplearning4j/deeplearning4j-docs/issues... We will be sending out a doc for this by next week with these updates. Thanks a lot for playing ball here. Beyond that, can you clarify what you mean? Do you mean just the gemm op? For that, that's the only case that mattered for us. We will be documenting the what/how/why of this in our docs. Beyond that, I'm not convinced the libraries are directly comparable when it comes to the sheer scope of the libraries to each other. You're treating nd4j as a gemm library rather than a fully fledged numpy/tensorflow with hundreds of ops and support for things you would likely have no interest in building. A big reason I built nd4j was to solve the general use case of building a tensor library for deep learning, not just a gemm library. Beyond that - I'll give you props for what you built. There's always lessons to learn when comparing libraries and making sure the numbers match. Our target isn't you though, it's the likes of google,facebook, and co and tackling the scope of tasks they are. That being said - could we spend some time on docs? Heck yeah we should. At most we have java doc and examples. We tend to help people as much as we can when profiling. Could we manage it better? Yes for sure. That's partially why we moved dl4j to the eclipse foundation to get more 3rd party contributions and build a better governance setup. Will it take time for all of this to evolve? Oh yeah most definitely. No project is perfect and always has things it could improve on. Anyways - let's be clear here. You're a one man shop who built an amazingly fast library that scratches your own itch for a very specific set of use cases. We're a company and community tackling a wider breadth of tasks and trying to focus more on serving customers and adding odd things like different kinds of serialization, spark interop,.. etc. We benefit from doing these comparisons and it forces us to document things better that we normally don't pay attention to. This little exercise is good for us. As mentioned, we will document the limitations a bit better but we will make sure to cover other topics like allocation and the like as well as the blas interface. Positive change has come out of this and I'd like to thank you for the work you put in. We will make sure to re run some of the comaprisons on our side.