4 ms·

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external lib

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-world programs doing matrix multiplication in pure Python. But it can show the relative speed of pure Python at purely computational tasks.

p-e-w 3y ago
> But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.
- scotty79 3y ago
  Every time you use a for loop, addition or multiplication operators, array indexing, assingment to numerical variable in Python, you are using exactly what is benchmarked in matmul.
- YetAnotherNick 3y ago
  Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.
  - mratsim 3y ago
    Pandas/Numpy uses a lot of C and Scipy a lot of Fortran
    - YetAnotherNick 3y ago
      No one said they don't. Just there still are and always will be parts which are computationally heavy but written in python. e.g. pandas map is 300x slower than numpy map[1] for the same operation. But not everything could be vectorized and even if some complex function could be vectorized it will involve multiple passes over memory for the same array. [1]: https://www.exxactcorp.com/blog/Deep-Learning/how-to-speed-up-python-pandas-by-over-300x https://www.exxactcorp.com/blog/Deep-Learning/how-to-speed-u...
- gpderetta 3y ago
  Nobody uses python for numerical tasks because it is atrociously slow at those tasks. This benchmark reiterates that.
lifthrasiir 3y ago
> Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. If this were the only concern, it should be valid to create a blob of binary and call that function for the optimal performance. (Python's ctypes makes this very easy, for example.) So you want an idiomatic solution instead, and Numpy for matrix computation is considered idiomatic in Python, even more than the pure Python code.
- tsimionescu 3y ago
  No, creating a binary blob is not a realistic option, and it's still not a Python/Java/C++ whatever solution. It would be handwritten assembly that you use the FFI facilities to call into. So, it's not fair game in any honest benchmark. And even if using a C library is idiomatic Python, it still has no place in a language benchmark. It's a C library, not a Python implementation.
  - lifthrasiir 3y ago
    > No, creating a binary blob is not a realistic option, and it's still not a Python/Java/C++ whatever solution. Once again, "realistic" is subjective and I would say no "realistic" user will try to multiply arbitrarily-sized matrices in pure Python. (I can see small enough matrices, like 3x3 or 4x4, might be different.) And... > So, it's not fair game in any honest benchmark. And even if using a C library is idiomatic Python, it still has no place in a language benchmark. It's a C library, not a Python implementation. ...you have correctly figured out that it's unfair for anyone using your (vague) definition of "realistic". It's also true that your proposal is also unfair for anyone using my definition of "idiomatic" however. I can try to defend my definition with quantative arguments, but I don't feel like doing so. In fact I tend to ignore most "language benchmarks" because it is virtually impossible to make them reasonably fair. This one is no exception. Best "language benchmarks" tend to be more like language showcases with useful commentaries, there will be no single winner but you will get a good sense of pros and cons of each language-implementation-strategy combination. They are generally not advertised as "benchmarks", of course.

p-e-w 3y ago

> But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.

scotty79 3y ago
Every time you use a for loop, addition or multiplication operators, array indexing, assingment to numerical variable in Python, you are using exactly what is benchmarked in matmul.
YetAnotherNick 3y ago
Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.
- mratsim 3y ago
  Pandas/Numpy uses a lot of C and Scipy a lot of Fortran
  - YetAnotherNick 3y ago
    No one said they don't. Just there still are and always will be parts which are computationally heavy but written in python. e.g. pandas map is 300x slower than numpy map[1] for the same operation. But not everything could be vectorized and even if some complex function could be vectorized it will involve multiple passes over memory for the same array. [1]: https://www.exxactcorp.com/blog/Deep-Learning/how-to-speed-up-python-pandas-by-over-300x https://www.exxactcorp.com/blog/Deep-Learning/how-to-speed-u...
gpderetta 3y ago
Nobody uses python for numerical tasks because it is atrociously slow at those tasks. This benchmark reiterates that.

lifthrasiir 3y ago

> Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. If this were the only concern, it should be valid to create a blob of binary and call that function for the optimal performance. (Python's ctypes makes this very easy, for example.) So you want an idiomatic solution instead, and Numpy for matrix computation is considered idiomatic in Python, even more than the pure Python code.

tsimionescu 3y ago
No, creating a binary blob is not a realistic option, and it's still not a Python/Java/C++ whatever solution. It would be handwritten assembly that you use the FFI facilities to call into. So, it's not fair game in any honest benchmark. And even if using a C library is idiomatic Python, it still has no place in a language benchmark. It's a C library, not a Python implementation.
- lifthrasiir 3y ago
  > No, creating a binary blob is not a realistic option, and it's still not a Python/Java/C++ whatever solution. Once again, "realistic" is subjective and I would say no "realistic" user will try to multiply arbitrarily-sized matrices in pure Python. (I can see small enough matrices, like 3x3 or 4x4, might be different.) And... > So, it's not fair game in any honest benchmark. And even if using a C library is idiomatic Python, it still has no place in a language benchmark. It's a C library, not a Python implementation. ...you have correctly figured out that it's unfair for anyone using your (vague) definition of "realistic". It's also true that your proposal is also unfair for anyone using my definition of "idiomatic" however. I can try to defend my definition with quantative arguments, but I don't feel like doing so. In fact I tend to ignore most "language benchmarks" because it is virtually impossible to make them reasonably fair. This one is no exception. Best "language benchmarks" tend to be more like language showcases with useful commentaries, there will be no single winner but you will get a good sense of pros and cons of each language-implementation-strategy combination. They are generally not advertised as "benchmarks", of course.