5 ms·

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimi

by p-e-w 3y ago

What is this supposed to demonstrate? There appears to be roughly the same code structure, ported to every language, while for some languages, arbitrary optimizations are introduced (such as using `array` instead of `list` in Python). But nobody working in Python uses matrix multiplication code written in Python. They use NumPy, which is a de facto standard library for people working in the relevant fields. It's as much part of "Python" as list comprehensions. Without taking such real-world conventions into account, such comparisons say essentially nothing about the languages involved (and their all-important ecosystems).

morepedantic 3y ago
It demonstrates that Python needs libraries like NumPy. Few problems are more heavily optimized than matrix multiplication in practice, so comparing matrix multiplication benchmarks across languages with NumPy is not representative of real-world performance for most programming use cases. It also means that adding performance to an existing Python program requires dropping into a different language, which is not only complicated, but also requires engineers capable in both Python and C (or similar).
- lifthrasiir 3y ago
  You don't need any C knowledge to use numpy. In fact, its conceptual similarity with Matlab is possibly the single most important reason for its popularity. Many other problems do need specialized treatments that would indeed require other languages, but numpy is not a good counterexample.
  - jakobnissen 3y ago
    You're missing the point. The point is that, for any application, Python needs an underlying C library to be fast. So if you need to solve problems where no such library exists, Python is slow. In other words, Python IS slow, but it can call fast code written in other languages.
    - lifthrasiir 3y ago
      That's true but irrelevant here, especially given the original claim: > adding performance to an existing Python program requires dropping into a different language ...is demonstrably false for a significant class of programs that can be rewritten into the array paradigm. The benchmark should have picked other numerical problem to avoid this issue. The Computer Language Benchmarks Game, for example, uses the `n-body` problem for this purpose.
- doix 3y ago
  > It also means that adding performance to an existing Python program requires dropping into a different language, which is not only complicated, but also requires engineers capable in both Python and C (or similar). It's actually not that bad. I think it's part of the reason Python became so popular, it's fairly easy to write C code and expose it via python.
- p-e-w 3y ago
  > It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also means that adding performance to an existing Python program requires dropping into a different language As stated above, this applies to all languages. BLAS routines used for serious numerical work are hand-vectorized Assembly fine-tuned for each processor architecture, written by a few hyper-experts who do nothing else. Nobody who needs performant matrix multiplication from C thinks "hey, let me just write two nested loops".
  - dsharlet 3y ago
    This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all to beat MKL, etc... if you are using a language like C++. If you are using python, you have no chance. It isn't even necessarily that different from a few nested loops. Clang is pretty damn good at autovectorizing, you just have to be a little careful about how you write the code.
    - lifthrasiir 3y ago
      You need to have enough experience to be able to be a little careful though. This is generally true for most languages, loop unrolling works even better in Python for example, but many Python programmers aren't even aware of this possibility.
    - p-e-w 3y ago
      > If you are using python, you have no chance. Of course you do. Every special-case multiplication algorithm you might need already has an optimized implementation that you can just `pip install`, and move on with what you're actually working on. The whole scientific computing world runs on Python. Straightforward numerics code using NumPy tends to murder C/C++ code in regard to performance, unless that code is written by people who make a living hand-optimizing computational routines.
- RHSeeger 3y ago
  > It demonstrates that Python needs libraries like NumPy. You need libraries to do _anything_ in Python. It's interpreted, so literally any call you make in Python will eventually make it back to something written in a compiled language (like a call to NumPy commands).
tsimionescu 3y ago
Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-world programs doing matrix multiplication in pure Python. But it can show the relative speed of pure Python at purely computational tasks.
- p-e-w 3y ago
  > But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.
  - scotty79 3y ago
    Every time you use a for loop, addition or multiplication operators, array indexing, assingment to numerical variable in Python, you are using exactly what is benchmarked in matmul.
  - YetAnotherNick 3y ago
    Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.
    - mratsim 3y ago
      Pandas/Numpy uses a lot of C and Scipy a lot of Fortran
      - 3y ago
deleted 3y ago
[deleted]
ggm 3y ago
So you'd be measuring the speed of loops over out calls to c or Fortran (nag library) or their vectorisation and only testing the cost of data representation changes
attractivechaos 3y ago
> What is this supposed to demonstrate? It is supposed to show the performance of a language when you have to implement a new algorithm in the language. This may happen if you can't find the algorithm in existing libraries. If you don't like matmul, ignore it and focus on nqueen, sudoku and bedcov.

by p-e-w 3y ago

morepedantic 3y ago

It demonstrates that Python needs libraries like NumPy. Few problems are more heavily optimized than matrix multiplication in practice, so comparing matrix multiplication benchmarks across languages with NumPy is not representative of real-world performance for most programming use cases. It also means that adding performance to an existing Python program requires dropping into a different language, which is not only complicated, but also requires engineers capable in both Python and C (or similar).

lifthrasiir 3y ago
You don't need any C knowledge to use numpy. In fact, its conceptual similarity with Matlab is possibly the single most important reason for its popularity. Many other problems do need specialized treatments that would indeed require other languages, but numpy is not a good counterexample.
- jakobnissen 3y ago
  You're missing the point. The point is that, for any application, Python needs an underlying C library to be fast. So if you need to solve problems where no such library exists, Python is slow. In other words, Python IS slow, but it can call fast code written in other languages.
  - lifthrasiir 3y ago
    That's true but irrelevant here, especially given the original claim: > adding performance to an existing Python program requires dropping into a different language ...is demonstrably false for a significant class of programs that can be rewritten into the array paradigm. The benchmark should have picked other numerical problem to avoid this issue. The Computer Language Benchmarks Game, for example, uses the `n-body` problem for this purpose.
doix 3y ago
> It also means that adding performance to an existing Python program requires dropping into a different language, which is not only complicated, but also requires engineers capable in both Python and C (or similar). It's actually not that bad. I think it's part of the reason Python became so popular, it's fairly easy to write C code and expose it via python.
p-e-w 3y ago
> It demonstrates that Python needs libraries like NumPy. People use matrix multiplication libraries (often written in Assembly) from every language if they really care about performance. That's because such libraries incorporate 100 PhD theses' worth of tricks that no individual can hope to reinvent in the course of solving another problem. There is absolutely nothing special about Python in this context. > It also means that adding performance to an existing Python program requires dropping into a different language As stated above, this applies to all languages. BLAS routines used for serious numerical work are hand-vectorized Assembly fine-tuned for each processor architecture, written by a few hyper-experts who do nothing else. Nobody who needs performant matrix multiplication from C thinks "hey, let me just write two nested loops".
- dsharlet 3y ago
  This is really overstating how hard it is to compete with matrix multiply libraries. The main reason those libraries are so big and have had so much work invested in them is their generality: they're reasonably fast for almost any kind of inputs. If you have a specific problem with constraints you can exploit (e.g. known fixed dimensions, sparsity patterns, data layouts, type conversions, etc.), it's not hard at all to beat MKL, etc... if you are using a language like C++. If you are using python, you have no chance. It isn't even necessarily that different from a few nested loops. Clang is pretty damn good at autovectorizing, you just have to be a little careful about how you write the code.
  - lifthrasiir 3y ago
    You need to have enough experience to be able to be a little careful though. This is generally true for most languages, loop unrolling works even better in Python for example, but many Python programmers aren't even aware of this possibility.
  - p-e-w 3y ago
    > If you are using python, you have no chance. Of course you do. Every special-case multiplication algorithm you might need already has an optimized implementation that you can just `pip install`, and move on with what you're actually working on. The whole scientific computing world runs on Python. Straightforward numerics code using NumPy tends to murder C/C++ code in regard to performance, unless that code is written by people who make a living hand-optimizing computational routines.
RHSeeger 3y ago
> It demonstrates that Python needs libraries like NumPy. You need libraries to do _anything_ in Python. It's interpreted, so literally any call you make in Python will eventually make it back to something written in a compiled language (like a call to NumPy commands).

tsimionescu 3y ago

Generally, the point of language benchmarks is to show how the languages compare at solving the same problem, without external libraries. Including external libraries is pointless since any language can call any library, ultimately, so at best you'd be comparing the FFI overhead. So this shouldn't be taken as "how fast does a real-world Python program do at matrix multiplication", since of course no one writes real-world programs doing matrix multiplication in pure Python. But it can show the relative speed of pure Python at purely computational tasks.

p-e-w 3y ago
> But it can show the relative speed of pure Python at purely computational tasks. But that's irrelevant if nobody uses "pure Python" for computational tasks. It's like asking "how well do these languages run on a Lisp machine from 1979?". It simply has no relevance to real-world considerations today.
- scotty79 3y ago
  Every time you use a for loop, addition or multiplication operators, array indexing, assingment to numerical variable in Python, you are using exactly what is benchmarked in matmul.
- YetAnotherNick 3y ago
  Everyone uses pure python for purely computational tasks. numpy or pytorch has far too few operation to even count as all the computational task. e.g. most of the operations of pandas is written in pure python, and at times I found using specialised libraries could give 10x improvement but with blow to developer experience compared to python.
  - mratsim 3y ago
    Pandas/Numpy uses a lot of C and Scipy a lot of Fortran
    - 3y ago

deleted 3y ago

[deleted]

ggm 3y ago

So you'd be measuring the speed of loops over out calls to c or Fortran (nag library) or their vectorisation and only testing the cost of data representation changes

attractivechaos 3y ago

> What is this supposed to demonstrate? It is supposed to show the performance of a language when you have to implement a new algorithm in the language. This may happen if you can't find the algorithm in existing libraries. If you don't like matmul, ignore it and focus on nqueen, sudoku and bedcov.