6 ms·

All of the operations in the plots are being calculated using tiny arrays. Even 128 elements is pretty small. Additionally, I'd be interested to know how they

All of the operations in the plots are being calculated using tiny arrays. Even 128 elements is pretty small. Additionally, I'd be interested to know how they compiled NumPy, and which BLAS implementation they linked against.

jcranmer 11y ago
The graphs are a travesty of detail, and I suspect that the more you poke it, the worse the results get. There's a definite downward trend in speedup as you increase vector size, and that is completely different from what you would naively expect (at larger vector sizes, you spend more time in the embarrassingly-parallelizable kernel)--so this suggests that the baseline itself is not scalar code but vector code. So what the graphs end up highlighting is that the CPython (NumPy?) has decent overhead on tiny calls that don't really matter for gross performance. As you say, 128 elements is tiny: that's where I'd consider starting a graph, not ending it. At 4 elements, you're looking at about 40 clock cycles to do a simple vector-add in the scalar case (I think x86 L1 cache hits are ~3 clock cycles and there are two load/store units; if these numbers are wrong, my estimate is off). This also means that there's absolutely no measurement of the overhead of the JIT autovectorizing operation (!), which is quite significant because the usual resistance to adding autovectorizers in JITs [1] is that they're way too slow for the speedup they give. [1] As far as I'm aware, the only other JIT of note that includes an autovectorizer is the Hotspot JVM. The approach taken by other people (e.g., .NET CLR) is to effectively expose a SIMD primitive in the bytecode and get compilers to target those via static autovectorization instead of autovectorizing at runtime.
- lqdc13 11y ago
  The demo should really be 128 thousand elements or 128 million. 128 elements is still useful for when you have an inner loop that has a small vector operation. Most of the time this is not the case though.
- mistercow 11y ago
  > This also means that there's absolutely no measurement of the overhead of the JIT autovectorizing operation I definitely agree that overhead should be included in the article, you wouldn't expect to see it in those graphs, because it's a once off, right?
- repsilat 11y ago
  > at larger vector sizes, you spend more time in the embarrassingly-parallelizable kernel Vectors are only fixed-width, so even though it is an "embarrassingly parallel" problem, you still only expect it to asymptote towards a fixed speedup. Moreover, the bigger the matrix, the more likely you are to see cache size and memory bandwidth effects in your performance numbers, meaning the SIMD could be less of a win in the limit.
zurn 11y ago
My guess is that since Numpy with CPython is all about calling into optimized native-code kernels, there's not much PyPy can do to win in big arrays. With small vectors the C FFI overhead would dominate on CPython and PyPy has an advantage. This would be more apparent if they provided also graphs with absolute numbers and not just speedups relative to classic Numpy.
- leecb 11y ago
  They have decided to rewrite the C parts of NumPy in RPython, essentially working towards a pure Python version of Numpy. There are a lot more details sprinkled across several pages: http://morepypy.blogspot.mx/2013/11/numpy-status-update.html http://morepypy.blogspot.mx/2013/11/numpy-status-update.html http://pypy.org/numpydonate.html http://pypy.org/numpydonate.html
  - fulafel 11y ago
    Yes, so PyPy can win on small arrays because there is no per-call C FFI overhead in calling into RPython-Numpy. The smaller the operations, the greater the win over CPython.

jcranmer 11y ago

The graphs are a travesty of detail, and I suspect that the more you poke it, the worse the results get. There's a definite downward trend in speedup as you increase vector size, and that is completely different from what you would naively expect (at larger vector sizes, you spend more time in the embarrassingly-parallelizable kernel)--so this suggests that the baseline itself is not scalar code but vector code. So what the graphs end up highlighting is that the CPython (NumPy?) has decent overhead on tiny calls that don't really matter for gross performance. As you say, 128 elements is tiny: that's where I'd consider starting a graph, not ending it. At 4 elements, you're looking at about 40 clock cycles to do a simple vector-add in the scalar case (I think x86 L1 cache hits are ~3 clock cycles and there are two load/store units; if these numbers are wrong, my estimate is off). This also means that there's absolutely no measurement of the overhead of the JIT autovectorizing operation (!), which is quite significant because the usual resistance to adding autovectorizers in JITs [1] is that they're way too slow for the speedup they give. [1] As far as I'm aware, the only other JIT of note that includes an autovectorizer is the Hotspot JVM. The approach taken by other people (e.g., .NET CLR) is to effectively expose a SIMD primitive in the bytecode and get compilers to target those via static autovectorization instead of autovectorizing at runtime.

lqdc13 11y ago
The demo should really be 128 thousand elements or 128 million. 128 elements is still useful for when you have an inner loop that has a small vector operation. Most of the time this is not the case though.
mistercow 11y ago
> This also means that there's absolutely no measurement of the overhead of the JIT autovectorizing operation I definitely agree that overhead should be included in the article, you wouldn't expect to see it in those graphs, because it's a once off, right?
repsilat 11y ago
> at larger vector sizes, you spend more time in the embarrassingly-parallelizable kernel Vectors are only fixed-width, so even though it is an "embarrassingly parallel" problem, you still only expect it to asymptote towards a fixed speedup. Moreover, the bigger the matrix, the more likely you are to see cache size and memory bandwidth effects in your performance numbers, meaning the SIMD could be less of a win in the limit.

zurn 11y ago

My guess is that since Numpy with CPython is all about calling into optimized native-code kernels, there's not much PyPy can do to win in big arrays. With small vectors the C FFI overhead would dominate on CPython and PyPy has an advantage. This would be more apparent if they provided also graphs with absolute numbers and not just speedups relative to classic Numpy.

leecb 11y ago
They have decided to rewrite the C parts of NumPy in RPython, essentially working towards a pure Python version of Numpy. There are a lot more details sprinkled across several pages: http://morepypy.blogspot.mx/2013/11/numpy-status-update.html http://morepypy.blogspot.mx/2013/11/numpy-status-update.html http://pypy.org/numpydonate.html http://pypy.org/numpydonate.html
- fulafel 11y ago
  Yes, so PyPy can win on small arrays because there is no per-call C FFI overhead in calling into RPython-Numpy. The smaller the operations, the greater the win over CPython.