6 ms·
The changes to handling strings and Python data types are welcome. However I am curious on how Arrow beats NumPy on regular ints and floats. For the last 10 y
by sireat 4y ago
The changes to handling strings and Python data types are welcome.
However I am curious on how Arrow beats NumPy on regular ints and floats.
For the last 10 years I've been under impression that int and float columns in Pandas are basically NumPy ndarrays with extra methods.
Then NumPy ndarrays are basically C arrays with well defined vector operations which are often trivially parallelizable.
So how does Arrow beat Numpy when calculating
mean (int64) 2.03 ms 1.11 ms 1.8x
mean (float64) 3.56 ms 1.73 ms 2.1x
What is the trick?
- fred123 4y agoSIMD, multithreading, better handling of NaN, maybe other factors.
- ddejohn 4y agoNumpy uses SIMD too.
- sireat 4y agoExactly why I asked, Numpy is already pretty optimized on ..ahem.. numerical operations. Maybe some optimization on multithreading Still I am reasonably sure Numpy utilizes multithreading already. https://in.nau.edu/arc/parallelism-in-python https://in.nau.edu/arc/parallelism-in-python And here we assume there are no NaNs. So something else.
- kylebarron 4y agoMaybe due to SIMD?
- ddejohn 4y agoNumpy uses SIMD too.