4 ms·

One common criticism of uproot is that it's not flexible when per-row computation gets complicated because for-loops in Python is too slow. For that one can eit

by moelf 2y ago

One common criticism of uproot is that it's not flexible when per-row computation gets complicated because for-loops in Python is too slow. For that one can either use Numba (when it works), or, here's the shameless plug, use Julia: https://github.com/JuliaHEP/UnROOT.jl https://github.com/JuliaHEP/UnROOT.jl Past HN discussion on Julia for particle physics: https://news.ycombinator.com/item?id=38512793 https://news.ycombinator.com/item?id=38512793

elashri 2y ago
That'a true and Julia might be a solution but I don't see the adoption happening anytime soon. But this particular problem (per row computation) have different options to tackle now in hep-python ecosystem. One approach is to leverage array programming with NumPy to vectorize operations as much as possible. By operating on entire arrays rather than looping over individual elements, significant speedups can often be achieved. Another possibility is to use a library like Awkward Array, which is designed to work with nested, variable-sized data structures. Awkward Array integrates well with uproot and provides a powerful and flexible framework for performing fast computations on i.e jagged arrays.
- moelf 2y ago
  Uproot already returns you Awkward array, so both things you mentioned are different ways of saying the same thing. The irreducible complexity of data analysis is there no matter how you do it, and "one-vector-at-a-time" sometimes feel like shoehorning (other terms people come up with include vector-style mental gymnastics). For the record, vector-style programming is great when it works, I mean Julia even has a dedicated syntax for broadcasting. I'm saying when the irreducible complexity arrives, you don't want to NOT be able to just write a for-loop Just a recent example, a double-for loop looks like this in Awkward array: https://github.com/Moelf/UnROOT_RDataFrame_MiniBenchmark/blob/acd5bdc87704ea9111c4adb1217a4bd894b52488/composite_benchmarks/Python_uproot.py#L8 https://github.com/Moelf/UnROOT_RDataFrame_MiniBenchmark/blo... -- the result looks "neat" as in a piece of art.
szvsw 2y ago
A great alternative to numba for accelerated Python is Taichi. Trivial to convert a regular python program into a taichi kernel, and then it can target CUDA (and a variety of other options) as the backend. No need to worry about block/grid/thread allocation etc. at the same time, it’s super deep with great support for data classes, custom memory layouts for complexly nested classes, etc etc, comes with autograd, etc. I’m a huge fan - makes writing code that runs on the GPU and integrates with your python libraries an absolute breeze. Super powerful. By far the best tool in the accelerated python toolbox IMO.
- OutOfHere 2y ago
  Negative, as Taichi doesn't even support Python 3.12, and it's unclear if it ever will. Why would I limit myself to an old version of Python?
  - almostgotcaught 2y ago
    Hn people are so haughty https://github.com/taichi-dev/taichi/pull/8522 https://github.com/taichi-dev/taichi/pull/8522
    - OutOfHere 2y ago
      The haughtiness is not for nothing. Since Dec 2023, they made a lame excuse that Pytorch didn't support 3.12: https://github.com/taichi-dev/taichi/issues/8365#issuecomment-1844819939 https://github.com/taichi-dev/taichi/issues/8365#issuecommen... Later, even when Pytorch added support for 3.12, nothing changed (so far) in Taichi.
      - almostgotcaught 2y ago
        >they made a lame excuse that Pytorch didn't support 3.12 how is this a lame excuse >but it fails on a bunch of PyTorch-related tests. We then figured out that PyTorch does not have Python 3.12 support they have a dep that was blocking them from upgrading. you would have them do what? push pytorch to upgrade? >Later, even when Pytorch added support for 3.12, nothing changed (so far) in Taichi. my friend that "Later" is feb/march of this year ie 2-3 months ago. exactly how fast would you like for this open source project to service your needs? not to mention there is a PR up for the bump. I stand by my original comment.

by moelf 2y ago

elashri 2y ago

That'a true and Julia might be a solution but I don't see the adoption happening anytime soon. But this particular problem (per row computation) have different options to tackle now in hep-python ecosystem. One approach is to leverage array programming with NumPy to vectorize operations as much as possible. By operating on entire arrays rather than looping over individual elements, significant speedups can often be achieved. Another possibility is to use a library like Awkward Array, which is designed to work with nested, variable-sized data structures. Awkward Array integrates well with uproot and provides a powerful and flexible framework for performing fast computations on i.e jagged arrays.

moelf 2y ago
Uproot already returns you Awkward array, so both things you mentioned are different ways of saying the same thing. The irreducible complexity of data analysis is there no matter how you do it, and "one-vector-at-a-time" sometimes feel like shoehorning (other terms people come up with include vector-style mental gymnastics). For the record, vector-style programming is great when it works, I mean Julia even has a dedicated syntax for broadcasting. I'm saying when the irreducible complexity arrives, you don't want to NOT be able to just write a for-loop Just a recent example, a double-for loop looks like this in Awkward array: https://github.com/Moelf/UnROOT_RDataFrame_MiniBenchmark/blob/acd5bdc87704ea9111c4adb1217a4bd894b52488/composite_benchmarks/Python_uproot.py#L8 https://github.com/Moelf/UnROOT_RDataFrame_MiniBenchmark/blo... -- the result looks "neat" as in a piece of art.

szvsw 2y ago

A great alternative to numba for accelerated Python is Taichi. Trivial to convert a regular python program into a taichi kernel, and then it can target CUDA (and a variety of other options) as the backend. No need to worry about block/grid/thread allocation etc. at the same time, it’s super deep with great support for data classes, custom memory layouts for complexly nested classes, etc etc, comes with autograd, etc. I’m a huge fan - makes writing code that runs on the GPU and integrates with your python libraries an absolute breeze. Super powerful. By far the best tool in the accelerated python toolbox IMO.

OutOfHere 2y ago
Negative, as Taichi doesn't even support Python 3.12, and it's unclear if it ever will. Why would I limit myself to an old version of Python?
- almostgotcaught 2y ago
  Hn people are so haughty https://github.com/taichi-dev/taichi/pull/8522 https://github.com/taichi-dev/taichi/pull/8522
  - OutOfHere 2y ago
    The haughtiness is not for nothing. Since Dec 2023, they made a lame excuse that Pytorch didn't support 3.12: https://github.com/taichi-dev/taichi/issues/8365#issuecomment-1844819939 https://github.com/taichi-dev/taichi/issues/8365#issuecommen... Later, even when Pytorch added support for 3.12, nothing changed (so far) in Taichi.
    - almostgotcaught 2y ago
      >they made a lame excuse that Pytorch didn't support 3.12 how is this a lame excuse >but it fails on a bunch of PyTorch-related tests. We then figured out that PyTorch does not have Python 3.12 support they have a dep that was blocking them from upgrading. you would have them do what? push pytorch to upgrade? >Later, even when Pytorch added support for 3.12, nothing changed (so far) in Taichi. my friend that "Later" is feb/march of this year ie 2-3 months ago. exactly how fast would you like for this open source project to service your needs? not to mention there is a PR up for the bump. I stand by my original comment.