3 ms·

Naive array programming is not really high performance in my book, because performing a lot of trivial arithmetic on large arrays leads to poor locality and hig

Naive array programming is not really high performance in my book, because performing a lot of trivial arithmetic on large arrays leads to poor locality and high bandwidth pressure. The better alternative is SPMD, i.e. something like CUDA or ISPC for CPUs. This is possible with some type of JIT if the numpy style of programming is to be maintained, for example tinygrad.

cl3misch 2y ago
If you like high-performance array programming a la "numpy with JIT" I suggest looking at JAX. It's very suitable for general numeric computing (not just ML) and a very mature ecosystem. https://github.com/jax-ml/jax https://github.com/jax-ml/jax
deleted 2y ago
[deleted]
CyberDildonics 2y ago
You are absolutely right, naive array programming might be much faster than raw python but it will never be high performance because you can't use caches and memory bandwidth effectively.
- bunderbunder 2y ago
  The unstated major premise here is that you're working with data that's big enough that this becomes your major bottleneck. There's also a subset - possibly even a silent majority - of problems where you're doing fairly intensive calculation on data that fairly comfortably fits into a modern CPU cache. For those, I'd still expect SIMD to be a great choice from a performance perspective.
  - CyberDildonics 2y ago
    I'm not exactly sure what this means, but the bandwidth to the level 3 cache isn't actually more than memory. If you only have a few megabytes to work with, it's a lot less likely performance is going to matter anyway, and if it does you would still want to do multiple operations on each array item if possible, because if you put a few more lines in a loop you're talking about registers at that point, not cache.
    - dzaima 2y ago
      L3 cache bandwidth is often more than main memory; most systems at https://old.chipsandcheese.com/memory-bandwidth-data/ https://old.chipsandcheese.com/memory-bandwidth-data/ give at least a ~2x difference from whatever sample I clicked.
      - bunderbunder 2y ago
        And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.
        CyberDildonics 2y ago
        What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?
        bunderbunder 2y ago
        Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
        CyberDildonics 2y ago
mlochbaum 2y ago
Memory traffic is certainly the biggest problem that the array paradigm presents for implementation, yes. I'd quibble with calling that "poor locality": when working with large arrays, if any part of a cache line is accessed it's very likely the entire line will be used in short order, which is the definition of good locality as I understand it. The issue is simply high memory use and a large number of accesses. I think it's an oversimplification to say SPMD is a better model full stop. Array programming has the advantage that the implementer optimizes a specific function, and its interaction with the rest of the program is much simpler: all the data is available at the start, and the result can be produced in any order. So things like multi-pass algorithms and temporary lookup tables are possible, particularly valuable if the program isn't spending that much time on arithmetic but rather "heavy" searching and sorting operations. Not that I claim NumPy or CuPy do a good job of this. Sections "Fusion versus fission" and "Dynamic versus static" are relevant here, I think: https://mlochbaum.github.io/BQN/implementation/versusc.html#fusion-versus-fission https://mlochbaum.github.io/BQN/implementation/versusc.html#...

cl3misch 2y ago

If you like high-performance array programming a la "numpy with JIT" I suggest looking at JAX. It's very suitable for general numeric computing (not just ML) and a very mature ecosystem. https://github.com/jax-ml/jax https://github.com/jax-ml/jax

deleted 2y ago

[deleted]

CyberDildonics 2y ago

You are absolutely right, naive array programming might be much faster than raw python but it will never be high performance because you can't use caches and memory bandwidth effectively.

bunderbunder 2y ago
The unstated major premise here is that you're working with data that's big enough that this becomes your major bottleneck. There's also a subset - possibly even a silent majority - of problems where you're doing fairly intensive calculation on data that fairly comfortably fits into a modern CPU cache. For those, I'd still expect SIMD to be a great choice from a performance perspective.
- CyberDildonics 2y ago
  I'm not exactly sure what this means, but the bandwidth to the level 3 cache isn't actually more than memory. If you only have a few megabytes to work with, it's a lot less likely performance is going to matter anyway, and if it does you would still want to do multiple operations on each array item if possible, because if you put a few more lines in a loop you're talking about registers at that point, not cache.
  - dzaima 2y ago
    L3 cache bandwidth is often more than main memory; most systems at https://old.chipsandcheese.com/memory-bandwidth-data/ https://old.chipsandcheese.com/memory-bandwidth-data/ give at least a ~2x difference from whatever sample I clicked.
    - bunderbunder 2y ago
      And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.
      - CyberDildonics 2y ago
        What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?
        bunderbunder 2y ago
        Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
        CyberDildonics 2y ago

mlochbaum 2y ago

Memory traffic is certainly the biggest problem that the array paradigm presents for implementation, yes. I'd quibble with calling that "poor locality": when working with large arrays, if any part of a cache line is accessed it's very likely the entire line will be used in short order, which is the definition of good locality as I understand it. The issue is simply high memory use and a large number of accesses. I think it's an oversimplification to say SPMD is a better model full stop. Array programming has the advantage that the implementer optimizes a specific function, and its interaction with the rest of the program is much simpler: all the data is available at the start, and the result can be produced in any order. So things like multi-pass algorithms and temporary lookup tables are possible, particularly valuable if the program isn't spending that much time on arithmetic but rather "heavy" searching and sorting operations. Not that I claim NumPy or CuPy do a good job of this. Sections "Fusion versus fission" and "Dynamic versus static" are relevant here, I think: https://mlochbaum.github.io/BQN/implementation/versusc.html#fusion-versus-fission https://mlochbaum.github.io/BQN/implementation/versusc.html#...