5 ms·

The unstated major premise here is that you're working with data that's big enough that this becomes your major bottleneck. There's also a subset - possibly ev

The unstated major premise here is that you're working with data that's big enough that this becomes your major bottleneck. There's also a subset - possibly even a silent majority - of problems where you're doing fairly intensive calculation on data that fairly comfortably fits into a modern CPU cache. For those, I'd still expect SIMD to be a great choice from a performance perspective.

CyberDildonics 2y ago
I'm not exactly sure what this means, but the bandwidth to the level 3 cache isn't actually more than memory. If you only have a few megabytes to work with, it's a lot less likely performance is going to matter anyway, and if it does you would still want to do multiple operations on each array item if possible, because if you put a few more lines in a loop you're talking about registers at that point, not cache.
- dzaima 2y ago
  L3 cache bandwidth is often more than main memory; most systems at https://old.chipsandcheese.com/memory-bandwidth-data/ https://old.chipsandcheese.com/memory-bandwidth-data/ give at least a ~2x difference from whatever sample I clicked.
  - bunderbunder 2y ago
    And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.
    - CyberDildonics 2y ago
      What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?
      - bunderbunder 2y ago
        Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
        CyberDildonics 2y ago
        It seems like instead of a "silent majority doing intensive calculations" what you're really saying is "I don't think of slower speeds as a performance loss, I don't pay attention to that, I don't care about slower speeds, my data is small and my programs run almost instantly". If someone says "this isn't really high performance" and you say "I don't personally care, my stuff is small and doesn't take long time to run" that isn't a coherent reply.
        bunderbunder 2y ago
        It's really more about acknowledging that, unless you're explicitly talking about something like HPC, "high" is a relative term. It quite reasonably means different things to different people in different contexts. In Python I can often improve a task's run time by multiple orders of magnitude, getting its run time from "irritating" to "not really worth thinking about" in the process, just by making sure I'm doing vectorized operations that can make use of SIMD instructions in the appropriate places. That's more than high enough performance for my needs in most cases. A similar situation may be true for the author of this library. I find it to be much less coherent, nearly to the point of knocking down straw men, when people get lost in arguing semantics over a term that we should all know by now is perennially ill-defined.

CyberDildonics 2y ago

I'm not exactly sure what this means, but the bandwidth to the level 3 cache isn't actually more than memory. If you only have a few megabytes to work with, it's a lot less likely performance is going to matter anyway, and if it does you would still want to do multiple operations on each array item if possible, because if you put a few more lines in a loop you're talking about registers at that point, not cache.

dzaima 2y ago
L3 cache bandwidth is often more than main memory; most systems at https://old.chipsandcheese.com/memory-bandwidth-data/ https://old.chipsandcheese.com/memory-bandwidth-data/ give at least a ~2x difference from whatever sample I clicked.
- bunderbunder 2y ago
  And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.
  - CyberDildonics 2y ago
    What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?
    - bunderbunder 2y ago
      Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
      - CyberDildonics 2y ago
        It seems like instead of a "silent majority doing intensive calculations" what you're really saying is "I don't think of slower speeds as a performance loss, I don't pay attention to that, I don't care about slower speeds, my data is small and my programs run almost instantly". If someone says "this isn't really high performance" and you say "I don't personally care, my stuff is small and doesn't take long time to run" that isn't a coherent reply.
        bunderbunder 2y ago
        It's really more about acknowledging that, unless you're explicitly talking about something like HPC, "high" is a relative term. It quite reasonably means different things to different people in different contexts. In Python I can often improve a task's run time by multiple orders of magnitude, getting its run time from "irritating" to "not really worth thinking about" in the process, just by making sure I'm doing vectorized operations that can make use of SIMD instructions in the appropriate places. That's more than high enough performance for my needs in most cases. A similar situation may be true for the author of this library. I find it to be much less coherent, nearly to the point of knocking down straw men, when people get lost in arguing semantics over a term that we should all know by now is perennially ill-defined.