3 ms·

And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared a

And on a recent generation Xeon, your mid-level data cache can total up to 64MB, depending on model. That's maybe the more interesting one? L3 cache is shared among cores, so access to it can be more heavily impacted by memory concurrency concerns and not just pure bandwidth concerns. Meaning there's potentially a strong incentive to keep your task size below 2MB per core.

CyberDildonics 2y ago
What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?
- bunderbunder 2y ago
  Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
  - CyberDildonics 2y ago
    It seems like instead of a "silent majority doing intensive calculations" what you're really saying is "I don't think of slower speeds as a performance loss, I don't pay attention to that, I don't care about slower speeds, my data is small and my programs run almost instantly". If someone says "this isn't really high performance" and you say "I don't personally care, my stuff is small and doesn't take long time to run" that isn't a coherent reply.
    - bunderbunder 2y ago
      It's really more about acknowledging that, unless you're explicitly talking about something like HPC, "high" is a relative term. It quite reasonably means different things to different people in different contexts. In Python I can often improve a task's run time by multiple orders of magnitude, getting its run time from "irritating" to "not really worth thinking about" in the process, just by making sure I'm doing vectorized operations that can make use of SIMD instructions in the appropriate places. That's more than high enough performance for my needs in most cases. A similar situation may be true for the author of this library. I find it to be much less coherent, nearly to the point of knocking down straw men, when people get lost in arguing semantics over a term that we should all know by now is perennially ill-defined.
      - CyberDildonics 2y ago
        It's not ill defined to everyone else. If I say my phone is fast but it's nowhere close to the speed of a new phone, no one else would say it's fast. High performance means you show that its speed is competitive against the other fastest programming methods. You labeling something "high performance" because you wrote something slow and made it faster has nothing to do with releasing something and calling it "high performance" publicly.

CyberDildonics 2y ago

What are doing that is computationally intensive and are using python and a CPU with 64MB of cache for, but is under 64 MB total? How do you know you aren't losing performance by doing single operations on each array in python?

bunderbunder 2y ago
Model training and inference, for example. Despite the hype, there's actually a lot of data science work that uses modestly-sized sets of structured data and doesn't really need a billion parameter deep learning model. And even when you're running inference on larger datasets it's often fine to just use mini batches. I don't really think of it as a "losing" performance problem. It's all about tradeoffs. SIMD tooling often offers an easier programming model than SPMD tooling, and spending an extra 60 minutes of development time to save myself 60 seconds of run time is generally not a worthwhile trade in my problem space. Especially if I also need to run it on more expensive compute instances to realize that speedup.
- CyberDildonics 2y ago
  It seems like instead of a "silent majority doing intensive calculations" what you're really saying is "I don't think of slower speeds as a performance loss, I don't pay attention to that, I don't care about slower speeds, my data is small and my programs run almost instantly". If someone says "this isn't really high performance" and you say "I don't personally care, my stuff is small and doesn't take long time to run" that isn't a coherent reply.
  - bunderbunder 2y ago
    It's really more about acknowledging that, unless you're explicitly talking about something like HPC, "high" is a relative term. It quite reasonably means different things to different people in different contexts. In Python I can often improve a task's run time by multiple orders of magnitude, getting its run time from "irritating" to "not really worth thinking about" in the process, just by making sure I'm doing vectorized operations that can make use of SIMD instructions in the appropriate places. That's more than high enough performance for my needs in most cases. A similar situation may be true for the author of this library. I find it to be much less coherent, nearly to the point of knocking down straw men, when people get lost in arguing semantics over a term that we should all know by now is perennially ill-defined.
    - CyberDildonics 2y ago
      It's not ill defined to everyone else. If I say my phone is fast but it's nowhere close to the speed of a new phone, no one else would say it's fast. High performance means you show that its speed is competitive against the other fastest programming methods. You labeling something "high performance" because you wrote something slow and made it faster has nothing to do with releasing something and calling it "high performance" publicly.