3 ms·

Author of the original blog post here. I agree, the point isn't really about how Python is super slow compared to C++, or any language. The point is about how

Author of the original blog post here. I agree, the point isn't really about how Python is super slow compared to C++, or any language. The point is about how easy it is for overhead to creep in when you're working with these unbelievably fast accelerators. Like, the point is that if you perform a single addition in Python, and then have your GPU crunching away at a matmul simultaneously, your GPU is going to get through 9.75 million FLOPS before Python finishes its single add.[1] Even if you did it in C++, how many single-threaded adds could you do? Probably on the order of a couple billion (let's round (generously) up to 10 billion). In that case, in the time you finish one add, an A100 will finish 31 thousand FLOPS. Of course, you can leverage multithreading/SIMD if it really is parallelizable. But say, if you're doing a ref-count bump, your A100 will finish 31 thousand FLOPS before you finish that ref-count bump.

derbOac 5y ago
Yes but if everything from the ground up is faster you might avoid some of the wrapper overhead. This is the problem with "legacy" languages like python. There's still a big difference between 9.75 million and 31 k, especially if you multiply that out. I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.
- chillee 5y ago
  Oh, wrapper overhead of Python is certainly annoying. There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU. But yeah, the Python overhead is annoying (see https://dev-discuss.pytorch.org/t/where-we-are-headed-and-why-it-looks-a-lot-like-julia-but-not-exactly-like-julia/ https://dev-discuss.pytorch.org/t/where-we-are-headed-and-wh...), just giving an explanation why it hasn't been a dealbreaker.
  - derbOac 5y ago
    It's a good essay/post/article, by the way. It's good to see stuff like this really laid out like this.

derbOac 5y ago

Yes but if everything from the ground up is faster you might avoid some of the wrapper overhead. This is the problem with "legacy" languages like python. There's still a big difference between 9.75 million and 31 k, especially if you multiply that out. I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.

chillee 5y ago
Oh, wrapper overhead of Python is certainly annoying. There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU. But yeah, the Python overhead is annoying (see https://dev-discuss.pytorch.org/t/where-we-are-headed-and-why-it-looks-a-lot-like-julia-but-not-exactly-like-julia/ https://dev-discuss.pytorch.org/t/where-we-are-headed-and-wh...), just giving an explanation why it hasn't been a dealbreaker.
- derbOac 5y ago
  It's a good essay/post/article, by the way. It's good to see stuff like this really laid out like this.