5 ms·

Heuristic parsing of title: probably benchmarking x=a*b; that's not how you do heavy math in Python, A100 needs lots of set up, oh wait, Carmack, he knows all t

Heuristic parsing of title: probably benchmarking x=a*b; that's not how you do heavy math in Python, A100 needs lots of set up, oh wait, Carmack, he knows all that. Moving on.

stingraycharles 5y ago
I don’t think you’ve actually read the blog post behind the Twitter thread. For starters, it’s not Carmack making that claim but merely quoting it, and secondly, it’s much more interesting than benchmarking “a*b”.
- andromaton 5y ago
  My comment was about the tweet.
nonameiguess 5y ago
It's actually saying Python can do roughly 32 million additions per second on his local hardware setup. I just tested mine, and it's a bit faster than that, but those aren't floating point operations anyway, and should actually be faster than FLOPs. Python uses arbitrary precision integers, though, so it's doing more than just a single hardware add per Python "add." You can see the actual implementation here: https://github.com/python/cpython/blob/main/Objects/longobject.c#L1519 https://github.com/python/cpython/blob/main/Objects/longobje... You've got multiple for loops over every digit, array dereferencing, plus the actual adds. Thus, "overhead." Python takes care of a lot for you, making it possible to do whatever you want with thousand digit numbers and it'll just work, but is definitely not meant for doing lots of math with small numbers. Hence why something like PyTorch even exists.
- chillee 5y ago
  Author of the original blog post here. I agree, the point isn't really about how Python is super slow compared to C++, or any language. The point is about how easy it is for overhead to creep in when you're working with these unbelievably fast accelerators. Like, the point is that if you perform a single addition in Python, and then have your GPU crunching away at a matmul simultaneously, your GPU is going to get through 9.75 million FLOPS before Python finishes its single add.[1] Even if you did it in C++, how many single-threaded adds could you do? Probably on the order of a couple billion (let's round (generously) up to 10 billion). In that case, in the time you finish one add, an A100 will finish 31 thousand FLOPS. Of course, you can leverage multithreading/SIMD if it really is parallelizable. But say, if you're doing a ref-count bump, your A100 will finish 31 thousand FLOPS before you finish that ref-count bump.
  - derbOac 5y ago
    Yes but if everything from the ground up is faster you might avoid some of the wrapper overhead. This is the problem with "legacy" languages like python. There's still a big difference between 9.75 million and 31 k, especially if you multiply that out. I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.
    - chillee 5y ago
      Oh, wrapper overhead of Python is certainly annoying. There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU. But yeah, the Python overhead is annoying (see https://dev-discuss.pytorch.org/t/where-we-are-headed-and-why-it-looks-a-lot-like-julia-but-not-exactly-like-julia/ https://dev-discuss.pytorch.org/t/where-we-are-headed-and-wh...), just giving an explanation why it hasn't been a dealbreaker.
      - derbOac 5y ago
        It's a good essay/post/article, by the way. It's good to see stuff like this really laid out like this.

Heuristic parsing of title: probably benchmarking x=a*b; that's not how you do heavy math in Python, A100 needs lots of set up, oh wait, Carmack, he knows all that. Moving on.

stingraycharles 5y ago

I don’t think you’ve actually read the blog post behind the Twitter thread. For starters, it’s not Carmack making that claim but merely quoting it, and secondly, it’s much more interesting than benchmarking “a*b”.

andromaton 5y ago
My comment was about the tweet.

nonameiguess 5y ago

It's actually saying Python can do roughly 32 million additions per second on his local hardware setup. I just tested mine, and it's a bit faster than that, but those aren't floating point operations anyway, and should actually be faster than FLOPs. Python uses arbitrary precision integers, though, so it's doing more than just a single hardware add per Python "add." You can see the actual implementation here: https://github.com/python/cpython/blob/main/Objects/longobject.c#L1519 https://github.com/python/cpython/blob/main/Objects/longobje... You've got multiple for loops over every digit, array dereferencing, plus the actual adds. Thus, "overhead." Python takes care of a lot for you, making it possible to do whatever you want with thousand digit numbers and it'll just work, but is definitely not meant for doing lots of math with small numbers. Hence why something like PyTorch even exists.

chillee 5y ago
Author of the original blog post here. I agree, the point isn't really about how Python is super slow compared to C++, or any language. The point is about how easy it is for overhead to creep in when you're working with these unbelievably fast accelerators. Like, the point is that if you perform a single addition in Python, and then have your GPU crunching away at a matmul simultaneously, your GPU is going to get through 9.75 million FLOPS before Python finishes its single add.[1] Even if you did it in C++, how many single-threaded adds could you do? Probably on the order of a couple billion (let's round (generously) up to 10 billion). In that case, in the time you finish one add, an A100 will finish 31 thousand FLOPS. Of course, you can leverage multithreading/SIMD if it really is parallelizable. But say, if you're doing a ref-count bump, your A100 will finish 31 thousand FLOPS before you finish that ref-count bump.
- derbOac 5y ago
  Yes but if everything from the ground up is faster you might avoid some of the wrapper overhead. This is the problem with "legacy" languages like python. There's still a big difference between 9.75 million and 31 k, especially if you multiply that out. I'm not meaning to badmouth python but I'm surprised it's still used as much as it is for numerical computing specifically because of these types of issues. It seems like it's only a matter of time before there's a switch to something else (Nim? Julia? Something not on most people's radar now?) or python gets a ground up overhaul.
  - chillee 5y ago
    Oh, wrapper overhead of Python is certainly annoying. There's a couple things that ameliorate that. For one, in many situations, it's possible to "trace" out the Python operations. The second is that, as in my blog post, you can "hide" the Python overhead by simply running asynchronously with the GPU. But yeah, the Python overhead is annoying (see https://dev-discuss.pytorch.org/t/where-we-are-headed-and-why-it-looks-a-lot-like-julia-but-not-exactly-like-julia/ https://dev-discuss.pytorch.org/t/where-we-are-headed-and-wh...), just giving an explanation why it hasn't been a dealbreaker.
    - derbOac 5y ago
      It's a good essay/post/article, by the way. It's good to see stuff like this really laid out like this.