8 ms·

I'm a bit confused about the premise. This is not comparing pure Python code against some native (C or Rust) code. It's comparing one Python wrapper around nati

I'm a bit confused about the premise. This is not comparing pure Python code against some native (C or Rust) code. It's comparing one Python wrapper around native code (Python's file read method) against another Python wrapper around some native code (OpenDAL). OK it's still interesting that there's a difference in performance, but it's very odd to describe it as "slower than Python". Did they expect that the Python standard library is all written in pure Python? On the contrary, I would expect the implementations of functions in Python's standard library to be native and, individually, highly optimised. I'm not surprised the conclusion had something to do with the way that native code works. Admittedly I was surprised at the specific answer - still a very interesting article despite the confusing start. Edit: The conclusion also took me a couple of attempts to parse. There's a heading "C is slower than Python with specified offset". To me, as a native English speaker, this reads as "C is slower (than Python) with specified offset" i.e. it sounds like they took the C code, specified the same offset as Python, and then it's still slower than Python. But it's the opposite: once the offset from Python was also specified in the C code, the C code was then faster. Still very interesting once I got what they were saying though.

xuanwo 3y ago
Thanks for the comments. I have fixed the headers :)
crabbone 3y ago
> individually, highly optimised. Now why would you expect that? What happened to OP is a pure chance. CPython's C code doesn't even care about const-consistency. It's flush with dynamic memory allocations, bunch of helper / convenience calls... Even stuff like arithmetic does dynamic memory allocation... Normally, you don't expect CPython to perform well, not if you have any experience working with it. Whenever you want to improve performance you want to sidestep all the functionality available there. Also, while Python doesn't have a standard library, since it doesn't have a standard... the library that's distributed with it is mostly written in Python. Of course, some of it comes written in C, but there's also a sizable fraction of that C code that's essentially Python code translated mechanically into C (a good example of this is Python's binary search implementation which was originally written in Python, and later translated into C using Python's C API). What one would expect is that functionality that is simple to map to operating system functionality has a relatively thin wrapper. I.e. reading files wouldn't require much in terms of binding code because, essentially, it goes straight into the system interface.
- codr7 3y ago
  Have you ever attempted to write a scripting language that performs better? I have, several, and it's far from trivial. The basics are seriously optimized for typical use cases, take a look at the source code for the dict type.
  - svieira 3y ago
    Raymond Hettinger's talk Modern Python Dictionaries: A confluence of a dozen great ideas is an awesome "history of how we got these optimizations" and a walk through why they are so effective - https://www.youtube.com/watch?v=npw4s1QTmPg https://www.youtube.com/watch?v=npw4s1QTmPg
    - codr7 3y ago
      Yeah, I had a nice chat with Raymond Hettinger at a Pycon in Birmingham/UK back in the days (had no idea who he was at the time). He seemed like a dedicated and intelligent person, I'm sure we can thank him for some of that.
  - crabbone 3y ago
    > Have you ever attempted to write a scripting language that performs better? No, because "scripting language" is not a thing. But, if we are talking about implementing languages, then I worked with many language implementations. The most comparable one that I know fairly well, inside-and-out would be the AVM, i.e. the ActionScript Virtual Machine. It's not well-written either unfortunately. I've looked at implementations of Lua, Emacs Lisp and Erlang at different times and to various degree. I'm also somewhat familiar with SBCL and ECL, the implementation side. There are different things the authors looked for in these implementations. For example, SBCL emphasizes performance, where ECL emphasizes simplicity and interop with C. If I had to grade language implementations I've seen, Erlang would absolutely take the cake. It's a very thoughtful and disciplined program where authors went to a great length to design and implement it. CPython is on the lower end of such programs. It's anarchic, very unevenly implemented, you run into comments testifying to the author not knowing what they are doing, what their predecessor did, nor what to do next. Sometimes the code is written from that perspective as well, as in if the author somehow manages to drive themselves in the corner they don't know what the reference count is anymore, they'll just hammer it until they hope all references are dead (well, maybe). It's the code style that, unfortunately, I associate with proprietary projects where deadlines and cost dictate the quality, where concurrency problems are solved with sleeps, and if that doesn't work, then the sleep delay is doubled. It's not because I specifically hate code being proprietary, but because I meet that kind of code in my day job more than I meet it in hobby open-source projects. > take a look at the source code for the dict type. I wrote a Protobuf parser in C with the intention of exposing its bindings to Python. Dictionaries were a natural choice for the hash-map Protobuf elements. I benchmarked my implementation against C++ (Google's) implementation only to discover that std::map wins against Python's dictionary by a landslide. Maybe Python's dict isn't as bad as most of the rest of the interpreter, but being the best of the worst still doesn't make it good.
qd011 3y ago
I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.
- paulddraper 3y ago
  Yeah, it's weird.
- afdbcreid 3y ago
  Usually, yes, but when it's a bug in the hardware, it's not really that Python is fast, more like that CPython developers were lucky enough to not have the bug.
  - munch117 3y ago
    How do you know that it's luck?
    - cozzyd 3y ago
      Because the offset is entirely due to space for the PyObject header.
      - munch117 3y ago
        The PyObject header is a target for optimisation. Performance regressions are likely to be noticed, and if a different header layout is faster, then it's entirely possible that it will be used for purely empirical reasons. Trying different options and picking the best performing one is not luck, even if you can't explain why it's the best performing.
        cozzyd 3y ago
        I suspect any size other than 0 would lead to this. But the Zen3/4 were developed far, far after the PyObject header...
fl0ki 3y ago
The premise is that any time you say "Python [...] faster than Rust [...]" you get page views even if it's not true. People have noticed after the last few dozen times something like this was posted.
- p5a0u9l 3y ago
  This is the answer. The thread is chasing various smart-people opinions about languages, interpreters, system calls. We got tricked into click bait title and are using the opportunity to rehash our favorite topics and biases. On the other hand… so what? It’s kind of fun.
lambda 3y ago
I'm a bit confused by why you are confused. It's surprising that something as simple as reading a file is slower in the Rust standard library as the Python standard library. Even knowing that a Python standard library call like this is written in C, you'd still expect the Rust standard library call to be of a similar speed; so you'd expect either that you're using it wrong, or that the Rust standard library has some weird behavior. In this case, it turns out that neither were the case; there's just a weird hardware performance cliff based on the exact alignment of an allocation on particular hardware. So, yeah, I'd expect a filesystem read to be pretty well optimized in Python, but I'd expect the same in Rust, so it's surprising that the latter was so much slower, and especially surprising that it turned out to be hardware and allocator dependent.
- quietbritishjim 3y ago
  It's just the spin of it that threw me off. You're right: "why is a C implementation so much faster than a widely used Rust implementation" is a valid and interesting question. But phrasing it as "why is a Python function faster than a Rust function", when it's clearly not the comparison at all, distracts from the real question.
deleted 3y ago
[deleted]

xuanwo 3y ago

Thanks for the comments. I have fixed the headers :)

crabbone 3y ago

> individually, highly optimised. Now why would you expect that? What happened to OP is a pure chance. CPython's C code doesn't even care about const-consistency. It's flush with dynamic memory allocations, bunch of helper / convenience calls... Even stuff like arithmetic does dynamic memory allocation... Normally, you don't expect CPython to perform well, not if you have any experience working with it. Whenever you want to improve performance you want to sidestep all the functionality available there. Also, while Python doesn't have a standard library, since it doesn't have a standard... the library that's distributed with it is mostly written in Python. Of course, some of it comes written in C, but there's also a sizable fraction of that C code that's essentially Python code translated mechanically into C (a good example of this is Python's binary search implementation which was originally written in Python, and later translated into C using Python's C API). What one would expect is that functionality that is simple to map to operating system functionality has a relatively thin wrapper. I.e. reading files wouldn't require much in terms of binding code because, essentially, it goes straight into the system interface.

codr7 3y ago
Have you ever attempted to write a scripting language that performs better? I have, several, and it's far from trivial. The basics are seriously optimized for typical use cases, take a look at the source code for the dict type.
- svieira 3y ago
  Raymond Hettinger's talk Modern Python Dictionaries: A confluence of a dozen great ideas is an awesome "history of how we got these optimizations" and a walk through why they are so effective - https://www.youtube.com/watch?v=npw4s1QTmPg https://www.youtube.com/watch?v=npw4s1QTmPg
  - codr7 3y ago
    Yeah, I had a nice chat with Raymond Hettinger at a Pycon in Birmingham/UK back in the days (had no idea who he was at the time). He seemed like a dedicated and intelligent person, I'm sure we can thank him for some of that.
- crabbone 3y ago
  > Have you ever attempted to write a scripting language that performs better? No, because "scripting language" is not a thing. But, if we are talking about implementing languages, then I worked with many language implementations. The most comparable one that I know fairly well, inside-and-out would be the AVM, i.e. the ActionScript Virtual Machine. It's not well-written either unfortunately. I've looked at implementations of Lua, Emacs Lisp and Erlang at different times and to various degree. I'm also somewhat familiar with SBCL and ECL, the implementation side. There are different things the authors looked for in these implementations. For example, SBCL emphasizes performance, where ECL emphasizes simplicity and interop with C. If I had to grade language implementations I've seen, Erlang would absolutely take the cake. It's a very thoughtful and disciplined program where authors went to a great length to design and implement it. CPython is on the lower end of such programs. It's anarchic, very unevenly implemented, you run into comments testifying to the author not knowing what they are doing, what their predecessor did, nor what to do next. Sometimes the code is written from that perspective as well, as in if the author somehow manages to drive themselves in the corner they don't know what the reference count is anymore, they'll just hammer it until they hope all references are dead (well, maybe). It's the code style that, unfortunately, I associate with proprietary projects where deadlines and cost dictate the quality, where concurrency problems are solved with sleeps, and if that doesn't work, then the sleep delay is doubled. It's not because I specifically hate code being proprietary, but because I meet that kind of code in my day job more than I meet it in hobby open-source projects. > take a look at the source code for the dict type. I wrote a Protobuf parser in C with the intention of exposing its bindings to Python. Dictionaries were a natural choice for the hash-map Protobuf elements. I benchmarked my implementation against C++ (Google's) implementation only to discover that std::map wins against Python's dictionary by a landslide. Maybe Python's dict isn't as bad as most of the rest of the interpreter, but being the best of the worst still doesn't make it good.

qd011 3y ago

I don't understand why Python gets shit for being a slow language when it's slow but no credit for being fast when it's fast just because "it's not really Python". If I write Python and my code is fast, to me that sounds like Python is fast, I couldn't care less whether it's because the implementation is in another language or for some other reason.

paulddraper 3y ago
Yeah, it's weird.
afdbcreid 3y ago
Usually, yes, but when it's a bug in the hardware, it's not really that Python is fast, more like that CPython developers were lucky enough to not have the bug.
- munch117 3y ago
  How do you know that it's luck?
  - cozzyd 3y ago
    Because the offset is entirely due to space for the PyObject header.
    - munch117 3y ago
      The PyObject header is a target for optimisation. Performance regressions are likely to be noticed, and if a different header layout is faster, then it's entirely possible that it will be used for purely empirical reasons. Trying different options and picking the best performing one is not luck, even if you can't explain why it's the best performing.
      - cozzyd 3y ago
        I suspect any size other than 0 would lead to this. But the Zen3/4 were developed far, far after the PyObject header...

fl0ki 3y ago

The premise is that any time you say "Python [...] faster than Rust [...]" you get page views even if it's not true. People have noticed after the last few dozen times something like this was posted.

p5a0u9l 3y ago
This is the answer. The thread is chasing various smart-people opinions about languages, interpreters, system calls. We got tricked into click bait title and are using the opportunity to rehash our favorite topics and biases. On the other hand… so what? It’s kind of fun.

lambda 3y ago

I'm a bit confused by why you are confused. It's surprising that something as simple as reading a file is slower in the Rust standard library as the Python standard library. Even knowing that a Python standard library call like this is written in C, you'd still expect the Rust standard library call to be of a similar speed; so you'd expect either that you're using it wrong, or that the Rust standard library has some weird behavior. In this case, it turns out that neither were the case; there's just a weird hardware performance cliff based on the exact alignment of an allocation on particular hardware. So, yeah, I'd expect a filesystem read to be pretty well optimized in Python, but I'd expect the same in Rust, so it's surprising that the latter was so much slower, and especially surprising that it turned out to be hardware and allocator dependent.

quietbritishjim 3y ago
It's just the spin of it that threw me off. You're right: "why is a C implementation so much faster than a widely used Rust implementation" is a valid and interesting question. But phrasing it as "why is a Python function faster than a Rust function", when it's clearly not the comparison at all, distracts from the real question.

deleted 3y ago

[deleted]