4 ms·

Hi, author here! Copying my comment from the previous submission[x]: This was a quite common criticism over r/Python[0] as well, saying that using C modules a

Hi, author here! Copying my comment from the previous submission[x]: This was a quite common criticism over r/Python[0] as well, saying that using C modules and then benchmarking "Python" with C isn't fair. Perhaps it was my wording that caused this confusion, for which I am sorry, but I never meant to compare "pure" Python with C. The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. [0]: https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redis_clone_in_python_3_to_disprove_some/ https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi... [1]: It's of course controversial whether 60% throughput is considerable or not. [x]: https://news.ycombinator.com/item?id=19293590 https://news.ycombinator.com/item?id=19293590

rualca 6y ago
> The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. You don't make that point. You instead make the point that calling C code with glue code is slower than C code alone, and the performance hit of using Python as glue code in your case is a 2x slowdown.
tutfbhuf 6y ago
From your README.md: > pydis is an experiment to disprove some of the falsehoods about performance and your comment: > The point I am trying to make is that, Python with C extensions can be as considerably performant as C code for network or memory bound tasks. So your point is regarding the falsehood belief that python with heavy use of C extensions is (not) much slower (in this case ~60%) than C? I don't think that such a falsehood belief exists and therefore pydis disproves nothing.
- jnwatson 6y ago
  You're missing the point. Obviously, you can write your application in Cython and have decent performance. The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.
  - sushshshsh 6y ago
    This, one hundred percent. There's 3 tiers of things happening here. 1) Write Redis in C. Hard for some devs who only know Python, but good performance. 2) Write Redis in Python with C libraries. Doable for a Python dev, performance hit. 3) Write Redis in pure Python. A fun experiment to see the performance limitations of Python when compared to C running on the same machine.
    - ZephyrBlu 6y ago
      The thing is that the title/project makes you think it's 3), when it's actually 2). It feels a bit sensationalist to me, and I see this all the time with projects these days. It's kind of disappointing.
    - UncleMeat 6y ago
      But the claim is that he is dispelling common falsehoods. Are there actually people out there saying that python glue with c libraries for performance critical parts is going to be orders of magnitude slower than an entire thing written in c?
      - sushshshsh 6y ago
        I actually don't like Python and much prefer Java, but I think #3 is really only for performance research and like, fun and games. It would be cool to have a quantified answer to "just how slow is Python"? I suspect the answer is many orders of magnitude slower. However, if the experiment in question is, "using python glue code, can a developer string together a service that replicates the functionality of redis", then that illustrates what would actually happen in the real world. For example, for a very junior Python developer, writing something in C is basically like asking a C developer to write assembly- possible, but they'd have to learn a lot of different things to do it. Similarly, when I write Java, I want to be calling Arrays.sort(myarray) instead of writing my own sort (if the focus is on developer convenience and development velocity). So, it's just a difference in what you're desiring to measure.
  - tutfbhuf 6y ago
- mekoka 6y ago
  > I don't think that such a falsehood belief exists and therefore pydis disproves nothing. You don't think such a falsehood belief exists, therefore you don't think pydis disproves anything.
tylermauthe 6y ago
This experiment reminds me of a rant from Ryan Dahl on software a few years back: > In the past year I think I have finally come to understand the ideals of Unix: file descriptors and processes orchestrated with C. It's a beautiful idea. We just keep moving up the stack, it seems :) Source: https://gist.github.com/cookrn/4015437 https://gist.github.com/cookrn/4015437 (edit: formatting)
- sushshshsh 6y ago
  The older I get, the more I appreciate the simplicty and power of C, text files, a terminal, and TCP/IP
brian_herman 6y ago
If you ran it would pypy would it be faster?
noname120 6y ago
What caused confusion is that you put “written in ~250 lines of idiomatic Python code” in the readme file and didn't mention anywhere that you were in fact using a bunch of C libraries to do the heavy work. The disclaimer section (which contains this information) was added 44 minutes ago. And it's still misleading. For example about hiredis you write: > Python extension that wraps protocol parsing code in hiredis. Where is the mention that the parsing code is actually C code? ------ Last but not least you already posted this a year ago on Reddit[1] and Hacker News[2] (edit: someone else did for HN), but only (partially) addressed criticism very recently. [1] https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redis_clone_in_python_3_to_disprove_some/ https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi... [2] https://news.ycombinator.com/item?id=19287717 https://news.ycombinator.com/item?id=19287717
- boramalper 6y ago
  I don't like your tone; this was a one-off experiment (that did not even gain much attention back then) that I did for fun, and no one is entitled to my time so I have addressed those issues when I wanted to. I am not the one who posted here, why so aggressive?
  - smoyer 6y ago
    I don't think the tone was hostile and if you ignore any hint of tone you might hear, you have to admit that the title is really click-baity (yes I understand that you didn't post it). Regardless of the criticisms of the project, it's great that you're experimenting and I would be curious to know what's causing the reduction in performance ... is it only part of the program or is that the performance of the Python code being interpreted.
    - boramalper 6y ago
      > I would be curious to know what's causing the reduction in performance ... is it only part of the program or is that the performance of the Python code being interpreted. Mostly the latter but also a bit of the former: I could have optimised it further — at the expense of idiomaticity — but I believe gains would be marginal. Other than that, Python is not that fast so you pay some penalties there too.
zo1 6y ago
Some of the comments are nitpicking, for sure. A good benefit of Python is that it abstracts a lower-level language into a higher-order one. So that you get some percentage of the raw speed of the lower-level language but in a "easier-to-use", "easier-to-reason-about" and safer format. On some level, it's like complaining that C# is just using C++ modules because the CLR that interprets the C# byte-code is written in C++. But it misses the point. You wrote this in Python and you were able to leverage the speed of a much more complicate language for your benefit. Without touching the lower language and without worrying about it's memory-management or lib dependencies. This is a win, and it's part of the beauty of programming! Btw, my compliments to you on writing this! Really cool project/poc.

rualca 6y ago

> The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. You don't make that point. You instead make the point that calling C code with glue code is slower than C code alone, and the performance hit of using Python as glue code in your case is a 2x slowdown.

tutfbhuf 6y ago

From your README.md: > pydis is an experiment to disprove some of the falsehoods about performance and your comment: > The point I am trying to make is that, Python with C extensions can be as considerably performant as C code for network or memory bound tasks. So your point is regarding the falsehood belief that python with heavy use of C extensions is (not) much slower (in this case ~60%) than C? I don't think that such a falsehood belief exists and therefore pydis disproves nothing.

jnwatson 6y ago
You're missing the point. Obviously, you can write your application in Cython and have decent performance. The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.
- sushshshsh 6y ago
  This, one hundred percent. There's 3 tiers of things happening here. 1) Write Redis in C. Hard for some devs who only know Python, but good performance. 2) Write Redis in Python with C libraries. Doable for a Python dev, performance hit. 3) Write Redis in pure Python. A fun experiment to see the performance limitations of Python when compared to C running on the same machine.
  - ZephyrBlu 6y ago
    The thing is that the title/project makes you think it's 3), when it's actually 2). It feels a bit sensationalist to me, and I see this all the time with projects these days. It's kind of disappointing.
  - UncleMeat 6y ago
    But the claim is that he is dispelling common falsehoods. Are there actually people out there saying that python glue with c libraries for performance critical parts is going to be orders of magnitude slower than an entire thing written in c?
    - sushshshsh 6y ago
      I actually don't like Python and much prefer Java, but I think #3 is really only for performance research and like, fun and games. It would be cool to have a quantified answer to "just how slow is Python"? I suspect the answer is many orders of magnitude slower. However, if the experiment in question is, "using python glue code, can a developer string together a service that replicates the functionality of redis", then that illustrates what would actually happen in the real world. For example, for a very junior Python developer, writing something in C is basically like asking a C developer to write assembly- possible, but they'd have to learn a lot of different things to do it. Similarly, when I write Java, I want to be calling Arrays.sort(myarray) instead of writing my own sort (if the focus is on developer convenience and development velocity). So, it's just a difference in what you're desiring to measure.
- tutfbhuf 6y ago
mekoka 6y ago
> I don't think that such a falsehood belief exists and therefore pydis disproves nothing. You don't think such a falsehood belief exists, therefore you don't think pydis disproves anything.

tylermauthe 6y ago

This experiment reminds me of a rant from Ryan Dahl on software a few years back: > In the past year I think I have finally come to understand the ideals of Unix: file descriptors and processes orchestrated with C. It's a beautiful idea. We just keep moving up the stack, it seems :) Source: https://gist.github.com/cookrn/4015437 https://gist.github.com/cookrn/4015437 (edit: formatting)

sushshshsh 6y ago
The older I get, the more I appreciate the simplicty and power of C, text files, a terminal, and TCP/IP

brian_herman 6y ago

If you ran it would pypy would it be faster?

noname120 6y ago

What caused confusion is that you put “written in ~250 lines of idiomatic Python code” in the readme file and didn't mention anywhere that you were in fact using a bunch of C libraries to do the heavy work. The disclaimer section (which contains this information) was added 44 minutes ago. And it's still misleading. For example about hiredis you write: > Python extension that wraps protocol parsing code in hiredis. Where is the mention that the parsing code is actually C code? ------ Last but not least you already posted this a year ago on Reddit[1] and Hacker News[2] (edit: someone else did for HN), but only (partially) addressed criticism very recently. [1] https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redis_clone_in_python_3_to_disprove_some/ https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi... [2] https://news.ycombinator.com/item?id=19287717 https://news.ycombinator.com/item?id=19287717

boramalper 6y ago
I don't like your tone; this was a one-off experiment (that did not even gain much attention back then) that I did for fun, and no one is entitled to my time so I have addressed those issues when I wanted to. I am not the one who posted here, why so aggressive?
- smoyer 6y ago
  I don't think the tone was hostile and if you ignore any hint of tone you might hear, you have to admit that the title is really click-baity (yes I understand that you didn't post it). Regardless of the criticisms of the project, it's great that you're experimenting and I would be curious to know what's causing the reduction in performance ... is it only part of the program or is that the performance of the Python code being interpreted.
  - boramalper 6y ago
    > I would be curious to know what's causing the reduction in performance ... is it only part of the program or is that the performance of the Python code being interpreted. Mostly the latter but also a bit of the former: I could have optimised it further — at the expense of idiomaticity — but I believe gains would be marginal. Other than that, Python is not that fast so you pay some penalties there too.

zo1 6y ago

Some of the comments are nitpicking, for sure. A good benefit of Python is that it abstracts a lower-level language into a higher-order one. So that you get some percentage of the raw speed of the lower-level language but in a "easier-to-use", "easier-to-reason-about" and safer format. On some level, it's like complaining that C# is just using C++ modules because the CLR that interprets the C# byte-code is written in C++. But it misses the point. You wrote this in Python and you were able to leverage the speed of a much more complicate language for your benefit. Without touching the lower language and without worrying about it's memory-management or lib dependencies. This is a win, and it's part of the beauty of programming! Btw, my compliments to you on writing this! Really cool project/poc.