7 ms·

This is essentially some Python glue for C code. It uses a C Redis client parser (hiredis), uvloop (Python wrapper for libuv) and the Python hashmap implementa

This is essentially some Python glue for C code. It uses a C Redis client parser (hiredis), uvloop (Python wrapper for libuv) and the Python hashmap implementation. All the performance critical code is implemented in C. I appreciate the point that higher level languages can be more than fast enough for many use cases. A tremendous amount of software is written in Python, PHP, Ruby et al after all. But this is not a good way to make that argument, since it almost refutes it's own point: if performance matters, you apparently implement it in C/C++/Rust.

pjmlp 6y ago
A better point would be .NET 5 server implementation of gRPC. https://devblogs.microsoft.com/aspnet/grpc-performance-improvements-in-net-5/ https://devblogs.microsoft.com/aspnet/grpc-performance-impro... However Python isn't never going to get there.
nemetroid 6y ago
I think the use of uvloop is reasonable, since it's not something domain specific to Redis. But doesn't the inclusion of hiredis invalidate the entire point? It's 7500 lines of C specifically for working with Redis.
- quietbritishjim 6y ago
  I agree with this. If you write an application in Python that spends the majority of the time in C-based extensions – for example if I were to write a differential equation GUI using numpy and pyqt – then I think it's still valid to say that I wrote the application in Python. In fact even most of the standard library modules are written in C (e.g. zipfile, io, sqlite3) so it's hard to avoid this. You could say that no program is ever really written in Python but I don't think that's a useful point to make.[1] But if you write a load of C code that's very specific to your application, wrap it in a Python extension, then call it from Python, then I don't think you could argue that you "wrote it in Python". If someone happened to have written that application-specific extension module for you then the situation is not materially different. ([1] A more useful motion is whether your code is vectorised. That is, if you want to do something with a data structure (e.g. mutate each element, find an element, do matrix multiplication), do you have to loop over the elements in Python or do you make a single call into a library that does the looping in C? Either way you're still ultimately going to be calling C code, if nothing else in the Python interpreter itself, but the distinction is obviously still important.)
  - formerly_proven 6y ago
    The Python zipfile module is actually just that, Python. It performs well enough, with compression zlib is the limiting factor anyway.
    - quietbritishjim 6y ago
      OK, by my own rules that means it's "written in Python" then. But in that case what I actually meant is that majority of time in calls to zipfile is spent in pure C, not in the Python interpreter loop, and if it's based on zlib then that agrees with what I meant. (There is an exception I didn't mention: if the zip is encrypted then the zipfile module falls back to pure Python code. The difference is very noticeable!)
- lr4444lr 6y ago
  uvloop also unfairly lets Python take credit for concurrent request handling performance too though, no?
ufo 6y ago
I'm not sure I see what is the problem here. Gluing together pieces of C code is one of the main reasons to use a scripting language like Python.
- jpgvm 6y ago
  Yes but if to achieve good performance in Python you have to implement all the domain specific code (in this case the redis protocol) in C then you: a) aren't writing Python b) invalidates all of the points for writing in Python. i.e speed of implementation, etc.
- timlatim 6y ago
  The problem is how the author frames it in the README: "an experiment to disprove some of the falsehoods about performance and optimisation regarding software and interpreted languages in particular", "unfortunately many programmers [...] spend countless hours by making life harder for themselves in the name of marginal performance gains, often trading many other conveniences (such as type safety)". Between the heavy use of C libraries and correctness issues (https://github.com/boramalper/pydis/issues/12 https://github.com/boramalper/pydis/issues/12), I don't see how this project proves that Python is faster or safer than other languages. That being said, the speed with which it was developed can certainly be seen as Python's strong point.
  - noname120 6y ago
    It's a failed experiment — an experiment nonetheless.
  - kortex 6y ago
    I definitely see both sides. Python is meant to be used exactly like this. It's an exemplar of python-as-library-glue. I also saw the title and clicked on it because I thought to myself "no fricken way...in pure python?" It's a touch baity.
secondcoming 6y ago
But the operations are all done on Python's data structures?
- VWWHFSfQ 6y ago
  aren't they also written in c though
  - secondcoming 6y ago
    Ya. This whole post confused me. I thought it was claiming that the Python port was faster. It's too early in the morning.
  - boramalper 6y ago
    That's taking it too far though isn't it? I mean, gcc is written in C++ too so what? :) You might also be interested in my other comment: https://news.ycombinator.com/item?id=25101200 https://news.ycombinator.com/item?id=25101200
    - rfoo 6y ago
      While what you said is true in general, I believe it is debatable in this case, as Redis (at least the subset implemented in this experiment) is something very very special, it is essentially a hashmap attached to the network. There aren't much stuff left if you offload the "hashmap" part and the "attached to the network" (I/O + protocol parser) part. In other words, in this case the hashmap implementation itself is the business logic of your application.
mattlondon 6y ago
> All the performance critical code is implemented in C. Isn't this why people like python? Just glue together some libraries that someone else has written and then you write a viral blog/twitter post about your <insert trendy thing> that you wrote in just 10 lines of code? Seriously is 't this the main reason people use it? I.e. it has a big library of stuff so it is easy for beginners and/or non-programmers to use? I'm struggling to think of any other reasons to deliberately pick python to start something new in, although it is at least a bit more same now with types.
- stingraycharles 6y ago
  All the reasons you list are indeed reasons to use Python, but none of them require performance critical code to be written in C.
- melenaboija 6y ago
  In my professional environment we use python because of the cost, definitely not because we want to post it anywhere. And this is not an opinion on how companies measure cost or how one language is better than another but maybe trying to explain, and I think underestimate, popularity of Python needs a little bit more insight.
- jefftk 6y ago
  Python is an extremely expressive language, that lets you prototype and experiment very rapidly. In many cases, with some cleanups, refactorings, and calls to C modules like numpy, you can convert those prototypes into sufficiently performant real implementations. Here's a recent write-up of mine where almost all of the time is spent in C extensions but where it was much faster to develop and code in python: https://www.jefftk.com/p/sharding-the-brigade https://www.jefftk.com/p/sharding-the-brigade
- HelloNurse 6y ago
  The important technical question is how good a Redis replacement can be built with 250 lines of Python "glue code". If performance is good enough, where are robustness and features? How large are the effort gap and the quality gap compared to the original Redis?
deleted 6y ago
[deleted]
The_rationalist 6y ago
It could be significantly faster if using io_uring
mhh__ 6y ago
The really annoying thing is is that you can quite happily write shorter code in D than in Python and have it run as fast as C++. One of the reasons D compiles so fast is because of this - the compiler isn't even specifically optimized for speed. Everyone seems to think that writing expressive code is slow, when really it's because Python is slow and C++ is a lumbering behemoth. Python isn't even a high-level language, think about it - it's higher level than C but the capability it has for abstraction it gives you is really just interpreted C with classes.
- krelian 6y ago
  Why did D never take off?
  - mhh__ 6y ago
    Bad timing and the GC
boramalper 6y ago
Hi, author here! Copying my comment from the previous submission[x]: This was a quite common criticism over r/Python[0] as well, saying that using C modules and then benchmarking "Python" with C isn't fair. Perhaps it was my wording that caused this confusion, for which I am sorry, but I never meant to compare "pure" Python with C. The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. [0]: https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redis_clone_in_python_3_to_disprove_some/ https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi... [1]: It's of course controversial whether 60% throughput is considerable or not. [x]: https://news.ycombinator.com/item?id=19293590 https://news.ycombinator.com/item?id=19293590
- rualca 6y ago
  > The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. You don't make that point. You instead make the point that calling C code with glue code is slower than C code alone, and the performance hit of using Python as glue code in your case is a 2x slowdown.
- tutfbhuf 6y ago
  From your README.md: > pydis is an experiment to disprove some of the falsehoods about performance and your comment: > The point I am trying to make is that, Python with C extensions can be as considerably performant as C code for network or memory bound tasks. So your point is regarding the falsehood belief that python with heavy use of C extensions is (not) much slower (in this case ~60%) than C? I don't think that such a falsehood belief exists and therefore pydis disproves nothing.
  - jnwatson 6y ago
    You're missing the point. Obviously, you can write your application in Cython and have decent performance. The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.
ketzu 6y ago
I get the point of hiredis and uvloop and agree to some aspect, but including the python hashmap implementation as "implemented in C" feels like claiming python programs do not really exist because in the end it's implemented in C/Assembler/Running CPU instructions.
- didibus 6y ago
  I think that's because people are kind of forgetting what the concrete thing being discussed even is. A programing language can be thought of in the asbtract, as simply a syntax for computation and programming. But if you go in the concrete, a programming language implementation translates language into machine code for a particular machine. From that angle, the Cython interpreter/compiler is pretty bad at doing that. This experiment actually shows it. It is 40% slower even if all it does is orchestrates the calls between C code. This is true of hash map as well. Python is so bad at generating efficient machine code from its own language syntax and semantics, that it would be unreasonably slow to implement hash map in Python. Why? Because Python is really slow. In order to speed up programs written in Python, you need to write more of your program in some other faster language, such as C. Now if you reuse a bunch of components written in fast languages, like hash map being implemented in C, well you avoid the need to write the C code yourself. But it doesn't make Python any more efficient. In practice though, you might be able to not have to implement 90% of your application, because there exist C libraries that do most of what you already need. Thus only 10% of your application will have degraded performance due to using Python. And for that 10%, it might be Python is a great choice, because you like the language and productivity, readability and all it gives you in order to implement that 10%, and can deal with 10% of the code being slower. But if there doesn't exist existing fast libraries you can leverage, well more and more of your program will be Python, and thus it'll get slower and slower, unless you start using C yourself and implementing your own C code libraries. Yes, that's true of hash map as well. For example, if for any reason you needed a hash map that behaved differently than the standard one, you would most likely need to write it in C to get any performance out of it. Writing it in Python would be just too slow. So to recap, Python is a slow language, because there are no known interpreter/compiler for it that generate fast machine code for some real machine. This is not a falsehood, but the current reality. But it turns out most applications can leverage existing C libraries for a large amount of their functionality, thus making it that you can build an app that is reasonably performant, with the only code you need to personally implement being implemented in Python.

pjmlp 6y ago

A better point would be .NET 5 server implementation of gRPC. https://devblogs.microsoft.com/aspnet/grpc-performance-improvements-in-net-5/ https://devblogs.microsoft.com/aspnet/grpc-performance-impro... However Python isn't never going to get there.

nemetroid 6y ago

I think the use of uvloop is reasonable, since it's not something domain specific to Redis. But doesn't the inclusion of hiredis invalidate the entire point? It's 7500 lines of C specifically for working with Redis.

quietbritishjim 6y ago
I agree with this. If you write an application in Python that spends the majority of the time in C-based extensions – for example if I were to write a differential equation GUI using numpy and pyqt – then I think it's still valid to say that I wrote the application in Python. In fact even most of the standard library modules are written in C (e.g. zipfile, io, sqlite3) so it's hard to avoid this. You could say that no program is ever really written in Python but I don't think that's a useful point to make.[1] But if you write a load of C code that's very specific to your application, wrap it in a Python extension, then call it from Python, then I don't think you could argue that you "wrote it in Python". If someone happened to have written that application-specific extension module for you then the situation is not materially different. ([1] A more useful motion is whether your code is vectorised. That is, if you want to do something with a data structure (e.g. mutate each element, find an element, do matrix multiplication), do you have to loop over the elements in Python or do you make a single call into a library that does the looping in C? Either way you're still ultimately going to be calling C code, if nothing else in the Python interpreter itself, but the distinction is obviously still important.)
- formerly_proven 6y ago
  The Python zipfile module is actually just that, Python. It performs well enough, with compression zlib is the limiting factor anyway.
  - quietbritishjim 6y ago
    OK, by my own rules that means it's "written in Python" then. But in that case what I actually meant is that majority of time in calls to zipfile is spent in pure C, not in the Python interpreter loop, and if it's based on zlib then that agrees with what I meant. (There is an exception I didn't mention: if the zip is encrypted then the zipfile module falls back to pure Python code. The difference is very noticeable!)
lr4444lr 6y ago
uvloop also unfairly lets Python take credit for concurrent request handling performance too though, no?

ufo 6y ago

I'm not sure I see what is the problem here. Gluing together pieces of C code is one of the main reasons to use a scripting language like Python.

jpgvm 6y ago
Yes but if to achieve good performance in Python you have to implement all the domain specific code (in this case the redis protocol) in C then you: a) aren't writing Python b) invalidates all of the points for writing in Python. i.e speed of implementation, etc.
timlatim 6y ago
The problem is how the author frames it in the README: "an experiment to disprove some of the falsehoods about performance and optimisation regarding software and interpreted languages in particular", "unfortunately many programmers [...] spend countless hours by making life harder for themselves in the name of marginal performance gains, often trading many other conveniences (such as type safety)". Between the heavy use of C libraries and correctness issues (https://github.com/boramalper/pydis/issues/12 https://github.com/boramalper/pydis/issues/12), I don't see how this project proves that Python is faster or safer than other languages. That being said, the speed with which it was developed can certainly be seen as Python's strong point.
- noname120 6y ago
  It's a failed experiment — an experiment nonetheless.
- kortex 6y ago
  I definitely see both sides. Python is meant to be used exactly like this. It's an exemplar of python-as-library-glue. I also saw the title and clicked on it because I thought to myself "no fricken way...in pure python?" It's a touch baity.

secondcoming 6y ago

But the operations are all done on Python's data structures?

VWWHFSfQ 6y ago
aren't they also written in c though
- secondcoming 6y ago
  Ya. This whole post confused me. I thought it was claiming that the Python port was faster. It's too early in the morning.
- boramalper 6y ago
  That's taking it too far though isn't it? I mean, gcc is written in C++ too so what? :) You might also be interested in my other comment: https://news.ycombinator.com/item?id=25101200 https://news.ycombinator.com/item?id=25101200
  - rfoo 6y ago
    While what you said is true in general, I believe it is debatable in this case, as Redis (at least the subset implemented in this experiment) is something very very special, it is essentially a hashmap attached to the network. There aren't much stuff left if you offload the "hashmap" part and the "attached to the network" (I/O + protocol parser) part. In other words, in this case the hashmap implementation itself is the business logic of your application.

mattlondon 6y ago

> All the performance critical code is implemented in C. Isn't this why people like python? Just glue together some libraries that someone else has written and then you write a viral blog/twitter post about your <insert trendy thing> that you wrote in just 10 lines of code? Seriously is 't this the main reason people use it? I.e. it has a big library of stuff so it is easy for beginners and/or non-programmers to use? I'm struggling to think of any other reasons to deliberately pick python to start something new in, although it is at least a bit more same now with types.

stingraycharles 6y ago
All the reasons you list are indeed reasons to use Python, but none of them require performance critical code to be written in C.
melenaboija 6y ago
In my professional environment we use python because of the cost, definitely not because we want to post it anywhere. And this is not an opinion on how companies measure cost or how one language is better than another but maybe trying to explain, and I think underestimate, popularity of Python needs a little bit more insight.
jefftk 6y ago
Python is an extremely expressive language, that lets you prototype and experiment very rapidly. In many cases, with some cleanups, refactorings, and calls to C modules like numpy, you can convert those prototypes into sufficiently performant real implementations. Here's a recent write-up of mine where almost all of the time is spent in C extensions but where it was much faster to develop and code in python: https://www.jefftk.com/p/sharding-the-brigade https://www.jefftk.com/p/sharding-the-brigade
HelloNurse 6y ago
The important technical question is how good a Redis replacement can be built with 250 lines of Python "glue code". If performance is good enough, where are robustness and features? How large are the effort gap and the quality gap compared to the original Redis?

deleted 6y ago

[deleted]

The_rationalist 6y ago

It could be significantly faster if using io_uring

mhh__ 6y ago

The really annoying thing is is that you can quite happily write shorter code in D than in Python and have it run as fast as C++. One of the reasons D compiles so fast is because of this - the compiler isn't even specifically optimized for speed. Everyone seems to think that writing expressive code is slow, when really it's because Python is slow and C++ is a lumbering behemoth. Python isn't even a high-level language, think about it - it's higher level than C but the capability it has for abstraction it gives you is really just interpreted C with classes.

krelian 6y ago
Why did D never take off?
- mhh__ 6y ago
  Bad timing and the GC

boramalper 6y ago

Hi, author here! Copying my comment from the previous submission[x]: This was a quite common criticism over r/Python[0] as well, saying that using C modules and then benchmarking "Python" with C isn't fair. Perhaps it was my wording that caused this confusion, for which I am sorry, but I never meant to compare "pure" Python with C. The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. [0]: https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redis_clone_in_python_3_to_disprove_some/ https://www.reddit.com/r/Python/comments/awav6k/pydis_a_redi... [1]: It's of course controversial whether 60% throughput is considerable or not. [x]: https://news.ycombinator.com/item?id=19293590 https://news.ycombinator.com/item?id=19293590

rualca 6y ago
> The point I am trying to make is that, Python with C extensions can be as considerably[1] performant as C code for network or memory bound tasks. You don't make that point. You instead make the point that calling C code with glue code is slower than C code alone, and the performance hit of using Python as glue code in your case is a 2x slowdown.
tutfbhuf 6y ago
From your README.md: > pydis is an experiment to disprove some of the falsehoods about performance and your comment: > The point I am trying to make is that, Python with C extensions can be as considerably performant as C code for network or memory bound tasks. So your point is regarding the falsehood belief that python with heavy use of C extensions is (not) much slower (in this case ~60%) than C? I don't think that such a falsehood belief exists and therefore pydis disproves nothing.
- jnwatson 6y ago
  You're missing the point. Obviously, you can write your application in Cython and have decent performance. The entire purpose is to demonstrate that you can build a reasonable performance application built on high performance libraries and stick with idiomatic Python code.

ketzu 6y ago

I get the point of hiredis and uvloop and agree to some aspect, but including the python hashmap implementation as "implemented in C" feels like claiming python programs do not really exist because in the end it's implemented in C/Assembler/Running CPU instructions.

didibus 6y ago
I think that's because people are kind of forgetting what the concrete thing being discussed even is. A programing language can be thought of in the asbtract, as simply a syntax for computation and programming. But if you go in the concrete, a programming language implementation translates language into machine code for a particular machine. From that angle, the Cython interpreter/compiler is pretty bad at doing that. This experiment actually shows it. It is 40% slower even if all it does is orchestrates the calls between C code. This is true of hash map as well. Python is so bad at generating efficient machine code from its own language syntax and semantics, that it would be unreasonably slow to implement hash map in Python. Why? Because Python is really slow. In order to speed up programs written in Python, you need to write more of your program in some other faster language, such as C. Now if you reuse a bunch of components written in fast languages, like hash map being implemented in C, well you avoid the need to write the C code yourself. But it doesn't make Python any more efficient. In practice though, you might be able to not have to implement 90% of your application, because there exist C libraries that do most of what you already need. Thus only 10% of your application will have degraded performance due to using Python. And for that 10%, it might be Python is a great choice, because you like the language and productivity, readability and all it gives you in order to implement that 10%, and can deal with 10% of the code being slower. But if there doesn't exist existing fast libraries you can leverage, well more and more of your program will be Python, and thus it'll get slower and slower, unless you start using C yourself and implementing your own C code libraries. Yes, that's true of hash map as well. For example, if for any reason you needed a hash map that behaved differently than the standard one, you would most likely need to write it in C to get any performance out of it. Writing it in Python would be just too slow. So to recap, Python is a slow language, because there are no known interpreter/compiler for it that generate fast machine code for some real machine. This is not a falsehood, but the current reality. But it turns out most applications can leverage existing C libraries for a large amount of their functionality, thus making it that you can build an app that is reasonably performant, with the only code you need to personally implement being implemented in Python.