10 ms·

Pydis – Redis clone in 250 lines of Python, for performance comparison

239 points by antman 6y ago

sillysaurusx 6y ago
Pydis code is awesome: https://github.com/boramalper/pydis/blob/0a51b6e31475dab083db1f87d69657dd5409526c/pydis/__main__.py https://github.com/boramalper/pydis/blob/0a51b6e31475dab083d... I've been a Python dev for roughly a decade and had no idea about asyncio.Protocol, asyncio.transports.Transport, or that asyncio's event loops could be used to create a TCP listening server. I always thought of asyncio as synonymous with async def and websockets. But perhaps the websockets should have clued me in that it's an actual server protocol. Between this and "On the beauty of Python's ExitStack" (https://www.rath.org/on-the-beauty-of-pythons-exitstack.html https://www.rath.org/on-the-beauty-of-pythons-exitstack.html) it's been a good week. I was hoping that it implemented a Redis protocol reader, but pydis uses hiredis.Reader(). I wonder if there's a pure python version... ./benchmark.sh produced some warnings related to a lack of CONFIG GET, so I added a basic one just now: https://github.com/boramalper/pydis/pull/13 https://github.com/boramalper/pydis/pull/13 The benchmarks are indeed pretty close to Redis on my MBP!
- user5994461 6y ago
  asyncio was introduced only recently, roughly python 3.4 and above. There wasn't anything for you to see. I'd wager it only became usable very recently since most development was stuck writing code compatible with python 2 and python 3, precluding the usage of asyncio.
- euiq 6y ago
  The problem with inheriting from asyncio.Protocol is that it ties your code to the callback-based low level asyncio API—I'm not surprised that this package doesn't have any tests.
lukehutch 6y ago
This is a straw man argument. Python's associative array code is written in C, not in Python, and Redis is written in C. So you're comparing C to C. The 40% loss in performance is due to Python being much slower at doing the stuff that's actually written in Python.
- dp2019 6y ago
  Not just that but hitting 50% or 60% of the target performance is usually not that difficult. It is always the last 10-20% of extra performance that are the ones really difficult to hit and the ones that might influence design decisions early on. Some of these frameworks really have tiny performance differences in the order of single digit percentages. Hitting 60% of any framework's performance is not really a feat.
- framecowbird 6y ago
  But isn't this exactly what the author is demonstrating? You can use the higher language of Python with its garbage collection, and since the critical parts are in C anyway, it ends up being competitive to the pure C implementation.
  - eecc 6y ago
    Well, 40% slower isn’t exactly what I’d call competitive...
    - coldtea 6y ago
      In what universe? People use Java, C#, Go, Node etc services that are 40% or more slower than equivalent C ones, and they're just fine with it...
      - wokwokwok 6y ago
        Yes, it’s probably fast enough to be perfectly usable. The point here is that implementing new features will be impossible, because this relies on an existing implementation that is not in python. It is therefore not a proof that arbitrary high performance applications can easily be written in python. It is simply an example of how high performance applications based on existing implementations someone else had written in another language can be wrapped in python.
        coldtea 6y ago
WJW 6y ago
Very cool! Making C integrations available with nice syntax and managed memory is such a huge win for Python. This POC glues together a hashmap implementation, async event loop and redis protocol parser (all in C) into a very readable (and quite performant) program. You could probably add pub/sub support in another 50 lines as well, since asyncio is already present.
- ttul 6y ago
  I appreciate the example code. I didn’t know that asyncio had a Protocol class that you can so easily just extend to make stuff like this. I’m saving this away for future work.
deleted 6y ago
[deleted]
oars 6y ago
Curious as I'm already in the industry, but I had a look at this person's GitHub profile and it says that he's still looking for graduate opportunities (finishing uni soon). Someone with this level of calibre should easily get a job after graduation right? Or are college graduates these days trained to a level where this type of project is expected hence it's still competitive to find an entry level role after college?
- capableweb 6y ago
  Getting a job as a programmer is less about being a savant programmer and more about your soft-skills like communication and connections, at least if you're aiming to work in a team environment. It helps to be a great programmer in order to get hired, but I never found that to be a requirement, you can always teach people to be great programmers. But if they are not great at communication or have other "personal faults" like easily irritable or arrogant, it's much harder to help them improve without spending a disproportionate amount of time helping them.
- accurrent 6y ago
  As a fresh grad I'd say the latter. Especially if you are applying for Software only roles in large web companies. Side projects often count for nothing in these cases. Its more important to be able to whiteboard.
- sillysaurusx 6y ago
  Landing a job has very little to do with programming skill and very much to do with networking and the way in which you present yourself and what the company is looking for. So the answer to your question depends entirely on external factors. Python jobs in general seem to be extremely rare compared to frontend JS and backend Rails, which I found disappointing several years ago. But python jobs do exist, and the author should find no trouble landing one with "Cloned Redis in 230 lines" as a bullet point. :)
  - user5994461 6y ago
    Jobs these days are just sending you back an automated coding test on HackerRank/Codility after you apply, so finding a job has a lot to do with having some programming skills. There's quite a bit of python jobs coming up in the past few years, if we're talking about US tech hubs. Location is everything.
deleted 6y ago
[deleted]
mrfusion 6y ago
I believe you can turn off garbage collection in python and manage it yourself? How much of a speed up would that give you?
- FartyMcFarter 6y ago
  How do you do that? Even if you can, it's unlikely to give you a 10-100x speedup, which other languages can often give you compared to Python. The garbage collector does not take anywhere near even half of the CPU time.
  - mrfusion 6y ago
    import gc gc.disable() Call gc.collect() when you’re at a good point to afford it and need it. Ive done this before in really deep inner loops where speed is important and shaved some significant time off. It’s also useful if you have something time critical and don’t want to be interrupted. Eg Profiling, etc.
    - fulafel 6y ago
      This affects only the cycle detecting part of Python's GC, most of the work is done by the refcounting system which you can't stop or pause.
  - mrfusion 6y ago
    To reply to your point. Others were saying the python is slower because of garbage collection. I’m just suggesting they try turning it off. Big fat downvote though :-(
EugeneOZ 6y ago
Maybe if author will try to write README in a less arrogant tone then performance will be even better, who knows.
the_duke 6y ago
This is essentially some Python glue for C code. It uses a C Redis client parser (hiredis), uvloop (Python wrapper for libuv) and the Python hashmap implementation. All the performance critical code is implemented in C. I appreciate the point that higher level languages can be more than fast enough for many use cases. A tremendous amount of software is written in Python, PHP, Ruby et al after all. But this is not a good way to make that argument, since it almost refutes it's own point: if performance matters, you apparently implement it in C/C++/Rust.
- pjmlp 6y ago
  A better point would be .NET 5 server implementation of gRPC. https://devblogs.microsoft.com/aspnet/grpc-performance-improvements-in-net-5/ https://devblogs.microsoft.com/aspnet/grpc-performance-impro... However Python isn't never going to get there.
- nemetroid 6y ago
  I think the use of uvloop is reasonable, since it's not something domain specific to Redis. But doesn't the inclusion of hiredis invalidate the entire point? It's 7500 lines of C specifically for working with Redis.
  - quietbritishjim 6y ago
    I agree with this. If you write an application in Python that spends the majority of the time in C-based extensions – for example if I were to write a differential equation GUI using numpy and pyqt – then I think it's still valid to say that I wrote the application in Python. In fact even most of the standard library modules are written in C (e.g. zipfile, io, sqlite3) so it's hard to avoid this. You could say that no program is ever really written in Python but I don't think that's a useful point to make.[1] But if you write a load of C code that's very specific to your application, wrap it in a Python extension, then call it from Python, then I don't think you could argue that you "wrote it in Python". If someone happened to have written that application-specific extension module for you then the situation is not materially different. ([1] A more useful motion is whether your code is vectorised. That is, if you want to do something with a data structure (e.g. mutate each element, find an element, do matrix multiplication), do you have to loop over the elements in Python or do you make a single call into a library that does the looping in C? Either way you're still ultimately going to be calling C code, if nothing else in the Python interpreter itself, but the distinction is obviously still important.)
antender 6y ago
BTW, first version of Redis was actually built in equally slow TCL and later rewritten in C.
Semaphor 6y ago
As there is much of the same criticism here as there was March 2019, you might find the last submission interesting: https://news.ycombinator.com/item?id=19287717 https://news.ycombinator.com/item?id=19287717
rcarmo 6y ago
I've had a nearly-pure Python "toy" implementation for a long time. The source for it isn't public, but it's based off this: https://github.com/rcarmo/miniredis https://github.com/rcarmo/miniredis I use it as a single-file, mock Redis server for CI testing :)
doteka 6y ago
This feels a bit like “Thinking quickly, Dave constructs a homemade megaphone using only some string, a squirrel, and a megaphone.”
atmin 6y ago
Reminds me of http://www.underengineering.com/2014/05/22/DIY-NoSql/ http://www.underengineering.com/2014/05/22/DIY-NoSql/ What I'd love to see in Pydis is durability option.
FartyMcFarter 6y ago
> falsehoods about performance and optimisation regarding software and interpreted languages in particular. At this point, I think this argument will fail to convince many. There are far too many people who have gotten 10x, 100x or even higher speedups by simply converting Python to other languages. That is without even trying to optimise the code much. CPython running idiomatic Python code is just slow, no amount of handwaving will change that.
sheeshkebab 6y ago
This hack seems to be missing a whole bunch of redis functionality that makes redis suitable for production workloads (clustering, persistence just for start). And I wonder what performance would be like once all that stuff is added in. Still, nice and easily readable program - great for education.
loxs 6y ago
Yeah, yeah, Python is fast... nevermind that my program (that does real work) that I ported from idiomatic Python to idiomatic Rust became around 140 times faster. Real world doesn't matter, synthetic benchmarks (with clever calling of C code) are the way to go :) PS. The Rust version is also easier to read, maintain and understand (for someone who groks algebraic types and Rust in general), not to mention a lot safer.
- throwdbaaway 6y ago
  Speaking of real world, in your case, would the Rust version come into existence without the Python version being written in the first place? Here's a real world example: https://github.com/sripathikrishnan/redis-rdb-tools https://github.com/sripathikrishnan/redis-rdb-tools - python, optional C library dependency to liblzf, fully-featured, pure-python performance is decent enough when using pypy https://github.com/badboy/rdb-rs https://github.com/badboy/rdb-rs - rust, inspired by the former, not fully-featured, abandoned, may or may not have great performance Rust is great, but I just want to contend that the real world picture is a lot more nuanced than how you painted it.
  - loxs 6y ago
    I don't know how I painted it in your eyes, but at least my intention was exactly to point out that the real world is nuanced. I never said that python should not exist. I celebrate its strong parts like being a great first language and maybe, if you know what and why you are doing, great language overall in some specific situations. My comment was mostly against trying to pretend that Python is fast, which it is definitely not, and everyone who knows another language can very easily prove it to themselves and to the world. Regarding this specific library... Redis is not really very useful to a Rust program, because one usually loses performance by using it, instead of gaining some (which would be the case for a Python program). One would either use native data-structures for simple things (read, where Redis would be useful) or a "real" database if something more complex is needed. If working with Redis (and its dump files) was some real priority, the rustlers would come up with something with equal quality to the python library you linked. Maybe something like this, but for Redis dump files: https://github.com/serde-rs/serde https://github.com/serde-rs/serde
ChrisMarshallNY 6y ago
Obligatory MonkeyUser: https://www.monkeyuser.com/2018/implementation/ https://www.monkeyuser.com/2018/implementation/
izietto 6y ago
You aren't benchmarking a Redis implementation, you are benchmarking a Redis wrapper
boramalper 6y ago
Hi again, author here! I am so pleased to see your interest, thanks a lot! I have addressed a few points here, and just updated the repo, so please keep that in mind while browsing the comments: 1. Some people found the README a bit arrogant/bratty --- which I agree, sorry! --- so I have edited it. 2. Some people found it clickbait-y it says Python whereas there are two C libraries (uvloop, and hiredis) for heavy-lifting the event loop and parsing (respectively.) I have added this fact as a disclaimer in the README, and ultimately it is up to you to decide whether that counts as cheating or not. :) 3. I have addressed some major correctness issues too (#12) but it is not 100% correct still. 4. This is a proof-of-concept that I wrote in a day (+ spent today fixing some bugs) that you are comparing to an 11 years old industry-grade software. =) Don't take the experiment too seriously! Kindly, Bora
- tbenst 6y ago
  Thanks for creating and sharing this! Always easier to criticize than create. I for one think your code reads as quite pythonic. Using convenient python wrappers for c libraries is the raison d'être for python in the first place. See numpy, pytorch, ...
- chrfrasco 6y ago
  I hope you didn't take too many of these comments to heart. You unfortunately got the standard HN comment section treatment – a delightful mix of nitpicking, assuming bad intent, and completely missing the point. I also hope you continue to share things on the internet – it's far easier to criticise than it is to create.
bb611 6y ago
I spend a lot of time giving advice to people changing careers into tech. I agree with a lot of the criticism here, but I think the author's choices are reflective of the new grad hiring climate. Hiring for any competitive tech job relies on having a stellar resume with impressive sounding projects, and ideally a degree from a top CS school. Interviewers have little time to review projects, so new grads market small projects like this as much more impressive than they are, knowing they'll only be asked about them in passing as a break from algorithm problems.
mrits 6y ago
One line bash file clone of redis: ./redis.exe
SergeAx 6y ago
There whas a moment when Tarantool[0] was looking for market niche and was checking Redis' turf. They published performance tests with 2-3% upside. It was done by a team of experienced C devs spending months trying hard to beat original. 40% is a very big gap for that matter. [0] https://www.tarantool.io/ https://www.tarantool.io/
adsharma 6y ago
The thing we need to time is - how long did it take the author to write the code vs writing in C. Number of lines of code is a proxy, but not always accurate. The emphasis on expressing ideas clearly, sometimes at the cost of performance has been a good bet for python in terms of data science fueled market share. But it does open the door to some competition. I'm also curious how this performs versus similar code in Julia https://github.com/markmo/HiRedis.jl https://github.com/markmo/HiRedis.jl This could help understand the sweet spot between runtime performance, time to code and understandability and type safety.
- adsharma 6y ago
  HiRedis.jl is a redis client, not a server. So not directly comparable without a server implementation.
jokethrowaway 6y ago
Well, 2x the speed to use a lower level language doesn't sound like a bad deal to me, especially given we have a few fast, modern languages which are as ergonomic as python

239 points by antman 6y ago

Pydis code is awesome: https://github.com/boramalper/pydis/blob/0a51b6e31475dab083db1f87d69657dd5409526c/pydis/__main__.py https://github.com/boramalper/pydis/blob/0a51b6e31475dab083d... I've been a Python dev for roughly a decade and had no idea about asyncio.Protocol, asyncio.transports.Transport, or that asyncio's event loops could be used to create a TCP listening server. I always thought of asyncio as synonymous with async def and websockets. But perhaps the websockets should have clued me in that it's an actual server protocol. Between this and "On the beauty of Python's ExitStack" (https://www.rath.org/on-the-beauty-of-pythons-exitstack.html https://www.rath.org/on-the-beauty-of-pythons-exitstack.html) it's been a good week. I was hoping that it implemented a Redis protocol reader, but pydis uses hiredis.Reader(). I wonder if there's a pure python version... ./benchmark.sh produced some warnings related to a lack of CONFIG GET, so I added a basic one just now: https://github.com/boramalper/pydis/pull/13 https://github.com/boramalper/pydis/pull/13 The benchmarks are indeed pretty close to Redis on my MBP!

user5994461 6y ago
asyncio was introduced only recently, roughly python 3.4 and above. There wasn't anything for you to see. I'd wager it only became usable very recently since most development was stuck writing code compatible with python 2 and python 3, precluding the usage of asyncio.
euiq 6y ago
The problem with inheriting from asyncio.Protocol is that it ties your code to the callback-based low level asyncio API—I'm not surprised that this package doesn't have any tests.

lukehutch 6y ago

This is a straw man argument. Python's associative array code is written in C, not in Python, and Redis is written in C. So you're comparing C to C. The 40% loss in performance is due to Python being much slower at doing the stuff that's actually written in Python.

dp2019 6y ago
Not just that but hitting 50% or 60% of the target performance is usually not that difficult. It is always the last 10-20% of extra performance that are the ones really difficult to hit and the ones that might influence design decisions early on. Some of these frameworks really have tiny performance differences in the order of single digit percentages. Hitting 60% of any framework's performance is not really a feat.
framecowbird 6y ago
But isn't this exactly what the author is demonstrating? You can use the higher language of Python with its garbage collection, and since the critical parts are in C anyway, it ends up being competitive to the pure C implementation.
- eecc 6y ago
  Well, 40% slower isn’t exactly what I’d call competitive...
  - coldtea 6y ago
    In what universe? People use Java, C#, Go, Node etc services that are 40% or more slower than equivalent C ones, and they're just fine with it...
    - wokwokwok 6y ago
      Yes, it’s probably fast enough to be perfectly usable. The point here is that implementing new features will be impossible, because this relies on an existing implementation that is not in python. It is therefore not a proof that arbitrary high performance applications can easily be written in python. It is simply an example of how high performance applications based on existing implementations someone else had written in another language can be wrapped in python.
      - coldtea 6y ago

WJW 6y ago

Very cool! Making C integrations available with nice syntax and managed memory is such a huge win for Python. This POC glues together a hashmap implementation, async event loop and redis protocol parser (all in C) into a very readable (and quite performant) program. You could probably add pub/sub support in another 50 lines as well, since asyncio is already present.

ttul 6y ago
I appreciate the example code. I didn’t know that asyncio had a Protocol class that you can so easily just extend to make stuff like this. I’m saving this away for future work.

deleted 6y ago

[deleted]

oars 6y ago

Curious as I'm already in the industry, but I had a look at this person's GitHub profile and it says that he's still looking for graduate opportunities (finishing uni soon). Someone with this level of calibre should easily get a job after graduation right? Or are college graduates these days trained to a level where this type of project is expected hence it's still competitive to find an entry level role after college?

capableweb 6y ago
Getting a job as a programmer is less about being a savant programmer and more about your soft-skills like communication and connections, at least if you're aiming to work in a team environment. It helps to be a great programmer in order to get hired, but I never found that to be a requirement, you can always teach people to be great programmers. But if they are not great at communication or have other "personal faults" like easily irritable or arrogant, it's much harder to help them improve without spending a disproportionate amount of time helping them.
accurrent 6y ago
As a fresh grad I'd say the latter. Especially if you are applying for Software only roles in large web companies. Side projects often count for nothing in these cases. Its more important to be able to whiteboard.
sillysaurusx 6y ago
Landing a job has very little to do with programming skill and very much to do with networking and the way in which you present yourself and what the company is looking for. So the answer to your question depends entirely on external factors. Python jobs in general seem to be extremely rare compared to frontend JS and backend Rails, which I found disappointing several years ago. But python jobs do exist, and the author should find no trouble landing one with "Cloned Redis in 230 lines" as a bullet point. :)
- user5994461 6y ago
  Jobs these days are just sending you back an automated coding test on HackerRank/Codility after you apply, so finding a job has a lot to do with having some programming skills. There's quite a bit of python jobs coming up in the past few years, if we're talking about US tech hubs. Location is everything.

deleted 6y ago

[deleted]

mrfusion 6y ago

I believe you can turn off garbage collection in python and manage it yourself? How much of a speed up would that give you?

FartyMcFarter 6y ago
How do you do that? Even if you can, it's unlikely to give you a 10-100x speedup, which other languages can often give you compared to Python. The garbage collector does not take anywhere near even half of the CPU time.
- mrfusion 6y ago
  import gc gc.disable() Call gc.collect() when you’re at a good point to afford it and need it. Ive done this before in really deep inner loops where speed is important and shaved some significant time off. It’s also useful if you have something time critical and don’t want to be interrupted. Eg Profiling, etc.
  - fulafel 6y ago
    This affects only the cycle detecting part of Python's GC, most of the work is done by the refcounting system which you can't stop or pause.
- mrfusion 6y ago
  To reply to your point. Others were saying the python is slower because of garbage collection. I’m just suggesting they try turning it off. Big fat downvote though :-(

EugeneOZ 6y ago

Maybe if author will try to write README in a less arrogant tone then performance will be even better, who knows.

the_duke 6y ago

This is essentially some Python glue for C code. It uses a C Redis client parser (hiredis), uvloop (Python wrapper for libuv) and the Python hashmap implementation. All the performance critical code is implemented in C. I appreciate the point that higher level languages can be more than fast enough for many use cases. A tremendous amount of software is written in Python, PHP, Ruby et al after all. But this is not a good way to make that argument, since it almost refutes it's own point: if performance matters, you apparently implement it in C/C++/Rust.

pjmlp 6y ago
A better point would be .NET 5 server implementation of gRPC. https://devblogs.microsoft.com/aspnet/grpc-performance-improvements-in-net-5/ https://devblogs.microsoft.com/aspnet/grpc-performance-impro... However Python isn't never going to get there.
nemetroid 6y ago
I think the use of uvloop is reasonable, since it's not something domain specific to Redis. But doesn't the inclusion of hiredis invalidate the entire point? It's 7500 lines of C specifically for working with Redis.
- quietbritishjim 6y ago
  I agree with this. If you write an application in Python that spends the majority of the time in C-based extensions – for example if I were to write a differential equation GUI using numpy and pyqt – then I think it's still valid to say that I wrote the application in Python. In fact even most of the standard library modules are written in C (e.g. zipfile, io, sqlite3) so it's hard to avoid this. You could say that no program is ever really written in Python but I don't think that's a useful point to make.[1] But if you write a load of C code that's very specific to your application, wrap it in a Python extension, then call it from Python, then I don't think you could argue that you "wrote it in Python". If someone happened to have written that application-specific extension module for you then the situation is not materially different. ([1] A more useful motion is whether your code is vectorised. That is, if you want to do something with a data structure (e.g. mutate each element, find an element, do matrix multiplication), do you have to loop over the elements in Python or do you make a single call into a library that does the looping in C? Either way you're still ultimately going to be calling C code, if nothing else in the Python interpreter itself, but the distinction is obviously still important.)

antender 6y ago

BTW, first version of Redis was actually built in equally slow TCL and later rewritten in C.

Semaphor 6y ago

As there is much of the same criticism here as there was March 2019, you might find the last submission interesting: https://news.ycombinator.com/item?id=19287717 https://news.ycombinator.com/item?id=19287717

rcarmo 6y ago

I've had a nearly-pure Python "toy" implementation for a long time. The source for it isn't public, but it's based off this: https://github.com/rcarmo/miniredis https://github.com/rcarmo/miniredis I use it as a single-file, mock Redis server for CI testing :)

doteka 6y ago

This feels a bit like “Thinking quickly, Dave constructs a homemade megaphone using only some string, a squirrel, and a megaphone.”

atmin 6y ago

Reminds me of http://www.underengineering.com/2014/05/22/DIY-NoSql/ http://www.underengineering.com/2014/05/22/DIY-NoSql/ What I'd love to see in Pydis is durability option.

FartyMcFarter 6y ago

> falsehoods about performance and optimisation regarding software and interpreted languages in particular. At this point, I think this argument will fail to convince many. There are far too many people who have gotten 10x, 100x or even higher speedups by simply converting Python to other languages. That is without even trying to optimise the code much. CPython running idiomatic Python code is just slow, no amount of handwaving will change that.

sheeshkebab 6y ago

This hack seems to be missing a whole bunch of redis functionality that makes redis suitable for production workloads (clustering, persistence just for start). And I wonder what performance would be like once all that stuff is added in. Still, nice and easily readable program - great for education.

loxs 6y ago

Yeah, yeah, Python is fast... nevermind that my program (that does real work) that I ported from idiomatic Python to idiomatic Rust became around 140 times faster. Real world doesn't matter, synthetic benchmarks (with clever calling of C code) are the way to go :) PS. The Rust version is also easier to read, maintain and understand (for someone who groks algebraic types and Rust in general), not to mention a lot safer.

throwdbaaway 6y ago
Speaking of real world, in your case, would the Rust version come into existence without the Python version being written in the first place? Here's a real world example: https://github.com/sripathikrishnan/redis-rdb-tools https://github.com/sripathikrishnan/redis-rdb-tools - python, optional C library dependency to liblzf, fully-featured, pure-python performance is decent enough when using pypy https://github.com/badboy/rdb-rs https://github.com/badboy/rdb-rs - rust, inspired by the former, not fully-featured, abandoned, may or may not have great performance Rust is great, but I just want to contend that the real world picture is a lot more nuanced than how you painted it.
- loxs 6y ago
  I don't know how I painted it in your eyes, but at least my intention was exactly to point out that the real world is nuanced. I never said that python should not exist. I celebrate its strong parts like being a great first language and maybe, if you know what and why you are doing, great language overall in some specific situations. My comment was mostly against trying to pretend that Python is fast, which it is definitely not, and everyone who knows another language can very easily prove it to themselves and to the world. Regarding this specific library... Redis is not really very useful to a Rust program, because one usually loses performance by using it, instead of gaining some (which would be the case for a Python program). One would either use native data-structures for simple things (read, where Redis would be useful) or a "real" database if something more complex is needed. If working with Redis (and its dump files) was some real priority, the rustlers would come up with something with equal quality to the python library you linked. Maybe something like this, but for Redis dump files: https://github.com/serde-rs/serde https://github.com/serde-rs/serde

ChrisMarshallNY 6y ago

Obligatory MonkeyUser: https://www.monkeyuser.com/2018/implementation/ https://www.monkeyuser.com/2018/implementation/

izietto 6y ago

You aren't benchmarking a Redis implementation, you are benchmarking a Redis wrapper

boramalper 6y ago

Hi again, author here! I am so pleased to see your interest, thanks a lot! I have addressed a few points here, and just updated the repo, so please keep that in mind while browsing the comments: 1. Some people found the README a bit arrogant/bratty --- which I agree, sorry! --- so I have edited it. 2. Some people found it clickbait-y it says Python whereas there are two C libraries (uvloop, and hiredis) for heavy-lifting the event loop and parsing (respectively.) I have added this fact as a disclaimer in the README, and ultimately it is up to you to decide whether that counts as cheating or not. :) 3. I have addressed some major correctness issues too (#12) but it is not 100% correct still. 4. This is a proof-of-concept that I wrote in a day (+ spent today fixing some bugs) that you are comparing to an 11 years old industry-grade software. =) Don't take the experiment too seriously! Kindly, Bora

tbenst 6y ago
Thanks for creating and sharing this! Always easier to criticize than create. I for one think your code reads as quite pythonic. Using convenient python wrappers for c libraries is the raison d'être for python in the first place. See numpy, pytorch, ...
chrfrasco 6y ago
I hope you didn't take too many of these comments to heart. You unfortunately got the standard HN comment section treatment – a delightful mix of nitpicking, assuming bad intent, and completely missing the point. I also hope you continue to share things on the internet – it's far easier to criticise than it is to create.

bb611 6y ago

I spend a lot of time giving advice to people changing careers into tech. I agree with a lot of the criticism here, but I think the author's choices are reflective of the new grad hiring climate. Hiring for any competitive tech job relies on having a stellar resume with impressive sounding projects, and ideally a degree from a top CS school. Interviewers have little time to review projects, so new grads market small projects like this as much more impressive than they are, knowing they'll only be asked about them in passing as a break from algorithm problems.

mrits 6y ago

One line bash file clone of redis: ./redis.exe

SergeAx 6y ago

There whas a moment when Tarantool[0] was looking for market niche and was checking Redis' turf. They published performance tests with 2-3% upside. It was done by a team of experienced C devs spending months trying hard to beat original. 40% is a very big gap for that matter. [0] https://www.tarantool.io/ https://www.tarantool.io/

adsharma 6y ago

The thing we need to time is - how long did it take the author to write the code vs writing in C. Number of lines of code is a proxy, but not always accurate. The emphasis on expressing ideas clearly, sometimes at the cost of performance has been a good bet for python in terms of data science fueled market share. But it does open the door to some competition. I'm also curious how this performs versus similar code in Julia https://github.com/markmo/HiRedis.jl https://github.com/markmo/HiRedis.jl This could help understand the sweet spot between runtime performance, time to code and understandability and type safety.

adsharma 6y ago
HiRedis.jl is a redis client, not a server. So not directly comparable without a server implementation.

jokethrowaway 6y ago

Well, 2x the speed to use a lower level language doesn't sound like a bad deal to me, especially given we have a few fast, modern languages which are as ergonomic as python