11 ms·

Redis re-implemented in Rust

400 points by wmwragg 11y ago

wyaeld 11y ago
The readme says its a learning project. Its a very interesting piece of work though. I'll be interested to see Antirez's view on the trade-offs between C and Rust for this.
- seppo0010 11y ago
  He is at least curious. https://twitter.com/antirez/status/611189939519229952 https://twitter.com/antirez/status/611189939519229952
undefined0 11y ago
For me, Redis was the first software written in C which I could easily customize with additional features (as I have low C knowledge). It was written beautifully. I've been learning Rust, I certainly find learning Rust easier than C and I like the fact that I can dive into software written in Rust without worrying about GC. You've done the best of both worlds for me by writing Redis in Rust. With that said, I'm still having an easier time reading Redis in C over your code as you are lacking comments and well named function/variable names. I admire your work nonetheless.
- seppo0010 11y ago
  That's a fair criticism, and well taken, but keep in mind I'm learning the language as I go, and most of the commits are just rewriting things because they were suboptimal, not idiomatic, or hard to read. At this stage, I would consider adding comments wasteful. I also have no intention of making this project live as long or have as many users as Redis does.
  - deleted 11y ago
    [deleted]
  - illumen 11y ago
    Learning to write readable code is a good thing I think worth the effort. Can rust be readable?
    - seppo0010 11y ago
      If you want good examples of Rust code, you should probably look into other repositories. Today I was checking out how Rust's HashMap works and I found it quite readable: https://github.com/rust-lang/rust/blob/master/src/libstd/collections/hash/map.rs#L1115 https://github.com/rust-lang/rust/blob/master/src/libstd/col... You can also look at `mio`, the asynchronous IO library that's popular in Rust: https://github.com/carllerche/mio/blob/master/src/notify.rs#L148 https://github.com/carllerche/mio/blob/master/src/notify.rs#...
      - anonfunction 11y ago
        I would hardly say this is readable: fn search_entry_hashed<'a, K: Eq, V>(table: &'a mut RawTable<K,V>, hash: SafeHash, k: K) -> Entry<'a, K, V>
        rmc 11y ago
        Depends if you know Rust. The language can do things with types and memory that (say) something like Python or Ruby don't do, which means Rust needs extra syntax to mark those things. Most of the <> or ' are to do with types or memory safety.
        yaongi 11y ago
        It's alright.
        pygy_ 11y ago
        I don't know Rust, and the only thing opaque to me in that line is the `'a` syntax. I suppose it is related to memory safety. In plain English: `search_entry_hashed` is a parametric function. It works in a type safe way on any kind of K and V. It takes as arguments a reference to a mutable hash table, a hash function and a key, and returns an `Entry` object whose precise type depends on the type of the parameters. You could hardly express that more succinctly. You can't understand it by skimming it, but it is a complex definition.
r0naa 11y ago
Impressive! Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds? To clarify, my metric for "value" is usefulness to other people. That is, without considering the (interesting) learning opportunity that it represent for the author. For example, someone developed a Python interface to the Stanford Core-NLP library (written in Java). Would re-writing the Core NLP library to Python be useful to the community? How to figure what are people needs? I am asking because while I think it would be ton of fun and allow me to learn a lot, I also value building useful software and re-writing a whole system sounds like an overkill except for a few very niche cases.. And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it. Looking forward to hear what HNers think! edit: clarified ambiguities
- themckman 11y ago
  The README answers this: To learn Rust. Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.
  - r0naa 11y ago
    Sorry if I wasn't clear, but I am looking for a more general answer! I would like to know in which case it is useful (to other people) and discuss it's value comparatively to writing interfaces to other languages.
    - seppo0010 11y ago
      I had no intention of making the end result useful, but I run into interesting problems. First, I wanted to make it as pure Rust as possible. I tried to avoid UNIX specific code, and since there is no library with Windows support for asynchronous IO in Rust, I was pushed into spawning thread and blocking waiting for client data. I quickly noticed that the benchmark was way below Redis (around 60% of ops/sec with 50 clients). But then someone point out to me[1] that I was running tests in a machine with two cores, and this actually may be better for machines with multiple cores[2]. I have yet to try it out and benchmark the results. So far Rust API was disappointing for network operations. For example, `TcpStream.read()`[3] and `TcpListener.incoming()`[4] do not have a timeout. Maybe because its development is driven for Servo and not for servers. I have thought about doubling down on multithreading and instead of a global database lock as rsedis is using now, having one per key (or some other arbitrary partition), and having concurrent operations, which is hard to do safely in C. But I have not gotten there yet. [1] https://github.com/jonhoo/rucache/issues/2 https://github.com/jonhoo/rucache/issues/2 [2] https://github.com/jonhoo/volley/ https://github.com/jonhoo/volley/ [3] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream... [4] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListener.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListen...
kibwen 11y ago
Since this seems to be just a learning project, note as well that there exist Rust bindings to Redis itself, from Armin Ronacher (though I'm not sure if they've yet been updated to work on 1.0): https://github.com/mitsuhiko/redis-rs https://github.com/mitsuhiko/redis-rs
- the_mitsuhiko 11y ago
  Yep, works with 1.0.
shmerl 11y ago
I was just thinking, that Rust is a great candidate for big data processing tools. So much more than Java (which is annoyingly used a lot there). Something like Spark and HDFS should be implemented in Rust.
- maxdemarzi 11y ago
  Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
  - shmerl 11y ago
    In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
    - deleted 11y ago
      [deleted]
    - maxdemarzi 11y ago
      See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
      - shmerl 11y ago
        I'm talking about really big data, which requires hundreds of nodes just to store it and etc.
thedufer 11y ago
Before anyone starts using this as a Redis replacement on Windows as the readme suggests, take a look at the TODO file. Notable missing features include: - maxmemory key eviction - hash values - ~2/3 of the set operators - multi/exec - lua scripting This is an interesting and potentially useful effort, but a replacement for Redis it is not.
- seppo0010 11y ago
  Also notice that the main developer only has two months of experience with Rust, so it is probably not as stable and well tested as Redis. As stated in the README, the main goal is to learn Rust.
- shankun 11y ago
  By the way, if you are looking for a production-quality Windows port of Redis, there is a fork available at https://github.com/MSOpenTech/redis https://github.com/MSOpenTech/redis. We (Microsoft) provide it in production as Azure's cache service today, and are committed to continuing to work on it.
  - kawsper 11y ago
    Although it sparked some debate when Microsoft ported Redis to Win32 with libuv (http://oldblog.antirez.com/post/redis-win32-msft-patch.html http://oldblog.antirez.com/post/redis-win32-msft-patch.html) I am impressed by their commitment that the fork is still going 4 years later.
  - netcraft 11y ago
    I want to thank you for that work and look forward to 3.0 there.
  - ddlutz 11y ago
    Are you part of the Azure cache service? I was an intern the the Edge Caching and Storage team and joining back fulltime next month. If things are the same way they were it would be worth it exploring using Redis for our cache and I'd like to talk details.
sudhirj 11y ago
I'm try the same thing for similar reasons in Go, but I'm wondering if at some point a Go version would perform better than C. On a machine with a large number of cores, perhaps? GitHub.com/sudhirj/restis Also wondering if some rethinking is possible - would a HTTP interface a la DynamoDB be more useful? Can complexity and performance be increased by using a purely memory backend with no disk persistence? If there were pluggable back ends would a Postgres or DynamoDB back end be more useful for terabytes / petabytes of data? Is the beauty of Redis the API or the implementation?
- pzduniak 11y ago
  https://github.com/siddontang/ledisdb https://github.com/siddontang/ledisdb
- alexchamberlain 11y ago
  Ah, HTTP is your hammer
- endymi0n 11y ago
  > but I'm wondering if at some point a Go version would perform better than C. The answer is "no" with a certain amount of probability. Redis isn't single threaded by lack of capability, but by design. Concurrency for multiple CPUs will actually slow down a lot of the stuff you see, as you will need to introduce locking mechanisms. Also, garbage collection is highly tuned and customized in Redis to the use case of an in-memory-DB (in stark contrast to usual allocation patterns of an application), up to the point where it's almost impossible to replicate the performance in a garbage collected language. I love Go and we're a 100% Go (and Angular) shop, but for an in-memory DB it wouldn't be a sane choice.
- vidarh 11y ago
  You can turn off disk persistence in Redis. My main usage of Redis is with disk persistence turned off (it handles a few hours worth of samples of data that we don't care if we lose - we care about having long term averages only of the data in question). There should be minimal overhead from having the capability in Redis due to the way it implements disk snapshots (RDB snapshots are done by fork()'ing and relying on copy on write to let the main process keep on doing its thing while the child process writes the snapshot, so the main process doesn't need to care; other than that Redis offers logging/journalling of changes, but the cost of having that as an option is trivial if it's switched off). Having pluggable backends for things like Postgres or DynamoDB seems a bit at odds with the purpose of Redis, which is exactly that you pay the (very low) cost of in-memory manipulation of simple data structures, though if a single Redis server could partition the key space between plugins, it might potentially be useful by letting you e.g. move keys between backends and still access them transparently to the client. E.g. for the samples I mentioned above, we roll up and transfer data to a CouchDB instance for archival now (doesn't matter that it's CouchDB really - we just need a persistent key-value store; Postgres or DynamoDB would also both have worked fine), but if I could archive them while still having them visible in a Redis-like server together with the in-memory keys, that'd make the client a tiny bit simpler. For most Redis usage, I think paying the cost of connection setup and teardown and sending HTTP headers etc. would likely slow things down immensely. At least it would for my usage. Having a HTTP interface as an addition might be useful in some scenarios to enable new use cases, but as a replacement for the current Redis API would be a disaster. If you want to explore alternative interfaces, I'd instead suggest going in the opposite direction, and experimenting with a UDP interface. In a typical data centre setting packet loss is low enough that while you'd need retry logic, it wouldn't necessarily get exercised much in normal situations. (On the other hand, for the typical request/reply cycle it might very well not give any benefits vs. tcp in most scenarios where multiple request/replies are done over a single connection and thus amortising the connection setup cost - would be interesting to benchmark, though)
resca79 11y ago
I like this kind of project. But the use case of redis it's a little bit exstream, I mean that the main feature of redis is the speed and the way how the memory consuption is handled. If this requirements are not satisfied, it is only a very good way to learn Rust( as the author goal) and the redis internal.
ahmetmsft 11y ago
Care to post details about this? Is this actually fast? Does it implement all features and guarantees of redis? Should anybody actually use this in production (maybe because it works on Windows)? Is it well tested? Looks like a really cool effort but authors of open source projects often think people would read the code and figure out all, the truth is people usually look at what's in the readme and that's all the attention span most people are going to have. My 2c: improve your README.md.
- detaro 11y ago
  He links a list of missing stuff in the readme. And if you read "Why? To learn rust" and ask "should I use this in production"...
GeertVL 11y ago
So how do you re-implement something like Redis in another language? Is it more of a translation job or do you start with splitting the concepts and try to implement it. Or take the idea and go your own way with implementing it?
vamitrou 11y ago
Is it compatible with the .rdb redis dumps?
beyondcompute 11y ago
Spectacular! Could you add synchronous replication though? And coalescing queries (so that entire system processes queries in batches, say 300 times per second)?
unfamiliar 11y ago
Could somebody give me a tl;dr on Redis? I keep hearing about it but from the summary I can't tell what kind of applications it is being used for.
- iagooar 11y ago
  Put key & read key with straight-forward abstractions. Simple and beautiful. It can be used for caching, queues and for applications with volatile data.
- kaitnieks 11y ago
  It's a memory cache to take some load off your DB. The good thing about Redis is the useful data structures it supports: lists, sets, hashes, bitmaps and somewhat more specific sorted sets.
- aaggarwal 11y ago
  From their official github page, Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, HyperLogLogs, Bitmaps. It simply means that the key-value store is directly loaded into the memory (RAM) and is available for fast access, but the data is retained (persistent) even after the application is closed. It is usually used as cache store, queuing messages to communicate with different processes locally or distributed.
  - rakoo 11y ago
    To add to that, Redis is a TCP server so you can speak to it from multiple processes on multiple machines (and Redis will easily support huge loads) Its data structure cover a good part of what you'd need with generic data structures, which makes Redis an easy way to do the logic of, say, List intersection of friends common between multiple people, sorted set of goods ranked by their amount, all of this shared with other processes. Redis also offers pubsub capabilities in two forms: - A standard PUB/SUB couple which does what you think it does - Blocking pop on a list for a client, and a push for another client, which will "wake up" the first one with the value. It's a very versatile swiss knife.
vicpara 11y ago
Why would someone do that? To what end? Why isn't anyone re-writing Redis in assembler to have it kick ass like pros? Can you write Windows in rust?
- derefr 11y ago
  What I'm personally really surprised about is that nobody's rewriting Redis as a unikernel to clear away all the OS context-switching/networking overhead from its basic operations.
  - Jweb_Guru 11y ago
    Redis is already leaving plenty of performance on the table, e.g. by not having any concurrent shared memory data structures (the fastest concurrent hash tables achieve better throughput even on inserts than the fastest single-threaded ones). It does this in the name of implementation simplicity. People focused on implementation simplicity don't generally abandon the operating system.
    - derefr 11y ago
      People run Redis mostly in Linux-in-a-VM (with Redis being the only "user-serving" process) already, though, no? I would think Redis-as-the-entire-VM would be less to think about, operation-wise, at least if your cloud or data-center templates its VMs with something like EC2 AMIs. You would just launch a "Redis appliance" AMI and move on. It's a feeling less of maintaining boxes, and more equivalent to paying a Redis-as-a-Service provider.
- itamarhaber 11y ago
  The "Why" is @seppo010's to answer (but having it run as is on all OSs is a big plus for one). As for writing it in Assembler, that makes little practical sense since Redis is written in (ANSI) C and it quite well optimized. In fact, if you profile Redis you'll see that very little time is actually spent by the code itself - OS, storage and network are the real bottlenecks usually.
- 11y ago
clu3 11y ago
Man you should have named it Rudis

400 points by wmwragg 11y ago

The readme says its a learning project. Its a very interesting piece of work though. I'll be interested to see Antirez's view on the trade-offs between C and Rust for this.

seppo0010 11y ago
He is at least curious. https://twitter.com/antirez/status/611189939519229952 https://twitter.com/antirez/status/611189939519229952

undefined0 11y ago

For me, Redis was the first software written in C which I could easily customize with additional features (as I have low C knowledge). It was written beautifully. I've been learning Rust, I certainly find learning Rust easier than C and I like the fact that I can dive into software written in Rust without worrying about GC. You've done the best of both worlds for me by writing Redis in Rust. With that said, I'm still having an easier time reading Redis in C over your code as you are lacking comments and well named function/variable names. I admire your work nonetheless.

seppo0010 11y ago
That's a fair criticism, and well taken, but keep in mind I'm learning the language as I go, and most of the commits are just rewriting things because they were suboptimal, not idiomatic, or hard to read. At this stage, I would consider adding comments wasteful. I also have no intention of making this project live as long or have as many users as Redis does.
- deleted 11y ago
  [deleted]
- illumen 11y ago
  Learning to write readable code is a good thing I think worth the effort. Can rust be readable?
  - seppo0010 11y ago
    If you want good examples of Rust code, you should probably look into other repositories. Today I was checking out how Rust's HashMap works and I found it quite readable: https://github.com/rust-lang/rust/blob/master/src/libstd/collections/hash/map.rs#L1115 https://github.com/rust-lang/rust/blob/master/src/libstd/col... You can also look at `mio`, the asynchronous IO library that's popular in Rust: https://github.com/carllerche/mio/blob/master/src/notify.rs#L148 https://github.com/carllerche/mio/blob/master/src/notify.rs#...
    - anonfunction 11y ago
      I would hardly say this is readable: fn search_entry_hashed<'a, K: Eq, V>(table: &'a mut RawTable<K,V>, hash: SafeHash, k: K) -> Entry<'a, K, V>
      - rmc 11y ago
        Depends if you know Rust. The language can do things with types and memory that (say) something like Python or Ruby don't do, which means Rust needs extra syntax to mark those things. Most of the <> or ' are to do with types or memory safety.
      - yaongi 11y ago
        It's alright.
      - pygy_ 11y ago
        I don't know Rust, and the only thing opaque to me in that line is the `'a` syntax. I suppose it is related to memory safety. In plain English: `search_entry_hashed` is a parametric function. It works in a type safe way on any kind of K and V. It takes as arguments a reference to a mutable hash table, a hash function and a key, and returns an `Entry` object whose precise type depends on the type of the parameters. You could hardly express that more succinctly. You can't understand it by skimming it, but it is a complex definition.

r0naa 11y ago

Impressive! Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds? To clarify, my metric for "value" is usefulness to other people. That is, without considering the (interesting) learning opportunity that it represent for the author. For example, someone developed a Python interface to the Stanford Core-NLP library (written in Java). Would re-writing the Core NLP library to Python be useful to the community? How to figure what are people needs? I am asking because while I think it would be ton of fun and allow me to learn a lot, I also value building useful software and re-writing a whole system sounds like an overkill except for a few very niche cases.. And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it. Looking forward to hear what HNers think! edit: clarified ambiguities

themckman 11y ago
The README answers this: To learn Rust. Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.
- r0naa 11y ago
  Sorry if I wasn't clear, but I am looking for a more general answer! I would like to know in which case it is useful (to other people) and discuss it's value comparatively to writing interfaces to other languages.
  - seppo0010 11y ago
    I had no intention of making the end result useful, but I run into interesting problems. First, I wanted to make it as pure Rust as possible. I tried to avoid UNIX specific code, and since there is no library with Windows support for asynchronous IO in Rust, I was pushed into spawning thread and blocking waiting for client data. I quickly noticed that the benchmark was way below Redis (around 60% of ops/sec with 50 clients). But then someone point out to me[1] that I was running tests in a machine with two cores, and this actually may be better for machines with multiple cores[2]. I have yet to try it out and benchmark the results. So far Rust API was disappointing for network operations. For example, `TcpStream.read()`[3] and `TcpListener.incoming()`[4] do not have a timeout. Maybe because its development is driven for Servo and not for servers. I have thought about doubling down on multithreading and instead of a global database lock as rsedis is using now, having one per key (or some other arbitrary partition), and having concurrent operations, which is hard to do safely in C. But I have not gotten there yet. [1] https://github.com/jonhoo/rucache/issues/2 https://github.com/jonhoo/rucache/issues/2 [2] https://github.com/jonhoo/volley/ https://github.com/jonhoo/volley/ [3] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream... [4] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListener.html http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListen...

kibwen 11y ago

Since this seems to be just a learning project, note as well that there exist Rust bindings to Redis itself, from Armin Ronacher (though I'm not sure if they've yet been updated to work on 1.0): https://github.com/mitsuhiko/redis-rs https://github.com/mitsuhiko/redis-rs

the_mitsuhiko 11y ago
Yep, works with 1.0.

shmerl 11y ago

I was just thinking, that Rust is a great candidate for big data processing tools. So much more than Java (which is annoyingly used a lot there). Something like Spark and HDFS should be implemented in Rust.

maxdemarzi 11y ago
Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
- shmerl 11y ago
  In many cases distributed computation is needed when there is BIG data, which won't fit on a laptop like in that example. I.e. distribution is needed not only because computation can't be done on one node (i.e. it would take too long), but because data can't possibly fit on any one node.
  - deleted 11y ago
    [deleted]
  - maxdemarzi 11y ago
    See his next post: http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... How big are the jobs being done? I remember reading a statistic[1] that is a couple of years old that 50% of the big data jobs were less than 10 GB. 90% was less than 100 GB and only something like 1% was bigger than a TB. You can fit multiple terabytes on a single machine. I wonder if anyone has new statistics about this. [1]: http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf
    - shmerl 11y ago
      I'm talking about really big data, which requires hundreds of nodes just to store it and etc.

thedufer 11y ago

Before anyone starts using this as a Redis replacement on Windows as the readme suggests, take a look at the TODO file. Notable missing features include: - maxmemory key eviction - hash values - ~2/3 of the set operators - multi/exec - lua scripting This is an interesting and potentially useful effort, but a replacement for Redis it is not.

seppo0010 11y ago
Also notice that the main developer only has two months of experience with Rust, so it is probably not as stable and well tested as Redis. As stated in the README, the main goal is to learn Rust.
shankun 11y ago
By the way, if you are looking for a production-quality Windows port of Redis, there is a fork available at https://github.com/MSOpenTech/redis https://github.com/MSOpenTech/redis. We (Microsoft) provide it in production as Azure's cache service today, and are committed to continuing to work on it.
- kawsper 11y ago
  Although it sparked some debate when Microsoft ported Redis to Win32 with libuv (http://oldblog.antirez.com/post/redis-win32-msft-patch.html http://oldblog.antirez.com/post/redis-win32-msft-patch.html) I am impressed by their commitment that the fork is still going 4 years later.
- netcraft 11y ago
  I want to thank you for that work and look forward to 3.0 there.
- ddlutz 11y ago
  Are you part of the Azure cache service? I was an intern the the Edge Caching and Storage team and joining back fulltime next month. If things are the same way they were it would be worth it exploring using Redis for our cache and I'd like to talk details.

sudhirj 11y ago

I'm try the same thing for similar reasons in Go, but I'm wondering if at some point a Go version would perform better than C. On a machine with a large number of cores, perhaps? GitHub.com/sudhirj/restis Also wondering if some rethinking is possible - would a HTTP interface a la DynamoDB be more useful? Can complexity and performance be increased by using a purely memory backend with no disk persistence? If there were pluggable back ends would a Postgres or DynamoDB back end be more useful for terabytes / petabytes of data? Is the beauty of Redis the API or the implementation?

pzduniak 11y ago
https://github.com/siddontang/ledisdb https://github.com/siddontang/ledisdb
alexchamberlain 11y ago
Ah, HTTP is your hammer
endymi0n 11y ago
> but I'm wondering if at some point a Go version would perform better than C. The answer is "no" with a certain amount of probability. Redis isn't single threaded by lack of capability, but by design. Concurrency for multiple CPUs will actually slow down a lot of the stuff you see, as you will need to introduce locking mechanisms. Also, garbage collection is highly tuned and customized in Redis to the use case of an in-memory-DB (in stark contrast to usual allocation patterns of an application), up to the point where it's almost impossible to replicate the performance in a garbage collected language. I love Go and we're a 100% Go (and Angular) shop, but for an in-memory DB it wouldn't be a sane choice.
vidarh 11y ago
You can turn off disk persistence in Redis. My main usage of Redis is with disk persistence turned off (it handles a few hours worth of samples of data that we don't care if we lose - we care about having long term averages only of the data in question). There should be minimal overhead from having the capability in Redis due to the way it implements disk snapshots (RDB snapshots are done by fork()'ing and relying on copy on write to let the main process keep on doing its thing while the child process writes the snapshot, so the main process doesn't need to care; other than that Redis offers logging/journalling of changes, but the cost of having that as an option is trivial if it's switched off). Having pluggable backends for things like Postgres or DynamoDB seems a bit at odds with the purpose of Redis, which is exactly that you pay the (very low) cost of in-memory manipulation of simple data structures, though if a single Redis server could partition the key space between plugins, it might potentially be useful by letting you e.g. move keys between backends and still access them transparently to the client. E.g. for the samples I mentioned above, we roll up and transfer data to a CouchDB instance for archival now (doesn't matter that it's CouchDB really - we just need a persistent key-value store; Postgres or DynamoDB would also both have worked fine), but if I could archive them while still having them visible in a Redis-like server together with the in-memory keys, that'd make the client a tiny bit simpler. For most Redis usage, I think paying the cost of connection setup and teardown and sending HTTP headers etc. would likely slow things down immensely. At least it would for my usage. Having a HTTP interface as an addition might be useful in some scenarios to enable new use cases, but as a replacement for the current Redis API would be a disaster. If you want to explore alternative interfaces, I'd instead suggest going in the opposite direction, and experimenting with a UDP interface. In a typical data centre setting packet loss is low enough that while you'd need retry logic, it wouldn't necessarily get exercised much in normal situations. (On the other hand, for the typical request/reply cycle it might very well not give any benefits vs. tcp in most scenarios where multiple request/replies are done over a single connection and thus amortising the connection setup cost - would be interesting to benchmark, though)

resca79 11y ago

I like this kind of project. But the use case of redis it's a little bit exstream, I mean that the main feature of redis is the speed and the way how the memory consuption is handled. If this requirements are not satisfied, it is only a very good way to learn Rust( as the author goal) and the redis internal.

ahmetmsft 11y ago

Care to post details about this? Is this actually fast? Does it implement all features and guarantees of redis? Should anybody actually use this in production (maybe because it works on Windows)? Is it well tested? Looks like a really cool effort but authors of open source projects often think people would read the code and figure out all, the truth is people usually look at what's in the readme and that's all the attention span most people are going to have. My 2c: improve your README.md.

detaro 11y ago
He links a list of missing stuff in the readme. And if you read "Why? To learn rust" and ask "should I use this in production"...

GeertVL 11y ago

So how do you re-implement something like Redis in another language? Is it more of a translation job or do you start with splitting the concepts and try to implement it. Or take the idea and go your own way with implementing it?

vamitrou 11y ago

Is it compatible with the .rdb redis dumps?

beyondcompute 11y ago

Spectacular! Could you add synchronous replication though? And coalescing queries (so that entire system processes queries in batches, say 300 times per second)?

unfamiliar 11y ago

Could somebody give me a tl;dr on Redis? I keep hearing about it but from the summary I can't tell what kind of applications it is being used for.

iagooar 11y ago
Put key & read key with straight-forward abstractions. Simple and beautiful. It can be used for caching, queues and for applications with volatile data.
kaitnieks 11y ago
It's a memory cache to take some load off your DB. The good thing about Redis is the useful data structures it supports: lists, sets, hashes, bitmaps and somewhat more specific sorted sets.
aaggarwal 11y ago
From their official github page, Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, HyperLogLogs, Bitmaps. It simply means that the key-value store is directly loaded into the memory (RAM) and is available for fast access, but the data is retained (persistent) even after the application is closed. It is usually used as cache store, queuing messages to communicate with different processes locally or distributed.
- rakoo 11y ago
  To add to that, Redis is a TCP server so you can speak to it from multiple processes on multiple machines (and Redis will easily support huge loads) Its data structure cover a good part of what you'd need with generic data structures, which makes Redis an easy way to do the logic of, say, List intersection of friends common between multiple people, sorted set of goods ranked by their amount, all of this shared with other processes. Redis also offers pubsub capabilities in two forms: - A standard PUB/SUB couple which does what you think it does - Blocking pop on a list for a client, and a push for another client, which will "wake up" the first one with the value. It's a very versatile swiss knife.

vicpara 11y ago

Why would someone do that? To what end? Why isn't anyone re-writing Redis in assembler to have it kick ass like pros? Can you write Windows in rust?

derefr 11y ago
What I'm personally really surprised about is that nobody's rewriting Redis as a unikernel to clear away all the OS context-switching/networking overhead from its basic operations.
- Jweb_Guru 11y ago
  Redis is already leaving plenty of performance on the table, e.g. by not having any concurrent shared memory data structures (the fastest concurrent hash tables achieve better throughput even on inserts than the fastest single-threaded ones). It does this in the name of implementation simplicity. People focused on implementation simplicity don't generally abandon the operating system.
  - derefr 11y ago
    People run Redis mostly in Linux-in-a-VM (with Redis being the only "user-serving" process) already, though, no? I would think Redis-as-the-entire-VM would be less to think about, operation-wise, at least if your cloud or data-center templates its VMs with something like EC2 AMIs. You would just launch a "Redis appliance" AMI and move on. It's a feeling less of maintaining boxes, and more equivalent to paying a Redis-as-a-Service provider.
itamarhaber 11y ago
The "Why" is @seppo010's to answer (but having it run as is on all OSs is a big plus for one). As for writing it in Assembler, that makes little practical sense since Redis is written in (ANSI) C and it quite well optimized. In fact, if you profile Redis you'll see that very little time is actually spent by the code itself - OS, storage and network are the real bottlenecks usually.
11y ago

clu3 11y ago

Man you should have named it Rudis