9 ms·

Edis: a Redis for larger-than-RAM datasets

147 points by nmcfarl 13y ago

glurgh 13y ago
> Edis is a protocol-compatable Server replacement for Redis, written in Erlang. Edis's goal is to be a drop-in replacement for Redis when persistence is more important than holding the dataset in-memory. Isn't the entire point of Redis that it's meticulously written in C by some Italian guy, specifically designed for in-memory datasets? Protocol compatible is one thing but maybe name it something with a greater edit distance?
nmcfarl 13y ago
This came up in the discussion of "Redis as the primary data store?" -> https://news.ycombinator.com/item?id=5620960 https://news.ycombinator.com/item?id=5620960 And I was wondering if anyone was using it - and what they thought… -- Also the github repo link: https://github.com/inaka/edis https://github.com/inaka/edis
- chuhnk 13y ago
  The more info link to the pdf for Erlang Factory 2012 talk appears to 404 but I found a direct source http://www.erlang-factory.com/upload/presentations/503/Edis-ImplementingRedisinErlang.pdf http://www.erlang-factory.com/upload/presentations/503/Edis-...
alcuadrado 13y ago
I'd like to read antirez's opinion about this. This doesn't make much sense to me, as in redis you have to design how you are gonna store your data with it's limitation and performance in mind. So why a drop in replacement instead of redesigning your storage?
- firefoxman1 13y ago
  >I'd like to read antirez's opinion about this. It looks like he already made his own (3yrs ago): https://github.com/antirez/Bigdis https://github.com/antirez/Bigdis > in redis you have to design how you are gonna store your data with it's limitation and performance in mind. Hmm, that's interesting. This is the very reason I've always loved Redis and I'd love a Redis-for-disk.
rubyrescue 13y ago
At my consulting company, Inaka, we built this originally as a side project when we were using Redis and thought we needed multi-master larger-than-RAM datasets, but then we ended up moving to CouchDB for that project. It's currently a research project for us and we have an intern from Uppsala University working on the HA features including paxos leader election and multi-master conflict resolution. See the mutinode branch. Some people have added some additional database backends, but I don't know of any serious production users yet. I gave a talk on it at the 2012 Erlang Factory as it's entirely Erlang-based. Here's the talk - http://vimeo.com/42630498 http://vimeo.com/42630498 Benefits over Redis: Faster startup time over big datasets, pluggable backends, (eventually) Multi-master and HA (with a different approach than Redis is currently taking with Sentinel). Drawbacks: 3x to 10x slower on all operations. We've done no work to optimize.
- lsb 13y ago
  Have you seen Raft? It tries to be a much simpler and much more understandable alternative to Paxos. https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf https://ramcloud.stanford.edu/wiki/download/attachments/1137...
  - rubyrescue 13y ago
    looks interesting, thanks!
- stesch 13y ago
  You moved from Redis to CouchDB? That sounds odd.
  - klaruz 13y ago
    Why? His data set is larger than ram, and couchdb is a really great disk based key/value store.
    - stesch 13y ago
      Last time I checked CouchDB was a document store with MapReduce queries. A very different beast from Redis.
Titanous 13y ago
I started on a similar project written in Go and built on LevelDB: https://github.com/cupcake/setdb https://github.com/cupcake/setdb
hosay123 13y ago
Since it's leveldb underneath the usual problems with write rate apply: its quite easy to get leveldb into a position where writes completely lock up for minutes at a time, and the machine is tearing itself to shreds trying to complete merges to keep up with writes. Maybe a nice backend for an IndexedDB implementation, but for 'serious' storage, no way.
- benbjohnson 13y ago
  Do you have any links for more information on writes locking for minutes at a time? I haven't seen this with LevelDB and I couldn't find anything after searching around.
  - hosay123 13y ago
    Try searching their mailing list for "hang" and suchlike (of course they don't openly advertise such an easily triggered flaw). LevelDB write rate initially seems amazing, since it's simply writing unsorted keys to an append-only file until the file hits 2MB or so. For bursty loads it feels great. But the moment writes are sustained for longer than it can merge segments (say while doing a bulk load), per-write latency spikes appear (average op time jumps from <1ms to >30000ms for a single record), and eventually it'll get so far behind that all attempts to progress will hang entirely, waiting for the background compactor to free up room in the youngest generation. The effect seems to worsen exponentially with database size. To attempt to mitigate this, when LevelDB notices it's falling behind it begins sleeping for 0.1s every write. It's especially easy to trigger on slow CPUs with spinning rust drives. https://groups.google.com/forum/?fromgroups=#!topic/leveldb/ylEXQAs6XVA https://groups.google.com/forum/?fromgroups=#!topic/leveldb/...
    - jamesaguilar 13y ago
      If it is a bulk load, surely slow writes aren't a serious issue as long as throughput is good? Or are you saying the average write takes 30s?
    - pilooch 13y ago
snaky 13y ago
>Edis (currently) uses Google's leveldb as a backend That's nice. That means it would be (relatively) easy to change it to lmdb[1] to make it fast. [1] http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/
- benbjohnson 13y ago
  LMDB looks awesome. I lost a lot of performance (~5x) moving from a memory mapped architecture to LevelDB in a database I wrote (http://skydb.io/ http://skydb.io/). I'll definitely be checking out lmdb to see if I can get that performance back. Thanks!
  - snaky 13y ago
    SkyDB is definitely most interesting thing I seen this year in DB field, mostly because it's a rare example of out-of-the-box thinking and considering higher level use cases than usual (data semantics level I'd say) may give you an orders of magnitude speedup. But should I suggest, considering SkyDB highly experimental, maybe port it to more flexible backend like Mapkeeper[1] that gives for free not only easy switchable storage backends, but (not switchable, but changeable with a little effort) network/API backends (now it's Thrift), event engine (now it's libevent), etc. It would be nice to focus on core SkyDB things (which I think are data organization, API, stored procs etc.) while "outsource" the non-core low-level jobs to things like Mapkeeper. And by the way, lmdb backend have being merged into Mapkeeper not so long ago :-) [1] https://github.com/m1ch1/mapkeeper/ https://github.com/m1ch1/mapkeeper/
    - benbjohnson 13y ago
      Mapkeeper looks cool. I'll definitely read up on it some more. Sky originally started out as a project to learn how to write a database and programming language. The storage was originally a block-based mmap format and the query engine was built on LLVM. It was a great learning experience but I am trying to take a higher level approach by delegating storage to LevelDB (or possibly LMDB instead) and the query engine uses LuaJIT. Focusing on a higher level use case (i.e. behavioral analytics) has given some crazy performance gains and great scalability. I think once Sky matures more then it would be more interesting to be able to swap out the backends. I worry about having too many moving pieces. I'd also be concerned about copying data with the extra layers. It looks like LDMB does a zero copy which would be awesome.
wtetzner 13y ago
I feel like Redisc would be a more clever name.
- firefoxman1 13y ago
  Me too, then I Googled it: https://github.com/Moodstocks/redisk https://github.com/Moodstocks/redisk https://github.com/quirkey/redisk https://github.com/quirkey/redisk
jondot 13y ago
In this context, I think one can see this as a variant of Riak...
- nbevans 13y ago
  Except with none of the distributedness?
rogerbraun 13y ago
Sounds interesting, but there's a typo in the first sentence: compatable -> compatible.
- gingerlime 13y ago
  I spotted that too. Maybe I'm a terrible nitpicker / extremely harsh, but it kinda turned me off. Made it look instantly less of a serious project, when otherwise the look & feel of the page were pretty solid.
JulianMorrison 13y ago
Honestly, Redis is useful for basically 2 reasons: it's a stonkingly fast quasi-persistent store for simple data, and if you're already using it, it's a tolerable message queue and pub/sub. It's not a convenient document store. It's not relational and it's only ACID in that the single thread serializes all access. It doesn't scale worth a damn. But it's fast enough that mostly doesn't matter. So the question arises: why create something with none of Redis's advantages, and all of its flaws?
- rubyrescue 13y ago
  I'm going to assume you're not just trolling here. There are whole companies who make elaborate servers that speak simple protocols. Membase is essentially a much more elaborate, multi-server replicated database that speaks the Memcache protocol. We started edis as an experiment to teach some developers Erlang. We had a use case that was quite interesting at the time, so it was a great way to teach Erlang while working through some interesting database scenarios. Imagine you have a bunch of users come into a site and they'll hit the server for an hour and then go away for at least a week, maybe more, like the demographics around a popular TV show. Imagine you need to handle a HUGE number of users for that hour, but after that hour, you have less than 1/10th of 1% of those users during the week. Also, imagine that you are pretty happy with Redis's speed, but that you've outgrown what you can comfortably handle on one server and so you are happy to tolerate less speed for more space. This is a really nice use case for a disk-backed Redis server. It's not the only option, of course, but we didn't need to store documents and we didn't need ACIDity.
  - JulianMorrison 13y ago
    Yeah, not trolling. I haven't tried to design for exactly that, but my first shot at it would be independent pairs of app servers and redis instances, with redis used as an expiring cache, fronted by a load balancer with affinity, backed by centralized regular SQL. Something like MySQL has no trouble hanging onto a scadload of records it mostly doesn't touch, and you can scale the expiry time on redis down as it gets more heavily loaded, so only the hottest tip of the hot set is kept in RAM.
iand 13y ago
Setdb is a similar project https://github.com/cupcake/setdb https://github.com/cupcake/setdb
ricardobeat 13y ago
I was looking into https://github.com/KDr2/redis-leveldb https://github.com/KDr2/redis-leveldb a just a couple days ago, same concept. I'm confused about Edis current version being 1.0. Is it production ready or just ignoring semantical versioning?
- Uchikoma 13y ago
  From this thread, sounds like the second point: "It's currently a research project for us and we have an intern from Uppsala University working on the HA features including paxos leader election and multi-master conflict resolution."
qwerta 13y ago
If you are using Java, you may want to try MapDB. It uses memory mapped files and has interesting performance. Redis server protocol implementation is on roadmap.

147 points by nmcfarl 13y ago

> Edis is a protocol-compatable Server replacement for Redis, written in Erlang. Edis's goal is to be a drop-in replacement for Redis when persistence is more important than holding the dataset in-memory. Isn't the entire point of Redis that it's meticulously written in C by some Italian guy, specifically designed for in-memory datasets? Protocol compatible is one thing but maybe name it something with a greater edit distance?

nmcfarl 13y ago

This came up in the discussion of "Redis as the primary data store?" -> https://news.ycombinator.com/item?id=5620960 https://news.ycombinator.com/item?id=5620960 And I was wondering if anyone was using it - and what they thought… -- Also the github repo link: https://github.com/inaka/edis https://github.com/inaka/edis

chuhnk 13y ago
The more info link to the pdf for Erlang Factory 2012 talk appears to 404 but I found a direct source http://www.erlang-factory.com/upload/presentations/503/Edis-ImplementingRedisinErlang.pdf http://www.erlang-factory.com/upload/presentations/503/Edis-...

alcuadrado 13y ago

I'd like to read antirez's opinion about this. This doesn't make much sense to me, as in redis you have to design how you are gonna store your data with it's limitation and performance in mind. So why a drop in replacement instead of redesigning your storage?

firefoxman1 13y ago
>I'd like to read antirez's opinion about this. It looks like he already made his own (3yrs ago): https://github.com/antirez/Bigdis https://github.com/antirez/Bigdis > in redis you have to design how you are gonna store your data with it's limitation and performance in mind. Hmm, that's interesting. This is the very reason I've always loved Redis and I'd love a Redis-for-disk.

rubyrescue 13y ago

At my consulting company, Inaka, we built this originally as a side project when we were using Redis and thought we needed multi-master larger-than-RAM datasets, but then we ended up moving to CouchDB for that project. It's currently a research project for us and we have an intern from Uppsala University working on the HA features including paxos leader election and multi-master conflict resolution. See the mutinode branch. Some people have added some additional database backends, but I don't know of any serious production users yet. I gave a talk on it at the 2012 Erlang Factory as it's entirely Erlang-based. Here's the talk - http://vimeo.com/42630498 http://vimeo.com/42630498 Benefits over Redis: Faster startup time over big datasets, pluggable backends, (eventually) Multi-master and HA (with a different approach than Redis is currently taking with Sentinel). Drawbacks: 3x to 10x slower on all operations. We've done no work to optimize.

lsb 13y ago
Have you seen Raft? It tries to be a much simpler and much more understandable alternative to Paxos. https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf https://ramcloud.stanford.edu/wiki/download/attachments/1137...
- rubyrescue 13y ago
  looks interesting, thanks!
stesch 13y ago
You moved from Redis to CouchDB? That sounds odd.
- klaruz 13y ago
  Why? His data set is larger than ram, and couchdb is a really great disk based key/value store.
  - stesch 13y ago
    Last time I checked CouchDB was a document store with MapReduce queries. A very different beast from Redis.

Titanous 13y ago

I started on a similar project written in Go and built on LevelDB: https://github.com/cupcake/setdb https://github.com/cupcake/setdb

hosay123 13y ago

Since it's leveldb underneath the usual problems with write rate apply: its quite easy to get leveldb into a position where writes completely lock up for minutes at a time, and the machine is tearing itself to shreds trying to complete merges to keep up with writes. Maybe a nice backend for an IndexedDB implementation, but for 'serious' storage, no way.

benbjohnson 13y ago
Do you have any links for more information on writes locking for minutes at a time? I haven't seen this with LevelDB and I couldn't find anything after searching around.
- hosay123 13y ago
  Try searching their mailing list for "hang" and suchlike (of course they don't openly advertise such an easily triggered flaw). LevelDB write rate initially seems amazing, since it's simply writing unsorted keys to an append-only file until the file hits 2MB or so. For bursty loads it feels great. But the moment writes are sustained for longer than it can merge segments (say while doing a bulk load), per-write latency spikes appear (average op time jumps from <1ms to >30000ms for a single record), and eventually it'll get so far behind that all attempts to progress will hang entirely, waiting for the background compactor to free up room in the youngest generation. The effect seems to worsen exponentially with database size. To attempt to mitigate this, when LevelDB notices it's falling behind it begins sleeping for 0.1s every write. It's especially easy to trigger on slow CPUs with spinning rust drives. https://groups.google.com/forum/?fromgroups=#!topic/leveldb/ylEXQAs6XVA https://groups.google.com/forum/?fromgroups=#!topic/leveldb/...
  - jamesaguilar 13y ago
    If it is a bulk load, surely slow writes aren't a serious issue as long as throughput is good? Or are you saying the average write takes 30s?
  - pilooch 13y ago

snaky 13y ago

>Edis (currently) uses Google's leveldb as a backend That's nice. That means it would be (relatively) easy to change it to lmdb[1] to make it fast. [1] http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/

benbjohnson 13y ago
LMDB looks awesome. I lost a lot of performance (~5x) moving from a memory mapped architecture to LevelDB in a database I wrote (http://skydb.io/ http://skydb.io/). I'll definitely be checking out lmdb to see if I can get that performance back. Thanks!
- snaky 13y ago
  SkyDB is definitely most interesting thing I seen this year in DB field, mostly because it's a rare example of out-of-the-box thinking and considering higher level use cases than usual (data semantics level I'd say) may give you an orders of magnitude speedup. But should I suggest, considering SkyDB highly experimental, maybe port it to more flexible backend like Mapkeeper[1] that gives for free not only easy switchable storage backends, but (not switchable, but changeable with a little effort) network/API backends (now it's Thrift), event engine (now it's libevent), etc. It would be nice to focus on core SkyDB things (which I think are data organization, API, stored procs etc.) while "outsource" the non-core low-level jobs to things like Mapkeeper. And by the way, lmdb backend have being merged into Mapkeeper not so long ago :-) [1] https://github.com/m1ch1/mapkeeper/ https://github.com/m1ch1/mapkeeper/
  - benbjohnson 13y ago
    Mapkeeper looks cool. I'll definitely read up on it some more. Sky originally started out as a project to learn how to write a database and programming language. The storage was originally a block-based mmap format and the query engine was built on LLVM. It was a great learning experience but I am trying to take a higher level approach by delegating storage to LevelDB (or possibly LMDB instead) and the query engine uses LuaJIT. Focusing on a higher level use case (i.e. behavioral analytics) has given some crazy performance gains and great scalability. I think once Sky matures more then it would be more interesting to be able to swap out the backends. I worry about having too many moving pieces. I'd also be concerned about copying data with the extra layers. It looks like LDMB does a zero copy which would be awesome.

wtetzner 13y ago

I feel like Redisc would be a more clever name.

firefoxman1 13y ago
Me too, then I Googled it: https://github.com/Moodstocks/redisk https://github.com/Moodstocks/redisk https://github.com/quirkey/redisk https://github.com/quirkey/redisk

jondot 13y ago

In this context, I think one can see this as a variant of Riak...

nbevans 13y ago
Except with none of the distributedness?

rogerbraun 13y ago

Sounds interesting, but there's a typo in the first sentence: compatable -> compatible.

gingerlime 13y ago
I spotted that too. Maybe I'm a terrible nitpicker / extremely harsh, but it kinda turned me off. Made it look instantly less of a serious project, when otherwise the look & feel of the page were pretty solid.

JulianMorrison 13y ago

Honestly, Redis is useful for basically 2 reasons: it's a stonkingly fast quasi-persistent store for simple data, and if you're already using it, it's a tolerable message queue and pub/sub. It's not a convenient document store. It's not relational and it's only ACID in that the single thread serializes all access. It doesn't scale worth a damn. But it's fast enough that mostly doesn't matter. So the question arises: why create something with none of Redis's advantages, and all of its flaws?

rubyrescue 13y ago
I'm going to assume you're not just trolling here. There are whole companies who make elaborate servers that speak simple protocols. Membase is essentially a much more elaborate, multi-server replicated database that speaks the Memcache protocol. We started edis as an experiment to teach some developers Erlang. We had a use case that was quite interesting at the time, so it was a great way to teach Erlang while working through some interesting database scenarios. Imagine you have a bunch of users come into a site and they'll hit the server for an hour and then go away for at least a week, maybe more, like the demographics around a popular TV show. Imagine you need to handle a HUGE number of users for that hour, but after that hour, you have less than 1/10th of 1% of those users during the week. Also, imagine that you are pretty happy with Redis's speed, but that you've outgrown what you can comfortably handle on one server and so you are happy to tolerate less speed for more space. This is a really nice use case for a disk-backed Redis server. It's not the only option, of course, but we didn't need to store documents and we didn't need ACIDity.
- JulianMorrison 13y ago
  Yeah, not trolling. I haven't tried to design for exactly that, but my first shot at it would be independent pairs of app servers and redis instances, with redis used as an expiring cache, fronted by a load balancer with affinity, backed by centralized regular SQL. Something like MySQL has no trouble hanging onto a scadload of records it mostly doesn't touch, and you can scale the expiry time on redis down as it gets more heavily loaded, so only the hottest tip of the hot set is kept in RAM.

iand 13y ago

Setdb is a similar project https://github.com/cupcake/setdb https://github.com/cupcake/setdb

ricardobeat 13y ago

I was looking into https://github.com/KDr2/redis-leveldb https://github.com/KDr2/redis-leveldb a just a couple days ago, same concept. I'm confused about Edis current version being 1.0. Is it production ready or just ignoring semantical versioning?

Uchikoma 13y ago
From this thread, sounds like the second point: "It's currently a research project for us and we have an intern from Uppsala University working on the HA features including paxos leader election and multi-master conflict resolution."

qwerta 13y ago

If you are using Java, you may want to try MapDB. It uses memory mapped files and has interesting performance. Redis server protocol implementation is on roadmap.