6 ms·

Diskstore and a new b-tree implementation

33 points by mattyb 16y ago

deleted 16y ago
[deleted]
mumrah 16y ago
I could see one running out of inodes for a sizable dataset. Also, this seems like it would incur a good deal of iowait opening and closing all those fds.
- antirez 16y ago
  Yes I guess the btree (or similar) will be much better for many small items, but the file solution may be the way to go for not su much items but much bigger... Eventually maybe the fs solution can just be removed... it was a way to get started with the caching layer asap.
rb2k_ 16y ago
Anybody tried using this on HFS+ in OSX? It was horribly slow for me, but it could also have been parameters in the conf that were bad.
- antirez 16y ago
  By default the flush time is set to zero and the cache size to 0, so everything will be always loaded / processed / saved again, and ASAP. That said HFS+ is hardly cool...
agib 16y ago
Does anyone know the basic tradeoffs between the two stores?
snissn 16y ago
Are you familiar with memcachedb? it's a berkleyDB back behind memcache with a lru key destruction in memcache that persists to disk using berkley db... you definitely don't need to re-invent the wheel here, you can possibly use innoDB as a key val store also if you can find the source/binaries for it....
- antirez 16y ago
  In Redis the use of the disk KV store is just one of our layers, live objects are encoded in a different way to allow the usual Redis data model, so memcachedb is not a replacement for this unfortunately. Both BDB and innoDB are mature implementations of on disk KV stores, but unfortunately there are licensing issues in one case, and code not clean / bloated in another case preventing me from using this solutions. Basically my reasoning is always: if there is a good supported library that I like, I can use this, otherwise it is probably more interesting to learn something new. When I'll write recovery tools, deal with bugs, and so forth, it is better to deal with your own code if there isn't something of readily available, sanely licensed, simple to undertand, and small enough.
  - snissn 16y ago
    you should connect with the tokyo/kyoto cabinet author if you aren't already, http://twitter.com/#!/mikio1978/status/27344938710601728 http://twitter.com/#!/mikio1978/status/27344938710601728, his kyoto cabinet network 'daemon' is under heavy development and it would be really cool if you two were to collaborate and share code
    - antirez 16y ago
      It will be cool to share code and ideas, and he is definitely an expert in this field, where I'm just a newbie. But... I see that Kyoto is taking the route to be GPLed selling BSD-alice licenses. This is incompatible with my vision of open source. So what I'll do is to provide a minimal, clean, possibly toy-sh btree implementation outside Redis as a stand alone github project. Put it into BSD. Wait for requests of collaboration and code. A BSD-licensed library not bound to Redis is my way to say: let's collaborate on this. If the author of Tokyo Cabinet will join the effort, well, we'll be very lucky as we have an expert on board. But I suspect that anyway we are going to take some good contribution anyway, as this is a project that can be useful for many.

33 points by mattyb 16y ago

deleted 16y ago

[deleted]

I could see one running out of inodes for a sizable dataset. Also, this seems like it would incur a good deal of iowait opening and closing all those fds.

antirez 16y ago
Yes I guess the btree (or similar) will be much better for many small items, but the file solution may be the way to go for not su much items but much bigger... Eventually maybe the fs solution can just be removed... it was a way to get started with the caching layer asap.

rb2k_ 16y ago

Anybody tried using this on HFS+ in OSX? It was horribly slow for me, but it could also have been parameters in the conf that were bad.

antirez 16y ago
By default the flush time is set to zero and the cache size to 0, so everything will be always loaded / processed / saved again, and ASAP. That said HFS+ is hardly cool...

agib 16y ago

Does anyone know the basic tradeoffs between the two stores?

snissn 16y ago

Are you familiar with memcachedb? it's a berkleyDB back behind memcache with a lru key destruction in memcache that persists to disk using berkley db... you definitely don't need to re-invent the wheel here, you can possibly use innoDB as a key val store also if you can find the source/binaries for it....

antirez 16y ago
In Redis the use of the disk KV store is just one of our layers, live objects are encoded in a different way to allow the usual Redis data model, so memcachedb is not a replacement for this unfortunately. Both BDB and innoDB are mature implementations of on disk KV stores, but unfortunately there are licensing issues in one case, and code not clean / bloated in another case preventing me from using this solutions. Basically my reasoning is always: if there is a good supported library that I like, I can use this, otherwise it is probably more interesting to learn something new. When I'll write recovery tools, deal with bugs, and so forth, it is better to deal with your own code if there isn't something of readily available, sanely licensed, simple to undertand, and small enough.
- snissn 16y ago
  you should connect with the tokyo/kyoto cabinet author if you aren't already, http://twitter.com/#!/mikio1978/status/27344938710601728 http://twitter.com/#!/mikio1978/status/27344938710601728, his kyoto cabinet network 'daemon' is under heavy development and it would be really cool if you two were to collaborate and share code
  - antirez 16y ago
    It will be cool to share code and ideas, and he is definitely an expert in this field, where I'm just a newbie. But... I see that Kyoto is taking the route to be GPLed selling BSD-alice licenses. This is incompatible with my vision of open source. So what I'll do is to provide a minimal, clean, possibly toy-sh btree implementation outside Redis as a stand alone github project. Put it into BSD. Wait for requests of collaboration and code. A BSD-licensed library not bound to Redis is my way to say: let's collaborate on this. If the author of Tokyo Cabinet will join the effort, well, we'll be very lucky as we have an expert on board. But I suspect that anyway we are going to take some good contribution anyway, as this is a project that can be useful for many.