6 ms·

>Edis (currently) uses Google's leveldb as a backend That's nice. That means it would be (relatively) easy to change it to lmdb[1] to make it fast. [1] http:/

by snaky 14y ago

>Edis (currently) uses Google's leveldb as a backend That's nice. That means it would be (relatively) easy to change it to lmdb[1] to make it fast. [1] http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/

benbjohnson 14y ago
LMDB looks awesome. I lost a lot of performance (~5x) moving from a memory mapped architecture to LevelDB in a database I wrote (http://skydb.io/ http://skydb.io/). I'll definitely be checking out lmdb to see if I can get that performance back. Thanks!
- snaky 14y ago
  SkyDB is definitely most interesting thing I seen this year in DB field, mostly because it's a rare example of out-of-the-box thinking and considering higher level use cases than usual (data semantics level I'd say) may give you an orders of magnitude speedup. But should I suggest, considering SkyDB highly experimental, maybe port it to more flexible backend like Mapkeeper[1] that gives for free not only easy switchable storage backends, but (not switchable, but changeable with a little effort) network/API backends (now it's Thrift), event engine (now it's libevent), etc. It would be nice to focus on core SkyDB things (which I think are data organization, API, stored procs etc.) while "outsource" the non-core low-level jobs to things like Mapkeeper. And by the way, lmdb backend have being merged into Mapkeeper not so long ago :-) [1] https://github.com/m1ch1/mapkeeper/ https://github.com/m1ch1/mapkeeper/
  - benbjohnson 14y ago
    Mapkeeper looks cool. I'll definitely read up on it some more. Sky originally started out as a project to learn how to write a database and programming language. The storage was originally a block-based mmap format and the query engine was built on LLVM. It was a great learning experience but I am trying to take a higher level approach by delegating storage to LevelDB (or possibly LMDB instead) and the query engine uses LuaJIT. Focusing on a higher level use case (i.e. behavioral analytics) has given some crazy performance gains and great scalability. I think once Sky matures more then it would be more interesting to be able to swap out the backends. I worry about having too many moving pieces. I'd also be concerned about copying data with the extra layers. It looks like LDMB does a zero copy which would be awesome.
    - snaky 14y ago
      Yeah, zero-copying is a great LMDB feature, even using LuaJIT stored procs you can pass pointer to mmap window and process the data in that window using ffi (did I say LuaJIT is just damn great?) and (optionally) DynASM.
saurik 14y ago
I actually made a fork of redis (as opposed to a reimplementation), with the design challenge "smallest possible patch" (to make it easier to merge forward), that used LMDB as a graveyard of "old keys" for this very purpose. git://git.saurik.com/redis.git I do not quite remember if there were any outstanding "didn't quite finish this corner case" items, but I don't think there were. (If I looked at it I could tell, but I'm on my phone, and should be looking at other things right now anyway.) (edit: To be clear, I mean in "normal usage". I almost certainly didn't think through issues involving, say, replication, to make certain the semantics were always sensible; my goal was "doesn't crash or break or corrupt".) (I think the problem was mostly that the higher-level architecture I was putting this piece into underwent a slight change that made me think I might need something slightly different, and the result became "since I literally only use the APPEND command, I can probably come up with something more directly solves my problem.) If you have any questions, I'm on IRC (various networks, including freenode and irc.saurik.com as "saurik").
- snaky 14y ago
  That's great, thank you! But I remember some time ago antirez was trying to implement the same idea (it was called "virtual memory"[1]), but failed [2] mainly due the complexity of implementation. [1] http://redis.io/topics/virtual-memory http://redis.io/topics/virtual-memory [2] http://oldblog.antirez.com/post/redis-virtual-memory-story.html http://oldblog.antirez.com/post/redis-virtual-memory-story.h...
  - saurik 14y ago
    Yeah. He, however, also was developing, himself, the storage backend, so that's already one major source of complexity you don't have if you use an off-the-shelf embedded key-value library. He also is simply working with a wide set of use cases, both at the time and on his roadmap, where VM causes things to "kind of suck". I certainly did not have the goal of making everything "work well", I just wanted things to "not break horribly" (I added a sentence to my previous post making this more clear). If I had actually built something that would be epic for some wide range of use cases, I'd have done something more with it than just leaving it in a public git repository for months ;P. In the end this didn't even satisfy my use case...

by snaky 14y ago

benbjohnson 14y ago

LMDB looks awesome. I lost a lot of performance (~5x) moving from a memory mapped architecture to LevelDB in a database I wrote (http://skydb.io/ http://skydb.io/). I'll definitely be checking out lmdb to see if I can get that performance back. Thanks!

snaky 14y ago
SkyDB is definitely most interesting thing I seen this year in DB field, mostly because it's a rare example of out-of-the-box thinking and considering higher level use cases than usual (data semantics level I'd say) may give you an orders of magnitude speedup. But should I suggest, considering SkyDB highly experimental, maybe port it to more flexible backend like Mapkeeper[1] that gives for free not only easy switchable storage backends, but (not switchable, but changeable with a little effort) network/API backends (now it's Thrift), event engine (now it's libevent), etc. It would be nice to focus on core SkyDB things (which I think are data organization, API, stored procs etc.) while "outsource" the non-core low-level jobs to things like Mapkeeper. And by the way, lmdb backend have being merged into Mapkeeper not so long ago :-) [1] https://github.com/m1ch1/mapkeeper/ https://github.com/m1ch1/mapkeeper/
- benbjohnson 14y ago
  Mapkeeper looks cool. I'll definitely read up on it some more. Sky originally started out as a project to learn how to write a database and programming language. The storage was originally a block-based mmap format and the query engine was built on LLVM. It was a great learning experience but I am trying to take a higher level approach by delegating storage to LevelDB (or possibly LMDB instead) and the query engine uses LuaJIT. Focusing on a higher level use case (i.e. behavioral analytics) has given some crazy performance gains and great scalability. I think once Sky matures more then it would be more interesting to be able to swap out the backends. I worry about having too many moving pieces. I'd also be concerned about copying data with the extra layers. It looks like LDMB does a zero copy which would be awesome.
  - snaky 14y ago
    Yeah, zero-copying is a great LMDB feature, even using LuaJIT stored procs you can pass pointer to mmap window and process the data in that window using ffi (did I say LuaJIT is just damn great?) and (optionally) DynASM.

saurik 14y ago

I actually made a fork of redis (as opposed to a reimplementation), with the design challenge "smallest possible patch" (to make it easier to merge forward), that used LMDB as a graveyard of "old keys" for this very purpose. git://git.saurik.com/redis.git I do not quite remember if there were any outstanding "didn't quite finish this corner case" items, but I don't think there were. (If I looked at it I could tell, but I'm on my phone, and should be looking at other things right now anyway.) (edit: To be clear, I mean in "normal usage". I almost certainly didn't think through issues involving, say, replication, to make certain the semantics were always sensible; my goal was "doesn't crash or break or corrupt".) (I think the problem was mostly that the higher-level architecture I was putting this piece into underwent a slight change that made me think I might need something slightly different, and the result became "since I literally only use the APPEND command, I can probably come up with something more directly solves my problem.) If you have any questions, I'm on IRC (various networks, including freenode and irc.saurik.com as "saurik").

snaky 14y ago
That's great, thank you! But I remember some time ago antirez was trying to implement the same idea (it was called "virtual memory"[1]), but failed [2] mainly due the complexity of implementation. [1] http://redis.io/topics/virtual-memory http://redis.io/topics/virtual-memory [2] http://oldblog.antirez.com/post/redis-virtual-memory-story.html http://oldblog.antirez.com/post/redis-virtual-memory-story.h...
- saurik 14y ago
  Yeah. He, however, also was developing, himself, the storage backend, so that's already one major source of complexity you don't have if you use an off-the-shelf embedded key-value library. He also is simply working with a wide set of use cases, both at the time and on his roadmap, where VM causes things to "kind of suck". I certainly did not have the goal of making everything "work well", I just wanted things to "not break horribly" (I added a sentence to my previous post making this more clear). If I had actually built something that would be epic for some wide range of use cases, I'd have done something more with it than just leaving it in a public git repository for months ;P. In the end this didn't even satisfy my use case...