4 ms·

Doesn't leveldb , kinda suck? Meaning every ~serious db doesn't use it or has created a fork of it ? or see the symas-lmdb benchmarks for many different usecase

Doesn't leveldb , kinda suck? Meaning every ~serious db doesn't use it or has created a fork of it ? or see the symas-lmdb benchmarks for many different usecases of different kv stores http://symas.com/mdb/#bench http://symas.com/mdb/#bench

maxpert 11y ago
Well LMDB is hands down the fastest of them all. All the benchmarks that I saw with LMDB were mostly on Linux and ext file system. It would be really interesting to see how the memory mapped files perform on a phone though. But doing so requires me to port complete LMDB on phone :P
- catwell 11y ago
  LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. For those who don't know, I/O on Android is a nightmare. On your typical phone, the filesystem you have access to is a FUSE that mirrors to an EXT4 filesystem mounted with crazy device-specific options which include noauto_da_alloc. I don't know any DB which works fast and consistently across all Android phones on the market. SQLite is probably the safest bet for now.
  - mtanski 11y ago
    > LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. How is that possible if they are using Linux and UBC (Unified Buffer Cache). It's unlikely they anybody would go undo that from the VM/FS layer. The only think I can think off this behavior happening in a Linux app is if you're creating a private mapping... Private mappings are COW and a write to the global would cause a copy of the old data for you app. They'll be no sycing till you msync()
    - catwell 11y ago
      To be honest I am not entirely sure of my interpretation of the issue yet. What I know is that on those devices you sometimes can't read your writes with LMDB if it's not using MDB_WRITEMAP. While trying to debug I ended up on the only question ever asked by Howard Chu (LMDB author) on Stack Overflow: http://stackoverflow.com/questions/7061910/when-does-an-o-sync-write-become-visible-in-the-pagecache-mmapd-file http://stackoverflow.com/questions/7061910/when-does-an-o-sy... I am still trying to figure out what exactly is happening here but it's not very easy given that I do not even have access to a rooted phone that exhibits the issue.
      - hyc_symas 11y ago
        It really depends on where you're storing your DB files. If you're using the /data partition it should Just Work. If you're using external storage, yes, android does a bunch of stupid tricks to implement its secure storage, and that interferes with mmap. Here's the code that resulted from that Stack Overflow question: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L4091-L4096 https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblm... After 36 years of writing code, I have more answers than questions.
    - deleted 11y ago
      [deleted]
- hyc_symas 11y ago
  This whole post is kinda laughable, so a mediocre bug-ridden key/value store is somewhat faster than an SQL database. Big deal, they're all slow. LMDB has been tested on quite a lot of filesystems, not just ext. http://symas.com/mdb/microbench/july/#sec11 http://symas.com/mdb/microbench/july/#sec11 And actually, LMDB was working fine back in the Froyo/Gingerbread days. I integrated it into XDAndroid way back then. http://forum.xda-developers.com/showthread.php?t=1171052 http://forum.xda-developers.com/showthread.php?t=1171052 (By way of SQLightning.) Never tested it on things like squashfs or whatnot though.
ivoras 11y ago
It sucks allright. The performance numbers are actually mediocre at best in "real world" conditions. When I tried using it in a project where both high performance and concurrency were needed, I also discovered that it doesn't scale past 1 CPU core - probably uses mutexes or inefficient locks. I gave up on it rather than trying to fix it. That said, it has a simple API and sometimes neither concurrency nor ACID are important.
- duaneb 11y ago
  I believe the core writer runs on a single thread, so this makes sense. Also, I'd argue leveldb shines in a distributed environment where the simple interface is easy to leverage at scale--after all, it is basically the tablet part of big table.
cat-dev-null 11y ago
It sucks a lot. https://github.com/bitcoin/bitcoin/issues/2770 https://github.com/bitcoin/bitcoin/issues/2770 http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/ http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/
- haberman 11y ago
  That appears to be a bug in munmap() on OS X that just happened to bite LevelDB.

maxpert 11y ago

Well LMDB is hands down the fastest of them all. All the benchmarks that I saw with LMDB were mostly on Linux and ext file system. It would be really interesting to see how the memory mapped files perform on a phone though. But doing so requires me to port complete LMDB on phone :P

catwell 11y ago
LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. For those who don't know, I/O on Android is a nightmare. On your typical phone, the filesystem you have access to is a FUSE that mirrors to an EXT4 filesystem mounted with crazy device-specific options which include noauto_da_alloc. I don't know any DB which works fast and consistently across all Android phones on the market. SQLite is probably the safest bet for now.
- mtanski 11y ago
  > LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. How is that possible if they are using Linux and UBC (Unified Buffer Cache). It's unlikely they anybody would go undo that from the VM/FS layer. The only think I can think off this behavior happening in a Linux app is if you're creating a private mapping... Private mappings are COW and a write to the global would cause a copy of the old data for you app. They'll be no sycing till you msync()
  - catwell 11y ago
    To be honest I am not entirely sure of my interpretation of the issue yet. What I know is that on those devices you sometimes can't read your writes with LMDB if it's not using MDB_WRITEMAP. While trying to debug I ended up on the only question ever asked by Howard Chu (LMDB author) on Stack Overflow: http://stackoverflow.com/questions/7061910/when-does-an-o-sync-write-become-visible-in-the-pagecache-mmapd-file http://stackoverflow.com/questions/7061910/when-does-an-o-sy... I am still trying to figure out what exactly is happening here but it's not very easy given that I do not even have access to a rooted phone that exhibits the issue.
    - hyc_symas 11y ago
      It really depends on where you're storing your DB files. If you're using the /data partition it should Just Work. If you're using external storage, yes, android does a bunch of stupid tricks to implement its secure storage, and that interferes with mmap. Here's the code that resulted from that Stack Overflow question: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L4091-L4096 https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblm... After 36 years of writing code, I have more answers than questions.
  - deleted 11y ago
    [deleted]
hyc_symas 11y ago
This whole post is kinda laughable, so a mediocre bug-ridden key/value store is somewhat faster than an SQL database. Big deal, they're all slow. LMDB has been tested on quite a lot of filesystems, not just ext. http://symas.com/mdb/microbench/july/#sec11 http://symas.com/mdb/microbench/july/#sec11 And actually, LMDB was working fine back in the Froyo/Gingerbread days. I integrated it into XDAndroid way back then. http://forum.xda-developers.com/showthread.php?t=1171052 http://forum.xda-developers.com/showthread.php?t=1171052 (By way of SQLightning.) Never tested it on things like squashfs or whatnot though.

ivoras 11y ago

It sucks allright. The performance numbers are actually mediocre at best in "real world" conditions. When I tried using it in a project where both high performance and concurrency were needed, I also discovered that it doesn't scale past 1 CPU core - probably uses mutexes or inefficient locks. I gave up on it rather than trying to fix it. That said, it has a simple API and sometimes neither concurrency nor ACID are important.

duaneb 11y ago
I believe the core writer runs on a single thread, so this makes sense. Also, I'd argue leveldb shines in a distributed environment where the simple interface is easy to leverage at scale--after all, it is basically the tablet part of big table.

cat-dev-null 11y ago

It sucks a lot. https://github.com/bitcoin/bitcoin/issues/2770 https://github.com/bitcoin/bitcoin/issues/2770 http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/ http://hackingdistributed.com/2013/11/27/bitcoin-leveldb/

haberman 11y ago
That appears to be a bug in munmap() on OS X that just happened to bite LevelDB.