5 ms·

Well LMDB is hands down the fastest of them all. All the benchmarks that I saw with LMDB were mostly on Linux and ext file system. It would be really interestin

by maxpert 11y ago

Well LMDB is hands down the fastest of them all. All the benchmarks that I saw with LMDB were mostly on Linux and ext file system. It would be really interesting to see how the memory mapped files perform on a phone though. But doing so requires me to port complete LMDB on phone :P

catwell 11y ago
LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. For those who don't know, I/O on Android is a nightmare. On your typical phone, the filesystem you have access to is a FUSE that mirrors to an EXT4 filesystem mounted with crazy device-specific options which include noauto_da_alloc. I don't know any DB which works fast and consistently across all Android phones on the market. SQLite is probably the safest bet for now.
- mtanski 11y ago
  > LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. How is that possible if they are using Linux and UBC (Unified Buffer Cache). It's unlikely they anybody would go undo that from the VM/FS layer. The only think I can think off this behavior happening in a Linux app is if you're creating a private mapping... Private mappings are COW and a write to the global would cause a copy of the old data for you app. They'll be no sycing till you msync()
  - catwell 11y ago
    To be honest I am not entirely sure of my interpretation of the issue yet. What I know is that on those devices you sometimes can't read your writes with LMDB if it's not using MDB_WRITEMAP. While trying to debug I ended up on the only question ever asked by Howard Chu (LMDB author) on Stack Overflow: http://stackoverflow.com/questions/7061910/when-does-an-o-sync-write-become-visible-in-the-pagecache-mmapd-file http://stackoverflow.com/questions/7061910/when-does-an-o-sy... I am still trying to figure out what exactly is happening here but it's not very easy given that I do not even have access to a rooted phone that exhibits the issue.
    - hyc_symas 11y ago
      It really depends on where you're storing your DB files. If you're using the /data partition it should Just Work. If you're using external storage, yes, android does a bunch of stupid tricks to implement its secure storage, and that interferes with mmap. Here's the code that resulted from that Stack Overflow question: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L4091-L4096 https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblm... After 36 years of writing code, I have more answers than questions.
  - deleted 11y ago
    [deleted]
hyc_symas 11y ago
This whole post is kinda laughable, so a mediocre bug-ridden key/value store is somewhat faster than an SQL database. Big deal, they're all slow. LMDB has been tested on quite a lot of filesystems, not just ext. http://symas.com/mdb/microbench/july/#sec11 http://symas.com/mdb/microbench/july/#sec11 And actually, LMDB was working fine back in the Froyo/Gingerbread days. I integrated it into XDAndroid way back then. http://forum.xda-developers.com/showthread.php?t=1171052 http://forum.xda-developers.com/showthread.php?t=1171052 (By way of SQLightning.) Never tested it on things like squashfs or whatnot though.

by maxpert 11y ago

catwell 11y ago

LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. For those who don't know, I/O on Android is a nightmare. On your typical phone, the filesystem you have access to is a FUSE that mirrors to an EXT4 filesystem mounted with crazy device-specific options which include noauto_da_alloc. I don't know any DB which works fast and consistently across all Android phones on the market. SQLite is probably the safest bet for now.

mtanski 11y ago
> LMDB already runs on iOS and Android. In my experience you will have issues on some Android devices though. For instance, I recently found out that on some Android devices writes to a mmap-ed file through the usual POSIX interfaces are not immediatly reflected on reads through the mmap, which means that on those devices you MUST use MDB_WRITEMAP. How is that possible if they are using Linux and UBC (Unified Buffer Cache). It's unlikely they anybody would go undo that from the VM/FS layer. The only think I can think off this behavior happening in a Linux app is if you're creating a private mapping... Private mappings are COW and a write to the global would cause a copy of the old data for you app. They'll be no sycing till you msync()
- catwell 11y ago
  To be honest I am not entirely sure of my interpretation of the issue yet. What I know is that on those devices you sometimes can't read your writes with LMDB if it's not using MDB_WRITEMAP. While trying to debug I ended up on the only question ever asked by Howard Chu (LMDB author) on Stack Overflow: http://stackoverflow.com/questions/7061910/when-does-an-o-sync-write-become-visible-in-the-pagecache-mmapd-file http://stackoverflow.com/questions/7061910/when-does-an-o-sy... I am still trying to figure out what exactly is happening here but it's not very easy given that I do not even have access to a rooted phone that exhibits the issue.
  - hyc_symas 11y ago
    It really depends on where you're storing your DB files. If you're using the /data partition it should Just Work. If you're using external storage, yes, android does a bunch of stupid tricks to implement its secure storage, and that interferes with mmap. Here's the code that resulted from that Stack Overflow question: https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c#L4091-L4096 https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblm... After 36 years of writing code, I have more answers than questions.
- deleted 11y ago
  [deleted]

hyc_symas 11y ago

This whole post is kinda laughable, so a mediocre bug-ridden key/value store is somewhat faster than an SQL database. Big deal, they're all slow. LMDB has been tested on quite a lot of filesystems, not just ext. http://symas.com/mdb/microbench/july/#sec11 http://symas.com/mdb/microbench/july/#sec11 And actually, LMDB was working fine back in the Froyo/Gingerbread days. I integrated it into XDAndroid way back then. http://forum.xda-developers.com/showthread.php?t=1171052 http://forum.xda-developers.com/showthread.php?t=1171052 (By way of SQLightning.) Never tested it on things like squashfs or whatnot though.