5 ms·

RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system https://ayende.com/bl

RavenDB's response to this paper: https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to-use-mmap-in-your-database-management-system https://ayende.com/blog/196161-C/re-are-you-sure-you-want-to...

tptacek 5y ago
From that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/ https://danluu.com/fsyncgate/
- ayende 5y ago
  That actually doesn't matter This is orthogonal to mmap
  - aidenn0 5y ago
    As you pointed out in your article, it invalidates much (all?) of the the "Problem #3: Error handling" section of TFA.
dwenzek 5y ago
Thank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".
tyingq 5y ago
The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?
- dboreham 5y ago
  When I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as <popular database du jour>.
  - ayende 5y ago
    Yes, isn't that wonderful? You get to take advantage of literally decades of experience What is more, if you can match the profile of the optimization, you can benefit even more
  - tytso 5y ago
    Um... Oracle (and other enterprise databases like DB2) don't use mmap. They use Direct I/O. Oracle does have anonymous (non-file-backed) memory which is mmap'ed and shared across various Oracle processes, called the Shared Global Area (SGA), but it's not used for I/O.
    - hyc_symas 5y ago
      Fwiw, I wrote a Direct I/O patch for BerkeleyDB but withdrew it later because it didn't ever improve I/O perf or memory footprint.
- AdamProut 5y ago
  I suppose. Some problems with mmap() are a bit hard to fix from user land though. You will hit contention on locks inside the kernel (mmap_sem) if the database does concurrent high throughput mmap()/unmap(). I don't follow linux kernel development closely to know if this has been improved recently, but it was easy to reproduce it 4-5 years ago.
  - ayende 5y ago
    Almost no one is going to have a lot of map calls Uou map the file once, then fault it in
10000truths 5y ago
You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch from disk. It is better equipped to determine when to buffer writes and when to flush to disk. Sure, it might be more work than using mmap. But it's also more correct, forces you to handle edge cases, and much more amenable to platform-specific improvements a la kqueue/io_uring.
- sltkr 5y ago
  The counterargument to this is that the kernel can make decisions based on nonlocal information about the system. If your database server is the only process in the system that is using significant memory, then sure, you might as well manage it yourself. But if there are multiple processes competing for memory, the kernel is better equipped to decide which processes' pages should be paged out or kept into memory.
  - electricshampo1 5y ago
    Generally for perf critical use cases you dedicate the machine to the database. This simplifies many things (avoiding having to reason about sharing, etc etc).
    - munchler 5y ago
      This makes me wonder whether there would be value in an OS that is also a DBMS (or vice versa). In other words, if the DBMS has total control over the hardware, perhaps performance can be maximized without too much additional complexity.
      - sedachv 5y ago
        This is a bad idea from the 1960s: IBM TPF, MUMPS, Pick. As soon as the hardware changes it becomes slower and more complicated.
ikawe 5y ago
> Off the top of my head, most embedded databases implement a single writer model. LMDB, Voron (RavenDB’s storage engine), LevelDB, Lucene And let's not forget sqlite! > There can only be a single writer at a time to an SQLite database. (from https://www.sqlite.org/isolation.html https://www.sqlite.org/isolation.html)

tptacek 5y ago

From that article: the whole fsyncgate thing seems like a pretty strong counterargument to "mmap adds more complexity than it removes": https://danluu.com/fsyncgate/ https://danluu.com/fsyncgate/

ayende 5y ago
That actually doesn't matter This is orthogonal to mmap
- aidenn0 5y ago
  As you pointed out in your article, it invalidates much (all?) of the the "Problem #3: Error handling" section of TFA.

dwenzek 5y ago

Thank you for these counter-arguments. It's good to have them to make up your own mind, especially when recognized experts use a mocking tone "you will no dare think the contrary".

tyingq 5y ago

The really key part seems to be this: "If you aren’t using mmap, on the other hand, you still need to handle of all those issues" Which seems like a reasonable statement. Is it less work to make your own top-to-bottom buffer pool, and would that necessarily avoid similar issues? Or is it less work to use mmap(), but address the issues?

dboreham 5y ago
When I worked on/with BerkeleyDB in the late 90s we came to the conclusion that the various OS mmap() implementations had been tweaked/fixed to the point where they worked for the popular high profile applications (in those days: Oracle). So it can appear like everything is fine, but that probably means your code behaves the same way as <popular database du jour>.
- ayende 5y ago
  Yes, isn't that wonderful? You get to take advantage of literally decades of experience What is more, if you can match the profile of the optimization, you can benefit even more
- tytso 5y ago
  Um... Oracle (and other enterprise databases like DB2) don't use mmap. They use Direct I/O. Oracle does have anonymous (non-file-backed) memory which is mmap'ed and shared across various Oracle processes, called the Shared Global Area (SGA), but it's not used for I/O.
  - hyc_symas 5y ago
    Fwiw, I wrote a Direct I/O patch for BerkeleyDB but withdrew it later because it didn't ever improve I/O perf or memory footprint.
AdamProut 5y ago
I suppose. Some problems with mmap() are a bit hard to fix from user land though. You will hit contention on locks inside the kernel (mmap_sem) if the database does concurrent high throughput mmap()/unmap(). I don't follow linux kernel development closely to know if this has been improved recently, but it was easy to reproduce it 4-5 years ago.
- ayende 5y ago
  Almost no one is going to have a lot of map calls Uou map the file once, then fault it in

10000truths 5y ago

You don't want the OS to take care of reading from disk and page caching/eviction. You want the DB itself to have explicit control over that, because the DB has information on access patterns and table format that the OS is not aware of. It is better equipped than the OS to anticipate what portions of tables/indices need to be cached in memory. It is better equipped to calculate when/where/what/how much to prefetch from disk. It is better equipped to determine when to buffer writes and when to flush to disk. Sure, it might be more work than using mmap. But it's also more correct, forces you to handle edge cases, and much more amenable to platform-specific improvements a la kqueue/io_uring.

sltkr 5y ago
The counterargument to this is that the kernel can make decisions based on nonlocal information about the system. If your database server is the only process in the system that is using significant memory, then sure, you might as well manage it yourself. But if there are multiple processes competing for memory, the kernel is better equipped to decide which processes' pages should be paged out or kept into memory.
- electricshampo1 5y ago
  Generally for perf critical use cases you dedicate the machine to the database. This simplifies many things (avoiding having to reason about sharing, etc etc).
  - munchler 5y ago
    This makes me wonder whether there would be value in an OS that is also a DBMS (or vice versa). In other words, if the DBMS has total control over the hardware, perhaps performance can be maximized without too much additional complexity.
    - sedachv 5y ago
      This is a bad idea from the 1960s: IBM TPF, MUMPS, Pick. As soon as the hardware changes it becomes slower and more complicated.

ikawe 5y ago

> Off the top of my head, most embedded databases implement a single writer model. LMDB, Voron (RavenDB’s storage engine), LevelDB, Lucene And let's not forget sqlite! > There can only be a single writer at a time to an SQLite database. (from https://www.sqlite.org/isolation.html https://www.sqlite.org/isolation.html)