5 ms·

There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a ca

There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a cache but it's definitely not what you want for a database!

deleted 6y ago
[deleted]
eqvinox 6y ago
That's what msync() is for.
- monocasa 6y ago
  Right, but it can sync arbitrary ranges sooner, which is also awful for consistency.
  - reader_mode 6y ago
    Shouldn't your write strategy be resilient to that kind of stuff (eg. shutdown during a partial update) ?
    - gmueckl 6y ago
      Don't you need exact guarantees on write ordering to achieve that?
      - bonzini 6y ago
        You can do write/sync/write/sync in order to achieve that. It would be nicer to have FUA support in system calls (or you can open the same file to two descriptors, one with O_SYNC and one without).
      - jorangreef 6y ago
        Yes, for almost all databases, although there was a cool paper from the University of Wisconsin Madison a few years ago that showed how to design something that could work without write barriers, and under the assumption that disks don't always fsync correctly: "the No-Order File System (NoFS), a simple, lightweight file system that employs a novel technique called backpointer based consistency to provide crash consistency without ordering writes as they go to disk" http://pages.cs.wisc.edu/~vijayc/nofs.htm http://pages.cs.wisc.edu/~vijayc/nofs.htm
        vlovich123 6y ago
        Does that generalize to databases? My understanding is that file systems are a restricted case of databases that don’t necessarily support all operations (eg transactions are smaller, can’t do arbitrary queries within a transaction, etc etc).
        jorangreef 6y ago
        I think it would, because if your underlying file system can provide crash consistency then that can be leveraged as a primitive by the database.
- cperciva 6y ago
  If you're tracking what needs to be flushed to disk when, you might as well just be making explicit pwrite syscalls.
- dooglius 6y ago
  I think you mean mlock

deleted 6y ago

[deleted]

eqvinox 6y ago

That's what msync() is for.

monocasa 6y ago
Right, but it can sync arbitrary ranges sooner, which is also awful for consistency.
- reader_mode 6y ago
  Shouldn't your write strategy be resilient to that kind of stuff (eg. shutdown during a partial update) ?
  - gmueckl 6y ago
    Don't you need exact guarantees on write ordering to achieve that?
    - bonzini 6y ago
      You can do write/sync/write/sync in order to achieve that. It would be nicer to have FUA support in system calls (or you can open the same file to two descriptors, one with O_SYNC and one without).
    - jorangreef 6y ago
      Yes, for almost all databases, although there was a cool paper from the University of Wisconsin Madison a few years ago that showed how to design something that could work without write barriers, and under the assumption that disks don't always fsync correctly: "the No-Order File System (NoFS), a simple, lightweight file system that employs a novel technique called backpointer based consistency to provide crash consistency without ordering writes as they go to disk" http://pages.cs.wisc.edu/~vijayc/nofs.htm http://pages.cs.wisc.edu/~vijayc/nofs.htm
      - vlovich123 6y ago
        Does that generalize to databases? My understanding is that file systems are a restricted case of databases that don’t necessarily support all operations (eg transactions are smaller, can’t do arbitrary queries within a transaction, etc etc).
        jorangreef 6y ago
        I think it would, because if your underlying file system can provide crash consistency then that can be leveraged as a primitive by the database.
cperciva 6y ago
If you're tracking what needs to be flushed to disk when, you might as well just be making explicit pwrite syscalls.
dooglius 6y ago
I think you mean mlock