2 ms·

> buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. If you don't k

> buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management. If you don't know about it already, you should look at Viktor Leis' et al's work on this -- kernel module + supporting buffer mgmt code to do the kind of thing you're talking about. The code is not production quality, but the idea is nifty: https://github.com/viktorleis/vmcache https://github.com/viktorleis/vmcache https://github.com/tuhhosg/exmap https://github.com/tuhhosg/exmap https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_uploads/vmcache.pdf https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up... I've personally experimented a bit with userfaultfd and this kind of thing, but it's got issues.

vlovich123 3y ago
Yeah I’m aware work is kind of being done in the space. Not pretending it’s my idea. Just a more formal official API would be supremely interesting and not just for databases - there’s lots of user space caches that should be migrated to that if they’re recomputable. Relatedly there’s an effort going on in the filesystem space for a long time to add transaction support which would also be useful. Some day.
- hyc_symas 3y ago
  Transactions in the filesystem API is misguided and useless. All you need is to expose command tagging to user level, probably with some fcntl: https://www.spinics.net/lists/linux-fsdevel/msg70047.html https://www.spinics.net/lists/linux-fsdevel/msg70047.html Trying to implement actual transactions can't work because it requires the FS to maintain unbounded state. https://lore.kernel.org/all/5283E6DC.8000801@symas.com/ https://lore.kernel.org/all/5283E6DC.8000801@symas.com/
  - vlovich123 3y ago
    By “transcations” I mean the ability to do a bunch of “invisible” writes to a file that are only published through an fsync (and also not made visible just because a sibling application invoked sync). Full transactions aren’t necessary and I agree would be overkill. The work is primarily driven by XFS developers if I recall correctly (too lazy to lookup the exact links). But there are clever workarounds in user space although they’re not particularly confidence inducing.

vlovich123 3y ago

Yeah I’m aware work is kind of being done in the space. Not pretending it’s my idea. Just a more formal official API would be supremely interesting and not just for databases - there’s lots of user space caches that should be migrated to that if they’re recomputable. Relatedly there’s an effort going on in the filesystem space for a long time to add transaction support which would also be useful. Some day.

hyc_symas 3y ago
Transactions in the filesystem API is misguided and useless. All you need is to expose command tagging to user level, probably with some fcntl: https://www.spinics.net/lists/linux-fsdevel/msg70047.html https://www.spinics.net/lists/linux-fsdevel/msg70047.html Trying to implement actual transactions can't work because it requires the FS to maintain unbounded state. https://lore.kernel.org/all/5283E6DC.8000801@symas.com/ https://lore.kernel.org/all/5283E6DC.8000801@symas.com/
- vlovich123 3y ago
  By “transcations” I mean the ability to do a bunch of “invisible” writes to a file that are only published through an fsync (and also not made visible just because a sibling application invoked sync). Full transactions aren’t necessary and I agree would be overkill. The work is primarily driven by XFS developers if I recall correctly (too lazy to lookup the exact links). But there are clever workarounds in user space although they’re not particularly confidence inducing.