5 ms·
This entire situation is a barely contained disaster. Doing all the steps listed at the top of the article is not only exhausting and error prone, it also drama
by josephg 9y ago
This entire situation is a barely contained disaster. Doing all the steps listed at the top of the article is not only exhausting and error prone, it also dramatically lowers write performance.
The most frustrating part is that (as I understand it) SSD and NVME hardware exposes strict read and write ordering to the OS anyway. That would allow file modification code to be written in a way thats both fast and correct. After all, these are the primitives we already know how to use for memory concurrency which enable fast lockless libraries like ConcurrencyKit. But for files, posix only exposes an inconsistently-implemented, coarse, heavyweight fsync() implementation. End users are forced to navigate weird, undocumented, imprecise hoops that are hard to understand, hard to test, and have poor performance just to do the one job the filesystem was supposed to have in the first place.
I'm curious how much better you could do by skipping the OS entirely and making a userspace disk API, similar to DPDK. If your database code stores data in a single data file anyway, you wouldn't lose much by ditching the filesystem. (You would need specialised tools to do backups and figure out how much space you have free, but it might be worth it.)
I've been writing my own tiny implementation of Kafka recently. I was reading through Kafka's design docs to figure out how they solved this problem. Kafka basically gives up on trusting the OS to store files safely. Instead they figure any fault tolerant kafka deployment will be a cluster of machines, so kafka stores all messages (+checksums) across all cluster instances. It hopes at least one of the machines in the cluster will survive without corruption when power goes out.
- aristidb 9y agoThere is in fact a DPDK related project for NVMe storage: https://github.com/spdk/spdk https://github.com/spdk/spdk (I have not had reason to use it yet, unfortunately)
- zejn 9y agoNot only is fsync coarse, but also has a bunch of other issues. See https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq4xziwwoEfhoTZgr07vdGG%2Bhu%3D1adXx59aTeaoQ%40mail.gmail.com#CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq...
- zzzcpan 9y ago> how much better you could do by skipping the OS entirely I'd say this is the main reason distributed systems even exist, because no amount of crash proofing file writes for the filesystem is going to make disks, raids correct and reliable, individual machines, datacenters reliable, operational mistakes not happen, interpretation and other mistakes not happen and so on. On the other hand consistency on random writes to files is not something absolutely necessary. We can get away with just sequential writes and there would be not much to reorder and to mess up to begin with.
- josephg 9y ago> On the other hand consistency on random writes to files is not something absolutely necessary. This stuff does matter; because most applications aren't distributed systems. Using file system operations in the way they were intended to be used shouldn't sometimes accidentally corrupt files by default. Look again at the table in the article showing consistency bugs in application code. They even found bugs in sqlite. If not even sqlite does this stuff correctly, mortals have no hope. And distributed systems themselves depend on the guarantees provided by filesystems for correctness. The guarantees filesystems provide are poorly documented, poorly understood and inconsistently implemented across filesystems and operating systems. Take some time to read about what redis, postgres and sqlite do to correctly write files - its an awful morass. At a minimum we should have a standard atomic write primitive that does all the stuff listed at the top of that article. I don't really care how slow it is - atomic writes are what everyone expects write() to actually do. Atomic write shouldn't be poorly reimplemented by every text editor and configuration tool. Video games shouldn't need to ask players not to switch off their consoles while they're saving. (What an absolute embarrassment.) If filesystems want to compete on performance, compete on the performance of atomic append. Right now any atomic append implementation in userland is both complicated and dog slow. With help from kernel-level filesystem code, we can do way better on both counts. Then for high performance applications (databases, etc) the OS should expose either fence() operations or some sort of durable write order. Thats the API databases will actually use. For extra credit I would also like a write API that notified me when my data has been durably synced to disk. A kqueue event would be perfect. And once we have those APIs I have no idea when anyone would use the current write/fsync implementation. Its a huge foot-gun. Its what you would get if mongodb made a filesystem API.
- falcolas 9y ago> Kafka basically gives up on trusting the OS to store files safely They're not alone. There are configs that you can pass to MySQL to have it manage files on a particular disk partition directly via drivers. It's a crazy level of optimization, but the OS does so much to make things fast that they aren't necessarily reliable. SSDs sometimes make it worse, implementing their own abstractions over the top of what's actually going on so they can pretend to be regular hard drives with sectors and head travel and... On the plus side, if you take the route of working with the SSD directly, you can get a whole different set of guarantees.
- ddorian43 9y agoSeastar/scylladb is looking for custom filesystem for per-core dbms https://github.com/scylladb/seastar/wiki/Google-Summer-of-Code-2018#extend-native-userspace-io-scheduler-to-support-multiple-disks https://github.com/scylladb/seastar/wiki/Google-Summer-of-Co...
- openasocket 9y ago> I'm curious how much better you could do by skipping the OS entirely and making a userspace disk API, similar to DPDK I've thought about the same thing myself. Some sort of direct access to block IO, with the ability to force writes (and reads?) to go directly to disk, and to force some sort of ordering between requests.
- colanderman 9y agoLinux gives you sync_file_range(2), which solves the granularity problem of fsync(2). (Not that it's a particularly big problem anyway if you store your data and log in two separate files. Your data need only be synced during checkpointing, and you always want to sync everything recently written to your log.) fsync(2) semantic issues aside. While SCSI does support explicit write ordering, many SANs do not. They define write ordering (and therefore crash consistency) using only request-after-reply semantics. So the fact that memory-barrier-style primitives aren't exposed for file or block isn't a big loss for portable applications either.
- zokier 9y ago> I'm curious how much better you could do by skipping the OS entirely and making a userspace disk API, similar to DPDK. The biggest concern I'd have is the somewhat questionable black-box abstraction layer that is the disk controllers. I personally think they do way too much stuff behind the curtain that would be better put in the open. Something like UBIFS would be better suited for modern world. > If your database code stores data in a single data file anyway, you wouldn't lose much by ditching the filesystem. Afaik Oracle DBs can use raw disks directly.