6 ms·

File consistency (2015)

127 points by navinsylvester 9y ago

mjw1007 9y ago
In linux, renameat2() with RENAME_EXCHANGE on directories ought to be a very helpful primitive. Does anyone know what the state of glibc support is? The last thing I saw was this thread: https://sourceware.org/ml/libc-alpha/2015-11/msg00459.html https://sourceware.org/ml/libc-alpha/2015-11/msg00459.html
- jwilk 9y ago
  From renameat2 manpage: Glibc does not provide a wrapper for the renameat2() system call; call it using syscall(2). http://man7.org/linux/man-pages/man2/renameat2.2.html http://man7.org/linux/man-pages/man2/renameat2.2.html
pletnes 9y ago
Looks to me like yet another reason to use sqlite instead of «flat files». E.g postgres seems like a many orders of magnitude increase in complexity. Possibly related: We’ve been using hdf5 for a lot of data storage at work (raw image data). I often discover corrupt files, even though we (think we) are flushing files etc. I’d love to see some work on reliability there, but it’s hard to know if the article is relevant to those issues. Also, what happens when you’re on RAID? Even more assumptions out the window I’d imagine?
- zejn 9y ago
  sqlite also uses fsync, which at least on Linux has corner cases where errors don't even get reported. When fsync does report error, you should panic, as even if the subsequent fsync returns ok, that does not mean the data for which there was an error was actually written, in fact the opposite is true - kernel clears the dirty bit on non-written data thus neither the application nor kernel know which data is not written.
  - nordsieck 9y ago
    > sqlite also uses fsync At least sqlite has done the hard work of figuring out how to be as reliable as possible. At some level, writing to disk is a bit like using crypto. It's a lot easier to get right if you use well tested libraries that only offer high level primitives.
josephg 9y ago
This entire situation is a barely contained disaster. Doing all the steps listed at the top of the article is not only exhausting and error prone, it also dramatically lowers write performance. The most frustrating part is that (as I understand it) SSD and NVME hardware exposes strict read and write ordering to the OS anyway. That would allow file modification code to be written in a way thats both fast and correct. After all, these are the primitives we already know how to use for memory concurrency which enable fast lockless libraries like ConcurrencyKit. But for files, posix only exposes an inconsistently-implemented, coarse, heavyweight fsync() implementation. End users are forced to navigate weird, undocumented, imprecise hoops that are hard to understand, hard to test, and have poor performance just to do the one job the filesystem was supposed to have in the first place. I'm curious how much better you could do by skipping the OS entirely and making a userspace disk API, similar to DPDK. If your database code stores data in a single data file anyway, you wouldn't lose much by ditching the filesystem. (You would need specialised tools to do backups and figure out how much space you have free, but it might be worth it.) I've been writing my own tiny implementation of Kafka recently. I was reading through Kafka's design docs to figure out how they solved this problem. Kafka basically gives up on trusting the OS to store files safely. Instead they figure any fault tolerant kafka deployment will be a cluster of machines, so kafka stores all messages (+checksums) across all cluster instances. It hopes at least one of the machines in the cluster will survive without corruption when power goes out.
- aristidb 9y ago
  There is in fact a DPDK related project for NVMe storage: https://github.com/spdk/spdk https://github.com/spdk/spdk (I have not had reason to use it yet, unfortunately)
- zejn 9y ago
  Not only is fsync coarse, but also has a bunch of other issues. See https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq4xziwwoEfhoTZgr07vdGG%2Bhu%3D1adXx59aTeaoQ%40mail.gmail.com#CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq...
- zzzcpan 9y ago
evrydayhustling 9y ago
This was a cool read. I'd be interested in a perspective on how the increasing focus on distributed consistency is impacting design and research at this local consistency level. In particular, given the findings about frequency of errors, I wonder if there are guidelines for coordinating local filesystem settings along with distributed system settings to maximize performance at the distributed scale. Anybody out there doing this?
notduncansmith 9y ago
Are there no filesystems which address this issue? What challenges are involved? Does hardware support safe append-write filesystems? Why/why not?
zeveb 9y ago
It's interesting that IBM's JFS is so broken. Was the Linux version written by IBM, or is it a reimplementation by a third party?
dang 9y ago
Discussed at the time: https://news.ycombinator.com/item?id=10725859 https://news.ycombinator.com/item?id=10725859
dis-sys 9y ago
I am wondering what is the easy way to run tests for such file consistency issues say how the system react to power loss? Unplugging the power cable is not that automated for most people.
- zzzcpan 9y ago
  Using virtual file system layer, where you can model faults is pretty standard.
- nickpsecurity 9y ago
  The most impressive setup I've seen is SQLite's described in sections 3.2-3.3 of first link with more detail in their file operations in second. https://sqlite.org/testing.html https://sqlite.org/testing.html https://www.sqlite.org/fileio.html https://www.sqlite.org/fileio.html

In linux, renameat2() with RENAME_EXCHANGE on directories ought to be a very helpful primitive. Does anyone know what the state of glibc support is? The last thing I saw was this thread: https://sourceware.org/ml/libc-alpha/2015-11/msg00459.html https://sourceware.org/ml/libc-alpha/2015-11/msg00459.html

jwilk 9y ago
From renameat2 manpage: Glibc does not provide a wrapper for the renameat2() system call; call it using syscall(2). http://man7.org/linux/man-pages/man2/renameat2.2.html http://man7.org/linux/man-pages/man2/renameat2.2.html

pletnes 9y ago

Looks to me like yet another reason to use sqlite instead of «flat files». E.g postgres seems like a many orders of magnitude increase in complexity. Possibly related: We’ve been using hdf5 for a lot of data storage at work (raw image data). I often discover corrupt files, even though we (think we) are flushing files etc. I’d love to see some work on reliability there, but it’s hard to know if the article is relevant to those issues. Also, what happens when you’re on RAID? Even more assumptions out the window I’d imagine?

zejn 9y ago
sqlite also uses fsync, which at least on Linux has corner cases where errors don't even get reported. When fsync does report error, you should panic, as even if the subsequent fsync returns ok, that does not mean the data for which there was an error was actually written, in fact the opposite is true - kernel clears the dirty bit on non-written data thus neither the application nor kernel know which data is not written.
- nordsieck 9y ago
  > sqlite also uses fsync At least sqlite has done the hard work of figuring out how to be as reliable as possible. At some level, writing to disk is a bit like using crypto. It's a lot easier to get right if you use well tested libraries that only offer high level primitives.

josephg 9y ago

This entire situation is a barely contained disaster. Doing all the steps listed at the top of the article is not only exhausting and error prone, it also dramatically lowers write performance. The most frustrating part is that (as I understand it) SSD and NVME hardware exposes strict read and write ordering to the OS anyway. That would allow file modification code to be written in a way thats both fast and correct. After all, these are the primitives we already know how to use for memory concurrency which enable fast lockless libraries like ConcurrencyKit. But for files, posix only exposes an inconsistently-implemented, coarse, heavyweight fsync() implementation. End users are forced to navigate weird, undocumented, imprecise hoops that are hard to understand, hard to test, and have poor performance just to do the one job the filesystem was supposed to have in the first place. I'm curious how much better you could do by skipping the OS entirely and making a userspace disk API, similar to DPDK. If your database code stores data in a single data file anyway, you wouldn't lose much by ditching the filesystem. (You would need specialised tools to do backups and figure out how much space you have free, but it might be worth it.) I've been writing my own tiny implementation of Kafka recently. I was reading through Kafka's design docs to figure out how they solved this problem. Kafka basically gives up on trusting the OS to store files safely. Instead they figure any fault tolerant kafka deployment will be a cluster of machines, so kafka stores all messages (+checksums) across all cluster instances. It hopes at least one of the machines in the cluster will survive without corruption when power goes out.

aristidb 9y ago
There is in fact a DPDK related project for NVMe storage: https://github.com/spdk/spdk https://github.com/spdk/spdk (I have not had reason to use it yet, unfortunately)
zejn 9y ago
Not only is fsync coarse, but also has a bunch of other issues. See https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq4xziwwoEfhoTZgr07vdGG%2Bhu%3D1adXx59aTeaoQ%40mail.gmail.com#CAMsr+YHh+5Oq4xziwwoEfhoTZgr07vdGG+hu=1adXx59aTeaoQ@mail.gmail.com https://www.postgresql.org/message-id/flat/CAMsr%2BYHh%2B5Oq...
zzzcpan 9y ago

evrydayhustling 9y ago

This was a cool read. I'd be interested in a perspective on how the increasing focus on distributed consistency is impacting design and research at this local consistency level. In particular, given the findings about frequency of errors, I wonder if there are guidelines for coordinating local filesystem settings along with distributed system settings to maximize performance at the distributed scale. Anybody out there doing this?

notduncansmith 9y ago

Are there no filesystems which address this issue? What challenges are involved? Does hardware support safe append-write filesystems? Why/why not?

zeveb 9y ago

It's interesting that IBM's JFS is so broken. Was the Linux version written by IBM, or is it a reimplementation by a third party?

dang 9y ago

Discussed at the time: https://news.ycombinator.com/item?id=10725859 https://news.ycombinator.com/item?id=10725859

dis-sys 9y ago

I am wondering what is the easy way to run tests for such file consistency issues say how the system react to power loss? Unplugging the power cable is not that automated for most people.

zzzcpan 9y ago
Using virtual file system layer, where you can model faults is pretty standard.
nickpsecurity 9y ago
The most impressive setup I've seen is SQLite's described in sections 3.2-3.3 of first link with more detail in their file operations in second. https://sqlite.org/testing.html https://sqlite.org/testing.html https://www.sqlite.org/fileio.html https://www.sqlite.org/fileio.html