3 ms·
Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground.
by bcrl 3y ago
Using fsync() to flush data to disk is, generally speaking, the wrong way to do things. fsync() is like using a piece of wood to hammer a nail into the ground. It'll work under very limited circumstances, but will fail horribly to do the heavy lifting required in other cases. fsync()'s semantics are complex and poorly defined, doubly so if multiple threads or processes are involved. Who gets the i/o error for a failed write? Nobody knows! fsync() is probably one of the worst APIs warts in Linux/Unix.
Developers competent in matters of storage generally use other methods / APIs like open()ing a file with O_SYNC and then checking the results of write() or pwrite() which will clearly identify when data is or is not successfully written to disk. If you want more performance, use async writes and it'll perform just fine. Seeing fsync() in code is a sign that the code has a bad smell and probably needs to be thoroughly reviewed.
- eatonphil 3y agoThere may be performance reasons to still use fsync with O_DIRECT instead of only O_SYNC. https://news.ycombinator.com/item?id=15539828 https://news.ycombinator.com/item?id=15539828
- jmclnx 3y ago> sing fsync() to flush data to disk is, generally speaking, the wrong way to do things. I think this is an old "statement" due to the Linux ext4 sync issue from quite awhile ago. As far as I know it has been fixed. I doubt it is discouraged on the BSDs. But, curious, what is the Right Thing to do ? There are cases where you want to be sure the data reaches the platter. On a mini I use to program on, if a filename started with an '@' sign, that said all writes went directly to disk. Based upon that were were able to create a roll-forward recovery for our custom applications. So, as far as I know, fsync(2) should accomplish the same thing using the open descriptor.
- bcrl 3y agoI stand by my position. The problem with fsync() is the question of when and where do errors get reported. If code checks errors on completion of each write it becomes significantly easier to know what write has failed. The error reporting is the granularity of the write being performed. In contrast, fsync() has to report errors for everything that has happened since the last fsync() call. If you are doing anything complicated like a database, which is virtually every single modern application, you need to have some idea of which writes have failed to figure out how to recover. fsync() gives you none of that. Different . In kernel you get results for each i/o performed, so filesystems can make the appropriate decision about how to respond to a failure. If you're building a high performance application, fsync() is a disaster. Unrelated writes will get bundled up when different threads call fsync() on the same file descriptor. Having multiple file descriptors open on the same file is silly and causes issues with applications that need to have many file descriptors open. The Linux kernel goes to great lengths to ensure that filesystems are high performance for writes on multiple threads. fsync() sucks for anything more complex than "I have a single file to write out".
- agallego 3y agoit tends to be true. we do things at the application tier to minimize these things. for example, assume you have to write A, B, C ... in syscalls it would be write( A ), flush() write( B ), flush() write( C ), flush() so if you add a debounce of say 4ms you get Write( A ), Write( B ), Write( C ), Flush() and saved 2 flush()-es This is common with protocol aware storage applications.
- eatonphil 3y agoAnd I guess the idea is furthermore that just because you don't fsync after every write does not mean that you haven't fsync-ed before responding to the user request saying the data is stored durably. I assume that you do actually guarantee the flush before returning a success to the user.
- agallego 3y agoexactly. you improve latency and throughput at the same time. is kinda cool. by delaying the reponses just a small bit you get huge benefits
- bcrl 3y agoWrong. Use something like aio or io_uring to submit 4 asynchronous writes in a single system call and you'll get way better performance. The kernel has all kinds of infrastructure that tries to coalesce and batch things that the write()+fsync() syscalls make horribly inefficient, and in the modern world of really fast nvme drives, you want to make your calls into the device driver as efficient as possible. You'll burn far fewer CPU cycles by giving the kernel the whole set of i/os in one single go, burn less on synchronization. It really is better to avoid write() + fsync().
- agallego 3y agowhat makes you think that we haven't tested this? seastar's io engine is defaulted to io-uring... there are about 10 things here to comment on. on optimized kernels we disable block coalescing at the kernel level, second we tell the kernel to use fifo, etc. these low hanging fruit was already something we've done for a very very long time.