11 ms·

Unix’s file durability problem

287 points by robinhouston 11y ago

rocky1138 11y ago
Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?
- mwcampbell 11y ago
  Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
  - lisper 11y ago
    It only takes one combination to start the chain reaction. If someone identifies one combination of kernel, disk driver, file system, hardware, and syscalls that results in reliable durability then people who care about reliable durability will start using it, and that will eventually turn into a de facto standard which will eventually turn into an actual standard.
    - tamana 11y ago
      Until someone who doesn't care changes some code and that pattern is no longer durable.
  - gcc_programmer 11y ago
    All of it - these things aren't straightforward, you need to go knee-deep in this area of programming to have a good hnderstanding of it.
- johnbender 11y ago
  Event assuming that you can look at the source code for your filesystem/kernel the results of a given write still depend on the conditions and orders under which your writes hit disk relative to other processes' writes. For example, if two processes issue writes to disk sectors that are adjacent but one of the processes' writes also affects a different sector farther away on the physical disk the I/O scheduler may re-order processes' sector level writes. Normally journalling addresses this issue but if the journalling transactions are reordered then it's no help at he application level.
- ktRolster 11y ago
  The problem isn't just the mainline source code, you also have drivers which of course vary for each device. Then even completely beyond control of the kernel, devices have their own RAM caches, which means that even if the device reports that something was written, it might not have been. So there's no way to absolutely be sure.
- lyle_nel 11y ago
  The Linux Documentation Project has this to say about flushing of buffer cache: "In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes."
  - planckscnst 11y ago
    Wow, I'd love to know what they mean by "imperfect". In Linux, specifically, the sync_dirty_buffers syncs all the dirty write buffers (nope - see edit) and isn't imperfect or incomplete in any particular way. Edit: Nevermind, I'm remebering now there is a dirty_background_ratio separate from a dirty_ratio. It will sync some of the dirty buffers as long as it's above dirty_background_ratio and under dirty_ratio. And it was actually bdflush that determined all of this; the sync_dirty_buffer function just took a specific buffer object and synced that. Edit 2: And I'm finding out that information seems to be outdated. Here's the old documentation about bdflush [1]. That was either moved or removed in the latest version, but I did find this [2], which suggests that there was an old bdflush vs something else now. Looks like I missed something important and have some research to do. [1] http://lxr.free-electrons.com/source/Documentation/sysctl/vm.txt?v=2.4.37#L53 http://lxr.free-electrons.com/source/Documentation/sysctl/vm... [2] http://lxr.free-electrons.com/source/Documentation/vm/active_mm.txt http://lxr.free-electrons.com/source/Documentation/vm/active...
faragon 11y ago
Rule of thumb for OS disk I/O: write as soon as possible, i.e. write as soon as possible without hurting performance using buffers on memory for slow component amortization, spreading operations during longer periods.
- planckscnst 11y ago
  The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync(). The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His trust in the OS was damaged and now he or she is understandably grumpy about it.
  - rdtsc 11y ago
    > How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-ish system which was recording data to disk, and noticed recording thread would time-out. (Timeouts were in the 10s of seconds and there were noticed by a watchdog system). Thought 10 seconds should be enough for that process. But it turns out because of how priorities were setup and how fast writes went, dirty page flushing was not keeping up with writes. Periodically it would hit the top threshold and at that point any process doing disk writes would be blocked. Was able to get it under control by essentially doing what gp suggested, writing a little bit at a time, but more often. The total throughput probably went went down but performance was smoothed out quite a bit.
    - planckscnst 11y ago
      That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);". Can you now tell the users that the transaction is complete?
rogerbinns 11y ago
A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().
- striking 11y ago
  I wonder what the implications of making an SQLite filesystem would be.
  - jclulow 11y ago
    If you then expose that file system through a POSIX file system API, you have all of the same issues of underspecified or unclear behaviour that the article mentions.
    - derefr 11y ago
      If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store, and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstractions.
      - hendzen 11y ago
        Microsoft actually tried something similar with WinFS [0] in the 00's, but the project failed. [0] - https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
        derefr 11y ago
        Oddly, Apple's Core Data framework achieves pretty much all the goals WinFS set out to achieve—but does so sitting (for no good reason, really) on top of a regular filesystem. It'd actually be pretty easy to flip things around: give Core Data a raw block device persistence backend, and then write a filesystem driver on top of it. One of my recently-planned hobby projects was to write a FUSE filesystem for OSX that would take all the shell-level glop (LaunchServices UTI-associated verbs, NSFileManager attributes, Spotlight metadata, etc.) and stick it onto the files themselves as plain xattrs, so POSIX tools could effectively be made to work with shell-namespace "Documents" (including treating directory-packages as files, and traversing archive files and container-file-formats as if they were directory-packages.) It'd be quite interesting to combine the two experiments, actually; it'd result in something very much like a POSIX compatibility layer on top of a WinFS-like filesystem.
userbinator 11y ago
I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepting changes that break this de facto standard'. You might even get this into the Single Unix Specification in some form if you tried hard, because I really do think there's a need here. Or we could just have everyone perform regular backups like they already are/should be doing, and decide that if systems are crashing so frequently as to lose data often enough, trying to "fix" this "problem" by adding what would likely be another mass of design-by-committee complexity to filesystems is not addressing the cause but only its symptoms. Then again, with over two decades of experience using the FAT filesystem and never a single instance of unrecoverable data loss despite sudden crashes while hearing countless tales of others corrupting their data frequently even when using far more complex and "robust" filesystems, it makes me wonder why I don't seem to suffer quite the same problems...
- wanderr 11y ago
  Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. You know, before a backup has a chance to run. Sure, those events should be extremely rare but that doesn't mean that we can or should just ignore it, at a large enough scale even extremely improbable events are guaranteed to happen.
  - userbinator 11y ago
    Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. It's not a "problem", it's just reality. You can't predict when exactly the crash will occur relative to the disk write or backups.
    - colanderman 11y ago
ChuckMcM 11y ago
One of the engineers at Google took the time to figure this out, and updated the page out code in the Linux kernel they were using so that the "correct" steps were known if you had to know your data was on disk. It was discussed on LKML as I recall but considered "not generally useful" and I doubt it made it into the main sources. One of the interesting things about writes is that they disrupt reads more significantly than you might expect. Greg Lindahl characterized the impact at Blekko when we were crawling so that we could optimize writes from the crawl to not disrupt latency on the search engine side. Later we completely separated those functions for similar reasons. I believe every disk I've evaluated over the years is slowest on the random read/write 50% test.
- _yosefk 11y ago
  Even DRAM will be slowest on a random read/write 50% test (it has read-to-write and write-to-read penalties; admittedly these will be dwarfed by precharge/activate penalties, but still, mixing reads and writes will make things worse.)
cowsandmilk 11y ago
As long as your hardware doesn't lie as well?? http://brad.livejournal.com/2094221.html http://brad.livejournal.com/2094221.html http://brad.livejournal.com/2116715.html http://brad.livejournal.com/2116715.html
em3rgent0rdr 11y ago
I've been told to type sync into terminal whenever I want writes to complete.
- jclulow 11y ago
  The sync(1M) command provided on many UNIX systems is often a thin wrapper around the sync(2) system call. The sync(2) entry point does not, as specified, guarantee that all inflight data _has_ been written; it's more of a guideline than an actual rule. The user must subsequently wait for the data to actually make it to disk, often for an arbitrarily long amount of time.
- rsync 11y ago
  "I've been told to type sync into terminal whenever I want writes to complete." If you'd really like a guarantee, you can always unmount the filesystem, or alternatively, mount it read-only: mount -ur /mnt/blah That will guarantee, at least at the filesystem/OS level, that the writes are committed.
  - rocky1138 11y ago
    My understanding that, after that point, the hardware drivers could lie and say they've written it and we'd never know, correct? And even after that, the hardware could return a "written OK" value but not actually do the job, right? So the point is that without complete transparency from end-to-end, there's no way to tell. Is that correct?
    - tremon 11y ago
      Yes, and there's also the problem that magnetization of material isn't permanent so your data may disappear after N days even if it were written correctly (typically, N > 5000). Should your "complete transparency" model also include expected longevity of the written bits?
ape4 11y ago
I think the man page http://linux.die.net/man/2/fsync http://linux.die.net/man/2/fsync answers the questions the article asks about fsync(). fsync() = YES metadata, YES data, NO dir entry fdatasync() = NO metadata, YES data, NO dir entry fsync() of dir = NO metadata, NO data, YES dir entry
- kr7 11y ago
  For Linux, but it's not portable across all Unixes. fsync is allowed to do nothing (1) and does not need to work on directories (2). On Mac OSX, fsync will not flush the disk cache, which can lead to data loss (3). (1) http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... (2) http://austingroupbugs.net/view.php?id=672 http://austingroupbugs.net/view.php?id=672 (3) https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html https://developer.apple.com/library/mac/documentation/Darwin...
  - valleyer 11y ago
    fsync(2) on OS X and other Darwins also does not sync metadata. You need to use fcntl(2) with F_FULLFSYNC for that.
    - ryao 11y ago
      It looks like doing `sysctl -w kern.always_do_fullfsync=1` Will produce a sane behavior: https://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/hfs/hfs_vnops.c https://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/hf... Alternatively, you could just use the ZFS driver for Mac OS X. It will do fsync properly.
Ericson2314 11y ago
Yet another example about how the filesystem, and its POSIX realization, is in fact a horrible abstraction.
- Ericson2314 11y ago
  Actually make that about how most layers / abstractions for IO in mainstream OSes are garbage.
- jstimpfle 11y ago
  I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly also ext4, which introduced a shift in paradigm, which broke old applications (or at least broke their performance, for example dpkg). - We're talking about a hierarchical file system, meaning it's easy for humans to find data, but terrible for machines because they must hunt pointers to get to files. Similar problem for syncing a bunch of logically connected files if they are not stored in a single directory. (How do you atomically commit multiple changes across directories?) What would be a better abstraction or non-abstraction (that's still a hierarchical file system)?
  - Ericson2314 11y ago
    A friend of mine had the realization that all optimization boils down to making lower layers understand higher layers' concerns, or higher level layers understand on lower levels' concerns. Here are some examples off the top of my head: POSIX fails as a high level interface: - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) - Overall pushes people to maintain their own structure within files rather than use FS's trees. POSIX fails as a low level interface: - No way to hint or control caching along the memory hierarchy. - Block size, locality out of control. - Can't statically disallow various non-free actions which are costly (ability to expand file), can only hope that not using them does incur penalty. Perhaps ZFS an BTRFS made other approaches more viable/obvious, but these weakness are inherent to the API itself and the high level stuff was noticeably missing from the get-go.
leni536 11y ago
What about O_DIRECT | O_SYNC options in linux's open()?
- rwmj 11y ago
  O_SYNC has abysmal performance. O_DIRECT has very underspecified semantics, but particularly it demands that you only read and write whole "blocks" (where "block" depends on the filesystem type, underlying block device and phase of the moon).
  - ymse 11y ago
    See also Linux Torvalds' opinion on O_DIRECT: The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances. https://lkml.org/lkml/2002/5/11/58 https://lkml.org/lkml/2002/5/11/58
kazinator 11y ago
Is it just "Unix"? What if fsync returns when the hardware indicates that it has completed a write, but it's actually in some drive controller cache for a few moments more?
- keypusher 11y ago
  If you care about these types of things, you can tell the drives to not cache writes.
jstimpfle 11y ago
I don't see why you would have to fsync the "parent of the directory", too. That may not be explicitly specified, but it works just this way: Everything is a file ("object"), whether it's a standard file, or a directory. If you create a new file, you want two things synced: the contents of the file, and the linking of the file (that is the pointer from the directory in which you created the file, to the file object). If you don't sync the link you may not be able to find the file again, even though it was synced to the disk just fine. (It's just how git works btw.) The link to the file is part of the directory object's contents. That's why the directory needs to be synced. There is no need to sync the "parent of the directory" because that was never modified.
keypusher 11y ago
We needed this recently when doing some custom filesystem work, ended up patching our kernel with something we found in a discussion from lkml.
late2part 11y ago
Can I solve most of this in linux like Ubuntu 14 by doing "# sync" ?
lazyant 11y ago
Not sure if I understand the problem. You don't want in most cases to flush out file changes or dirty memory to disk, so you an batch those operations or write only final change and not all intermediary ones. You don't want to wait forever since RAM is volatile. This is why databases have setting for checkpoint intervals, you can set the % or time for dirty pages to be written, I thought the default was 30 seconds, doesn't the same idea apply for file buffers?
Animats 11y ago
I've suggested an approach to this before. There should be several types of files. - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()" - "Temp" files always go away completely after a crash. This should be the default for designated temp directories. - "Log" files can only be appended. Writers cannot seek. The file system guarantees that after a crash, the end of the file is at the end of some write; the file may not tail off into junk. This should be the default for files opened for append. - "Managed" files are for structured databases. They have an additional API. "writemanaged()" has a callback parameter. It returns when the data has been queued, but in addition, the writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. This provides fine-grained information of when data has been committed to disk, which is what a database needs. It improves performance by not blocking waiting for disk commit to take place. The database can have several I/O operations going at once in different parts of the file without blocking.
- jstimpfle 11y ago
  > - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2): If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing" means to create a _new_ file, writing it, syncing it, and then replacing the old version with rename(2) (and fsyncing the directory). > - "Temp" files always go away completely after a crash. This should be the default for designated temp directories. You can have that: Just don't link the inode of the file. On Linux, you can use O_TMPFILE. Alternatively, just create the file, and keep it open but immediately unlink it (it's a little hack but nothing dramatic). You can also write to /tmp. Normally that gets cleaned at reboot, which might be enough. > - "Log" files can only be appended. In open(2), look for O_APPEND. > Writers cannot seek. That's not how you write a kernel API. If you don't want to seek, then don't. > The file system guarantees that after a crash, the end of the file is at the end of some write How do you intend to implement that? Disks don't support arbitrarily large atomic commits. Making such a guarantee in software is necessarily inefficient. The kernel will not choose your tradeoffs for you. You can do it from userland. Decide yourself what is the least bad way to do it. > "Managed" files are for structured databases. [..] The writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. Fsync does just that, except it isn't asynchronous. Maybe there's something behind aio(7)?
jwatte 11y ago
fdatasync() followed by fsync() "should" do it although on very recently created/mounted directories you'll want to use sync() as well. Luckily sync() is synchronous on Linux (sanity!) even though traditional Unix doesn't require it to be. So perhaps the article can be shortened to "sync() is not guaranteed to be synchronous on non-Linux Unix."
lamontcg 11y ago
Even if you solve all these problems, if your RAID array completely loses its mind, then you can still lose/corrupt data. The best solution here it to replicate data across multiple machines, and take good backups that you can restore from, and plan on corruption to happen. You also need to assess exactly how much you really need those transactions. If they're financial transactions where each one could be millions of dollars of stock, then you probably need to care about this a bit. For most web transactions I think the risk/cost analysis here is that you don't need to worry about being perfect. You might lose the last one or two transactions in the case of a hard crash, but your customer service team should be able to handle that and fix it for the customer out of band. You really should consider if you have a business justifcation for worrying about perfection. On the other hand, I did live through the ext2 era before ext3 was production ready, and fully async super-duper fast filesystems really are bad. They would regularly corrupt the disk and require rebuilds. That wasn't a data availability problem, but one or two of those a week was problematic when it came to the operational load (out of 400-800 servers that we managed at the time). We later scaled out ext3 to 30,000 servers with some sets of servers having 2,500 hosts of basically the same type of webserver, and while kernel crashes were a daily issue, corruption and rebuilding was relatively low. If you're not at Google/Amazon/etc and dealing with servers counts an order of magnitude higher than this, you don't really need to worry about it. ext3 or ext4 and fdatasync should be fine, and then apply proper levels of engineering principles to ensure that you don't stay offline for too long or lose too much data. You are dealing with free commodity hardware and software that isn't ever going to be perfect. If you really needed to never lose a transaction you'd probably be buying some kind of awfully expensive mainframe system. Oh and I do recall one case of filesystem corruption leading to a service being down for over a week and probably the loss of a multi-million dollar business deal. But in that case the software ran on a single box. The dev team that was responsible for it never saw that as a problem even though the ops/sysadmin teams kinda yelled at them about it. Then one day the RAID array lost its mind and the server was unrecoverable. When we attempted to rebuild it, it was discovered that over the years the software devs had tweaked the versions of libraries that their software linked against and by crashing all that information had been lost, so it took ages to debug and find the right incantations to get it all back up again. Huge business risk there, but nothing that could be mitigated by naval gazing analysis of filesystems and fdatasync -- backups, documentation, replication, proper config management practices, etc were what was needed.
zzzcpan 11y ago
> One issue is that unlike many other Unix API issues, it's impossible to test to see if you got it all correct and complete. If your steps are incomplete, you don't get any errors; your data is just silently sometimes at risk. There is no problem with the API. The issue is in an underlying assumption, that data sometimes is not at risk. Sadly or maybe luckily, there is nothing you can do to guaranty 100% durability, no point trying to do the impossible. Your data is always silently at risk. Accept it, deal with it, minimize the risk, if you need to. Like replicate your data synchronously cross continent.
zakalwe2000 11y ago
There's worse problems at a lower level - hardware caches on the disk do not really guarantee flushes are honored either. There's a discussion of this here - honoring writes is an enterprise feature... http://serverfault.com/questions/460864/safety-of-write-cache-on-sata-drives-with-barriers http://serverfault.com/questions/460864/safety-of-write-cach...
cmurf 11y ago
This is kinda old now, 2013, but I found it a useful quick read that's related to some of these issues, at least on Linux. Many of the comments are also interesting. Atomic I/O operations in Linux https://lwn.net/Articles/552095/ https://lwn.net/Articles/552095/
schmichael 11y ago
Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data. While there's no excuse for bad documentation or poor APIs, you can never consider data written to a single local disk "safe". It never is. It's a shame making a best effort at safety is nontrivial, but it does force developers to write more defensive and crash-safe code which is all that can save you in the end.
- GauntletWizard 11y ago
  Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.
  - planckscnst 11y ago
    It's even worse than that. All writes have some probability of being incorrect, even after having been written to disk properly (the programmer did the right thing, the OS did the right thing, the filesystem did the right thing, and the disk did the right thing). After writing, your data can be modified by actions on other cells or by simply leaving it alone completely (see read disturbance and charge leakage). Some of this can be fixed by error correcting codes, but there is still a chance of loosing data simply by doing nothing wrong. So yes, agreed++. You need to have a level of redundancy appropriate to the criticality of the data.
- memracom 11y ago
  EBS, Riak, etc. Lots of easy solutions to this problem of putting the data in several diverse places with one write call. Use what works and don't complain that broken stuff is broken.
marvy 11y ago
I ran across a nice paper a few years ago: Rethink the Sync. https://www.usenix.org/legacy/event/osdi06/tech/nightingale/nightingale.pdf https://www.usenix.org/legacy/event/osdi06/tech/nightingale/... The basic idea is that the file system provides two guarantees, one boring, and one interesting. I'll illustrate using code instead of words. The boring guarantee is: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document fclose(f); The system will try to make the write durable within 5 seconds of executing the second line. The interesting guarantee is this: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document puts("Saving complete!") fclose(f); The system guarantees that the user will not see the message until the data is safely on disk! The way they do this is to implement a fancy dependency tracking mechanism that makes sure that the computer never generates output until the writes that the output depends on have completed. They do a bunch of benchmarks that show that their system is almost as fast as mounting ext3 _asynchronously_. (In fact, not much worse than a RAM disk even.) Of course, they also show that in the case of power loss, their system behaves well, whereas ext3 does not, unless you turn up all the paranoia knobs to 11, and then the performance is WAY worse than their system. I'm oversimplifying quite a bit, since this comment is already pretty long. Read the paper for details. Or ask questions here, but I'll probably forget to check the comments, because HN doesn't remind me :(
- marvy 11y ago
  Looks like I remembered, but there aren't any comments.
- planckscnst 11y ago
  I misread this comment and it could be dangerous if someone who doesn't know misreads it the same way. So I want to put a big warning here. Warning! This comment is _not_ saying that your operating system provides these guarantees. In fact, it almost certainly does not. This is a novel suggestion (and implementation?) presented in this particular paper.
  - marvy 11y ago
    Yes, definitely! If the edit timer hadn't expired, I would rewrite this. I wrote this in a horribly unclear way.
cm2187 11y ago
Stupid question: why don't all computers (irrespective of the OS) have a built in power loss mechanism? It seems to be such a common and obvious problem. 1. The PSU would have a big enough capacitor to keep the computer running for a few seconds at its stated output power 2. The PSU would notify the OS of a power loss 3. The OS would immediately flush all caches and adopt a "brace position". 4. Events are spread system wide so that apps can also flush and brace. It should work even if it is the PSU that fails (as long as the capacitor is there). Surely the problem cannot be the cost. Why don't modern desktops have that feature?
- illotum 11y ago
  I would guess because no promise is much better than a broken promise. No one wants to take blame.
  - cm2187 11y ago
    We have to accept that it wouldn't save the day in all scenarios. If there is a 60MB excel file to save on a slow network drive, the file will be lost. But this mechanism would still reduce file/disk/database corruptions by an order of magnitude.
- rasz_pl 11y ago
  1 is not possible on anything with more than one >100MHz chip
  - cm2187 11y ago
    You mean there is no capacitor powerful enough?
    - rasz_pl 11y ago
      Its one thing to add $0.5 supercap and a diode to SSD (and still the likes of OCZ didnt bother for cost reasons). Its entirely another to slap bank of boost/buck converters, 100F cap and spider web of control circuity all over the place. Adding $20-50 in bom the size of two D batteries just in case something _very rare_ happens is not economically viable. There are better alternatives, like banks of batteries ala google boxes. Turning server into oversized laptop gives you more than couple of seconds buffer.
POSIXprog 11y ago
Most programmers, and especially the hipsterish HN crowd, simply cannot be trusted to write correct file manipulation code. In general, anything they produce will be riddled with race conditions and erroneous assumptions (e.g., that rename works cross-device, that close cannot fail) and that breaks in rare but possibly catastrophic circumstances. The solution is copy-on-write file systems such as ZFS and Btrfs that ensure neither data nor meta-data are ever altered in-place and the reuse of correct file manipulation code (written by adults) rather than rolling your own, either from a library or something higher-level like SQLite.
rikkus 11y ago
I use something akin to djb's Maildir delivery procedure when I want to be as close to sure as I can be that a file 'has been saved'. 1. Create a temp file on the same mount 2. Write data to file, checking return of each write(), then a final fsync(), and close() 3. link() to filename we actually want I still haven't figured out how to be this safe on Windows, because AFAICT there's no atomic way to link() or 'move' a file - and they canned the transactional API for the filesystem. Any pointers to how to write files 'safely' on Windows would be much appreciated!
- pbsd 11y ago
  `ReplaceFile` is supposed to be atomic, at least according to the documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx#applications_updating_a_single_file_with_document-like_data https://msdn.microsoft.com/en-us/library/windows/desktop/hh8...
jakub_g 11y ago
Related link with a very detailed writeup: http://danluu.com/file-consistency/ http://danluu.com/file-consistency/ (surprised no one posted it yet). TL;DR: files are hard; filesystems differ a lot; sqlite is very robust, most other software (git and mercurial including) not that much.
pgaddict 11y ago
FWIW this uncertainty in fsync behavior is essentially the cause of the recent PostgreSQL bug on ext4 (fixed in the last round of minor releases). Basically, on XFS and all other tested filesystems is seems to work just fine, on EXT4 we may loose the last rename() effects. So this makes it difficult to get the durability right, even when the project is as careful about it as PostgreSQL.
amelius 11y ago
But what if, behind the scenes, the hardware is performing similar tricks as the OS?
colin_mccabe 11y ago
UNIX has "a standard way to deal with the file durability problem." If you fsync the file, then fsync the containing directory, that should be durable on all properly behaved setups. Of course this comes with a lot of footnotes. There was a bug on some older Linux kernels where they didn't tell the hard drive to flush the data after an fsync. This is why a lot of people still incorrectly believe that enabling hard disk write cache is unsafe. But this was a bug, and the bug was fixed. There are also reports of hard drives that don't honor flush requests. There's not much UNIX or any other OS can do about this-- if the hardware lies, you are in trouble. You could also imagine a much richer durability API. Empirically, databases need such a richer API rather than simple fsync. The POSIX async I/O standard was supposed to standardize all this, but Linux's glibc just implements it as a thread pool making blocking system calls. If you want real async I/O on Linux, you need to use an OS-specific API.

287 points by robinhouston 11y ago

Can't someone smart just read the source code and figure out exactly under which conditions files get written to the disk?

mwcampbell 11y ago
Which source code? There's more than one implementation of all of the following: OS kernel, disk driver, and filesystem.
- lisper 11y ago
  It only takes one combination to start the chain reaction. If someone identifies one combination of kernel, disk driver, file system, hardware, and syscalls that results in reliable durability then people who care about reliable durability will start using it, and that will eventually turn into a de facto standard which will eventually turn into an actual standard.
  - tamana 11y ago
    Until someone who doesn't care changes some code and that pattern is no longer durable.
- gcc_programmer 11y ago
  All of it - these things aren't straightforward, you need to go knee-deep in this area of programming to have a good hnderstanding of it.
johnbender 11y ago
Event assuming that you can look at the source code for your filesystem/kernel the results of a given write still depend on the conditions and orders under which your writes hit disk relative to other processes' writes. For example, if two processes issue writes to disk sectors that are adjacent but one of the processes' writes also affects a different sector farther away on the physical disk the I/O scheduler may re-order processes' sector level writes. Normally journalling addresses this issue but if the journalling transactions are reordered then it's no help at he application level.
ktRolster 11y ago
The problem isn't just the mainline source code, you also have drivers which of course vary for each device. Then even completely beyond control of the kernel, devices have their own RAM caches, which means that even if the device reports that something was written, it might not have been. So there's no way to absolutely be sure.
lyle_nel 11y ago
The Linux Documentation Project has this to say about flushing of buffer cache: "In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdflush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes."
- planckscnst 11y ago
  Wow, I'd love to know what they mean by "imperfect". In Linux, specifically, the sync_dirty_buffers syncs all the dirty write buffers (nope - see edit) and isn't imperfect or incomplete in any particular way. Edit: Nevermind, I'm remebering now there is a dirty_background_ratio separate from a dirty_ratio. It will sync some of the dirty buffers as long as it's above dirty_background_ratio and under dirty_ratio. And it was actually bdflush that determined all of this; the sync_dirty_buffer function just took a specific buffer object and synced that. Edit 2: And I'm finding out that information seems to be outdated. Here's the old documentation about bdflush [1]. That was either moved or removed in the latest version, but I did find this [2], which suggests that there was an old bdflush vs something else now. Looks like I missed something important and have some research to do. [1] http://lxr.free-electrons.com/source/Documentation/sysctl/vm.txt?v=2.4.37#L53 http://lxr.free-electrons.com/source/Documentation/sysctl/vm... [2] http://lxr.free-electrons.com/source/Documentation/vm/active_mm.txt http://lxr.free-electrons.com/source/Documentation/vm/active...

faragon 11y ago

Rule of thumb for OS disk I/O: write as soon as possible, i.e. write as soon as possible without hurting performance using buffers on memory for slow component amortization, spreading operations during longer periods.

planckscnst 11y ago
The article wasn't about the decision of when to write to disk, but how to actually do it. Say you have a point in your program where you have made the decision that it is necessary to write to disk. How do you actually do that for sure? It's not write() or even necessarily fsync(). The author is finding this frustrating because there are several things to do that can seem arbitrary, random, and counterintuitive. His trust in the OS was damaged and now he or she is understandably grumpy about it.
- rdtsc 11y ago
  > How do you actually do that for sure? It's not write() or even necessarily fsync(). Right. For decision of "when" you can configure a few kernel parameters, namely dirty writes thresholds. That assumes you just do write()s from your process and kernel decides to flush that data out. That can be configured to be a function of time and or amount of unwritten data. I had to do it a few times. Once it was a realtime-ish system which was recording data to disk, and noticed recording thread would time-out. (Timeouts were in the 10s of seconds and there were noticed by a watchdog system). Thought 10 seconds should be enough for that process. But it turns out because of how priorities were setup and how fast writes went, dirty page flushing was not keeping up with writes. Periodically it would hit the top threshold and at that point any process doing disk writes would be blocked. Was able to get it under control by essentially doing what gp suggested, writing a little bit at a time, but more often. The total throughput probably went went down but performance was smoothed out quite a bit.
  - planckscnst 11y ago
    That's still missing the point. Say you're writing a program that stores a log of transactions. People are trading resources and you log those transactions so you can find out who owns which resources. User A just gave 100 units to user B; you need to store this information and tell both users that the transaction is complete. How do you do that? Your program will do something like "write(t_log_fd, t_entry, t_bytes);". Can you now tell the users that the transaction is complete?

rogerbinns 11y ago

A good solution is to use SQLite. It addresses the issues (pretty much by doing all the fsync etc mentioned including on directories) and has a very comprehensive test suite. It is also used very widely on desktops, mobile devices, applications etc. https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html A notable quote: SQLite does not compete with client/server databases. SQLite competes with fopen().

striking 11y ago
I wonder what the implications of making an SQLite filesystem would be.
- jclulow 11y ago
  If you then expose that file system through a POSIX file system API, you have all of the same issues of underspecified or unclear behaviour that the article mentions.
  - derefr 11y ago
    If I were designing a userland from the ground up, I'd probably give processes a transactional MVCC object store, and make guarantees about that; and then implement a "POSIX compatibility layer" file system API in terms of that, but explicitly say that none of the same guarantees from the object-store layer apply. Some days I really do wish we weren't so inured to the particular 50-year-old systems-programming abstractions.
    - hendzen 11y ago
      Microsoft actually tried something similar with WinFS [0] in the 00's, but the project failed. [0] - https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
      - derefr 11y ago
        Oddly, Apple's Core Data framework achieves pretty much all the goals WinFS set out to achieve—but does so sitting (for no good reason, really) on top of a regular filesystem. It'd actually be pretty easy to flip things around: give Core Data a raw block device persistence backend, and then write a filesystem driver on top of it. One of my recently-planned hobby projects was to write a FUSE filesystem for OSX that would take all the shell-level glop (LaunchServices UTI-associated verbs, NSFileManager attributes, Spotlight metadata, etc.) and stick it onto the files themselves as plain xattrs, so POSIX tools could effectively be made to work with shell-namespace "Documents" (including treating directory-packages as files, and traversing archive files and container-file-formats as if they were directory-packages.) It'd be quite interesting to combine the two experiments, actually; it'd result in something very much like a POSIX compatibility layer on top of a WinFS-like filesystem.

userbinator 11y ago

I'll admit that one reason I'm unusually grumpy about this is that I feel rather unhappy not knowing what I need to do to safeguard data that I care about. ...backups? This issue is not unsolvable at a technical level, but it probably is at a political level. Someone would have to determine and write up what is good enough now (on sane setups), and then Unix kernel people would have to say 'enough, we are not accepting changes that break this de facto standard'. You might even get this into the Single Unix Specification in some form if you tried hard, because I really do think there's a need here. Or we could just have everyone perform regular backups like they already are/should be doing, and decide that if systems are crashing so frequently as to lose data often enough, trying to "fix" this "problem" by adding what would likely be another mass of design-by-committee complexity to filesystems is not addressing the cause but only its symptoms. Then again, with over two decades of experience using the FAT filesystem and never a single instance of unrecoverable data loss despite sudden crashes while hearing countless tales of others corrupting their data frequently even when using far more complex and "robust" filesystems, it makes me wonder why I don't seem to suffer quite the same problems...

wanderr 11y ago
Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. You know, before a backup has a chance to run. Sure, those events should be extremely rare but that doesn't mean that we can or should just ignore it, at a large enough scale even extremely improbable events are guaranteed to happen.
- userbinator 11y ago
  Backups do not solve the problem of not knowing whether data you just wrote to disk will still be there in the event of a power outage or system crash. It's not a "problem", it's just reality. You can't predict when exactly the crash will occur relative to the disk write or backups.
  - colanderman 11y ago

ChuckMcM 11y ago

One of the engineers at Google took the time to figure this out, and updated the page out code in the Linux kernel they were using so that the "correct" steps were known if you had to know your data was on disk. It was discussed on LKML as I recall but considered "not generally useful" and I doubt it made it into the main sources. One of the interesting things about writes is that they disrupt reads more significantly than you might expect. Greg Lindahl characterized the impact at Blekko when we were crawling so that we could optimize writes from the crawl to not disrupt latency on the search engine side. Later we completely separated those functions for similar reasons. I believe every disk I've evaluated over the years is slowest on the random read/write 50% test.

_yosefk 11y ago
Even DRAM will be slowest on a random read/write 50% test (it has read-to-write and write-to-read penalties; admittedly these will be dwarfed by precharge/activate penalties, but still, mixing reads and writes will make things worse.)

cowsandmilk 11y ago

As long as your hardware doesn't lie as well?? http://brad.livejournal.com/2094221.html http://brad.livejournal.com/2094221.html http://brad.livejournal.com/2116715.html http://brad.livejournal.com/2116715.html

em3rgent0rdr 11y ago

I've been told to type sync into terminal whenever I want writes to complete.

jclulow 11y ago
The sync(1M) command provided on many UNIX systems is often a thin wrapper around the sync(2) system call. The sync(2) entry point does not, as specified, guarantee that all inflight data _has_ been written; it's more of a guideline than an actual rule. The user must subsequently wait for the data to actually make it to disk, often for an arbitrarily long amount of time.
rsync 11y ago
"I've been told to type sync into terminal whenever I want writes to complete." If you'd really like a guarantee, you can always unmount the filesystem, or alternatively, mount it read-only: mount -ur /mnt/blah That will guarantee, at least at the filesystem/OS level, that the writes are committed.
- rocky1138 11y ago
  My understanding that, after that point, the hardware drivers could lie and say they've written it and we'd never know, correct? And even after that, the hardware could return a "written OK" value but not actually do the job, right? So the point is that without complete transparency from end-to-end, there's no way to tell. Is that correct?
  - tremon 11y ago
    Yes, and there's also the problem that magnetization of material isn't permanent so your data may disappear after N days even if it were written correctly (typically, N > 5000). Should your "complete transparency" model also include expected longevity of the written bits?

ape4 11y ago

I think the man page http://linux.die.net/man/2/fsync http://linux.die.net/man/2/fsync answers the questions the article asks about fsync(). fsync() = YES metadata, YES data, NO dir entry fdatasync() = NO metadata, YES data, NO dir entry fsync() of dir = NO metadata, NO data, YES dir entry

kr7 11y ago
For Linux, but it's not portable across all Unixes. fsync is allowed to do nothing (1) and does not need to work on directories (2). On Mac OSX, fsync will not flush the disk cache, which can lead to data loss (3). (1) http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html http://pubs.opengroup.org/onlinepubs/9699919799/functions/fs... (2) http://austingroupbugs.net/view.php?id=672 http://austingroupbugs.net/view.php?id=672 (3) https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html https://developer.apple.com/library/mac/documentation/Darwin...
- valleyer 11y ago
  fsync(2) on OS X and other Darwins also does not sync metadata. You need to use fcntl(2) with F_FULLFSYNC for that.
  - ryao 11y ago
    It looks like doing `sysctl -w kern.always_do_fullfsync=1` Will produce a sane behavior: https://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/hfs/hfs_vnops.c https://opensource.apple.com/source/xnu/xnu-2422.1.72/bsd/hf... Alternatively, you could just use the ZFS driver for Mac OS X. It will do fsync properly.

Ericson2314 11y ago

Yet another example about how the filesystem, and its POSIX realization, is in fact a horrible abstraction.

Ericson2314 11y ago
Actually make that about how most layers / abstractions for IO in mainstream OSes are garbage.
jstimpfle 11y ago
I don't think there is anything wrong with the POSIX file system API. Two things: - I think it's mainly the modern file systems like Btrfs and I think partly also ext4, which introduced a shift in paradigm, which broke old applications (or at least broke their performance, for example dpkg). - We're talking about a hierarchical file system, meaning it's easy for humans to find data, but terrible for machines because they must hunt pointers to get to files. Similar problem for syncing a bunch of logically connected files if they are not stored in a single directory. (How do you atomically commit multiple changes across directories?) What would be a better abstraction or non-abstraction (that's still a hierarchical file system)?
- Ericson2314 11y ago
  A friend of mine had the realization that all optimization boils down to making lower layers understand higher layers' concerns, or higher level layers understand on lower levels' concerns. Here are some examples off the top of my head: POSIX fails as a high level interface: - No (at all powerful) notion of transactions. Mutation without transactions is extraordinarily primitive. - No multiple FS roots to indicate boundary across which data will never be synchronized. (This is also good for how to spread data across multiple devices, a low-level concern.) - Overall pushes people to maintain their own structure within files rather than use FS's trees. POSIX fails as a low level interface: - No way to hint or control caching along the memory hierarchy. - Block size, locality out of control. - Can't statically disallow various non-free actions which are costly (ability to expand file), can only hope that not using them does incur penalty. Perhaps ZFS an BTRFS made other approaches more viable/obvious, but these weakness are inherent to the API itself and the high level stuff was noticeably missing from the get-go.

leni536 11y ago

What about O_DIRECT | O_SYNC options in linux's open()?

rwmj 11y ago
O_SYNC has abysmal performance. O_DIRECT has very underspecified semantics, but particularly it demands that you only read and write whole "blocks" (where "block" depends on the filesystem type, underlying block device and phase of the moon).
- ymse 11y ago
  See also Linux Torvalds' opinion on O_DIRECT: The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances. https://lkml.org/lkml/2002/5/11/58 https://lkml.org/lkml/2002/5/11/58

kazinator 11y ago

Is it just "Unix"? What if fsync returns when the hardware indicates that it has completed a write, but it's actually in some drive controller cache for a few moments more?

keypusher 11y ago
If you care about these types of things, you can tell the drives to not cache writes.

jstimpfle 11y ago

I don't see why you would have to fsync the "parent of the directory", too. That may not be explicitly specified, but it works just this way: Everything is a file ("object"), whether it's a standard file, or a directory. If you create a new file, you want two things synced: the contents of the file, and the linking of the file (that is the pointer from the directory in which you created the file, to the file object). If you don't sync the link you may not be able to find the file again, even though it was synced to the disk just fine. (It's just how git works btw.) The link to the file is part of the directory object's contents. That's why the directory needs to be synced. There is no need to sync the "parent of the directory" because that was never modified.

keypusher 11y ago

We needed this recently when doing some custom filesystem work, ended up patching our kernel with something we found in a discussion from lkml.

late2part 11y ago

Can I solve most of this in linux like Ubuntu 14 by doing "# sync" ?

lazyant 11y ago

Not sure if I understand the problem. You don't want in most cases to flush out file changes or dirty memory to disk, so you an batch those operations or write only final change and not all intermediary ones. You don't want to wait forever since RAM is volatile. This is why databases have setting for checkpoint intervals, you can set the % or time for dirty pages to be written, I thought the default was 30 seconds, doesn't the same idea apply for file buffers?

Animats 11y ago

I've suggested an approach to this before. There should be several types of files. - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. This should be the default when a file is opened via "creat()" - "Temp" files always go away completely after a crash. This should be the default for designated temp directories. - "Log" files can only be appended. Writers cannot seek. The file system guarantees that after a crash, the end of the file is at the end of some write; the file may not tail off into junk. This should be the default for files opened for append. - "Managed" files are for structured databases. They have an additional API. "writemanaged()" has a callback parameter. It returns when the data has been queued, but in addition, the writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. This provides fine-grained information of when data has been committed to disk, which is what a database needs. It improves performance by not blocking waiting for disk commit to take place. The database can have several I/O operations going at once in different parts of the file without blocking.

jstimpfle 11y ago
> - "Unit" files commit when closed properly (this does not include a program exit without close), and then replace the old version of the file. The file system should guarantee that, after a crash, you have either the old version or the new complete version. Incidentally, this is the way it currently is. From rename(2): If "newpath already exists, it will be atomically replaced". Just don't forget that "replacing" means to create a _new_ file, writing it, syncing it, and then replacing the old version with rename(2) (and fsyncing the directory). > - "Temp" files always go away completely after a crash. This should be the default for designated temp directories. You can have that: Just don't link the inode of the file. On Linux, you can use O_TMPFILE. Alternatively, just create the file, and keep it open but immediately unlink it (it's a little hack but nothing dramatic). You can also write to /tmp. Normally that gets cleaned at reboot, which might be enough. > - "Log" files can only be appended. In open(2), look for O_APPEND. > Writers cannot seek. That's not how you write a kernel API. If you don't want to seek, then don't. > The file system guarantees that after a crash, the end of the file is at the end of some write How do you intend to implement that? Disks don't support arbitrarily large atomic commits. Making such a guarantee in software is necessarily inefficient. The kernel will not choose your tradeoffs for you. You can do it from userland. Decide yourself what is the least bad way to do it. > "Managed" files are for structured databases. [..] The writer gets a callback when the write has committed to disk. The file system must guarantee that a write for which the callback has been made will survive a crash. Fsync does just that, except it isn't asynchronous. Maybe there's something behind aio(7)?

jwatte 11y ago

fdatasync() followed by fsync() "should" do it although on very recently created/mounted directories you'll want to use sync() as well. Luckily sync() is synchronous on Linux (sanity!) even though traditional Unix doesn't require it to be. So perhaps the article can be shortened to "sync() is not guaranteed to be synchronous on non-Linux Unix."

lamontcg 11y ago

Even if you solve all these problems, if your RAID array completely loses its mind, then you can still lose/corrupt data. The best solution here it to replicate data across multiple machines, and take good backups that you can restore from, and plan on corruption to happen. You also need to assess exactly how much you really need those transactions. If they're financial transactions where each one could be millions of dollars of stock, then you probably need to care about this a bit. For most web transactions I think the risk/cost analysis here is that you don't need to worry about being perfect. You might lose the last one or two transactions in the case of a hard crash, but your customer service team should be able to handle that and fix it for the customer out of band. You really should consider if you have a business justifcation for worrying about perfection. On the other hand, I did live through the ext2 era before ext3 was production ready, and fully async super-duper fast filesystems really are bad. They would regularly corrupt the disk and require rebuilds. That wasn't a data availability problem, but one or two of those a week was problematic when it came to the operational load (out of 400-800 servers that we managed at the time). We later scaled out ext3 to 30,000 servers with some sets of servers having 2,500 hosts of basically the same type of webserver, and while kernel crashes were a daily issue, corruption and rebuilding was relatively low. If you're not at Google/Amazon/etc and dealing with servers counts an order of magnitude higher than this, you don't really need to worry about it. ext3 or ext4 and fdatasync should be fine, and then apply proper levels of engineering principles to ensure that you don't stay offline for too long or lose too much data. You are dealing with free commodity hardware and software that isn't ever going to be perfect. If you really needed to never lose a transaction you'd probably be buying some kind of awfully expensive mainframe system. Oh and I do recall one case of filesystem corruption leading to a service being down for over a week and probably the loss of a multi-million dollar business deal. But in that case the software ran on a single box. The dev team that was responsible for it never saw that as a problem even though the ops/sysadmin teams kinda yelled at them about it. Then one day the RAID array lost its mind and the server was unrecoverable. When we attempted to rebuild it, it was discovered that over the years the software devs had tweaked the versions of libraries that their software linked against and by crashing all that information had been lost, so it took ages to debug and find the right incantations to get it all back up again. Huge business risk there, but nothing that could be mitigated by naval gazing analysis of filesystems and fdatasync -- backups, documentation, replication, proper config management practices, etc were what was needed.

zzzcpan 11y ago

> One issue is that unlike many other Unix API issues, it's impossible to test to see if you got it all correct and complete. If your steps are incomplete, you don't get any errors; your data is just silently sometimes at risk. There is no problem with the API. The issue is in an underlying assumption, that data sometimes is not at risk. Sadly or maybe luckily, there is nothing you can do to guaranty 100% durability, no point trying to do the impossible. Your data is always silently at risk. Accept it, deal with it, minimize the risk, if you need to. Like replicate your data synchronously cross continent.

zakalwe2000 11y ago

There's worse problems at a lower level - hardware caches on the disk do not really guarantee flushes are honored either. There's a discussion of this here - honoring writes is an enterprise feature... http://serverfault.com/questions/460864/safety-of-write-cache-on-sata-drives-with-barriers http://serverfault.com/questions/460864/safety-of-write-cach...

cmurf 11y ago

This is kinda old now, 2013, but I found it a useful quick read that's related to some of these issues, at least on Linux. Many of the comments are also interesting. Atomic I/O operations in Linux https://lwn.net/Articles/552095/ https://lwn.net/Articles/552095/

schmichael 11y ago

Disks can buffer, disks can have firmware bugs, disks can fail both catastrophically and subtly. Even the fanciest battery-backed RAID controllers have firmware bugs and dead batteries. Those are just ways your storage hardware can fail you even if the kernel and libraries are bug free and you follow the right mystic incantations to sync data. While there's no excuse for bad documentation or poor APIs, you can never consider data written to a single local disk "safe". It never is. It's a shame making a best effort at safety is nontrivial, but it does force developers to write more defensive and crash-safe code which is all that can save you in the end.

GauntletWizard 11y ago
Yeah, from a SRE perspective, the last N writes are always purely probabilistic. The real quest is to have enough redundancy that that curve is fairly close to 1, and enough failure warning that your system will fix itself before it droops. That means Battery Backup, SMART detection, ECC memory, etc.
- planckscnst 11y ago
  It's even worse than that. All writes have some probability of being incorrect, even after having been written to disk properly (the programmer did the right thing, the OS did the right thing, the filesystem did the right thing, and the disk did the right thing). After writing, your data can be modified by actions on other cells or by simply leaving it alone completely (see read disturbance and charge leakage). Some of this can be fixed by error correcting codes, but there is still a chance of loosing data simply by doing nothing wrong. So yes, agreed++. You need to have a level of redundancy appropriate to the criticality of the data.
memracom 11y ago
EBS, Riak, etc. Lots of easy solutions to this problem of putting the data in several diverse places with one write call. Use what works and don't complain that broken stuff is broken.

marvy 11y ago

I ran across a nice paper a few years ago: Rethink the Sync. https://www.usenix.org/legacy/event/osdi06/tech/nightingale/nightingale.pdf https://www.usenix.org/legacy/event/osdi06/tech/nightingale/... The basic idea is that the file system provides two guarantees, one boring, and one interesting. I'll illustrate using code instead of words. The boring guarantee is: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document fclose(f); The system will try to make the write durable within 5 seconds of executing the second line. The interesting guarantee is this: FILE* f = fopen("autosave.bak", "w"); fwrite(f, buffer, length); // save current document puts("Saving complete!") fclose(f); The system guarantees that the user will not see the message until the data is safely on disk! The way they do this is to implement a fancy dependency tracking mechanism that makes sure that the computer never generates output until the writes that the output depends on have completed. They do a bunch of benchmarks that show that their system is almost as fast as mounting ext3 _asynchronously_. (In fact, not much worse than a RAM disk even.) Of course, they also show that in the case of power loss, their system behaves well, whereas ext3 does not, unless you turn up all the paranoia knobs to 11, and then the performance is WAY worse than their system. I'm oversimplifying quite a bit, since this comment is already pretty long. Read the paper for details. Or ask questions here, but I'll probably forget to check the comments, because HN doesn't remind me :(

marvy 11y ago
Looks like I remembered, but there aren't any comments.
planckscnst 11y ago
I misread this comment and it could be dangerous if someone who doesn't know misreads it the same way. So I want to put a big warning here. Warning! This comment is _not_ saying that your operating system provides these guarantees. In fact, it almost certainly does not. This is a novel suggestion (and implementation?) presented in this particular paper.
- marvy 11y ago
  Yes, definitely! If the edit timer hadn't expired, I would rewrite this. I wrote this in a horribly unclear way.

cm2187 11y ago

Stupid question: why don't all computers (irrespective of the OS) have a built in power loss mechanism? It seems to be such a common and obvious problem. 1. The PSU would have a big enough capacitor to keep the computer running for a few seconds at its stated output power 2. The PSU would notify the OS of a power loss 3. The OS would immediately flush all caches and adopt a "brace position". 4. Events are spread system wide so that apps can also flush and brace. It should work even if it is the PSU that fails (as long as the capacitor is there). Surely the problem cannot be the cost. Why don't modern desktops have that feature?

illotum 11y ago
I would guess because no promise is much better than a broken promise. No one wants to take blame.
- cm2187 11y ago
  We have to accept that it wouldn't save the day in all scenarios. If there is a 60MB excel file to save on a slow network drive, the file will be lost. But this mechanism would still reduce file/disk/database corruptions by an order of magnitude.
rasz_pl 11y ago
1 is not possible on anything with more than one >100MHz chip
- cm2187 11y ago
  You mean there is no capacitor powerful enough?
  - rasz_pl 11y ago
    Its one thing to add $0.5 supercap and a diode to SSD (and still the likes of OCZ didnt bother for cost reasons). Its entirely another to slap bank of boost/buck converters, 100F cap and spider web of control circuity all over the place. Adding $20-50 in bom the size of two D batteries just in case something _very rare_ happens is not economically viable. There are better alternatives, like banks of batteries ala google boxes. Turning server into oversized laptop gives you more than couple of seconds buffer.

POSIXprog 11y ago

Most programmers, and especially the hipsterish HN crowd, simply cannot be trusted to write correct file manipulation code. In general, anything they produce will be riddled with race conditions and erroneous assumptions (e.g., that rename works cross-device, that close cannot fail) and that breaks in rare but possibly catastrophic circumstances. The solution is copy-on-write file systems such as ZFS and Btrfs that ensure neither data nor meta-data are ever altered in-place and the reuse of correct file manipulation code (written by adults) rather than rolling your own, either from a library or something higher-level like SQLite.

rikkus 11y ago

I use something akin to djb's Maildir delivery procedure when I want to be as close to sure as I can be that a file 'has been saved'. 1. Create a temp file on the same mount 2. Write data to file, checking return of each write(), then a final fsync(), and close() 3. link() to filename we actually want I still haven't figured out how to be this safe on Windows, because AFAICT there's no atomic way to link() or 'move' a file - and they canned the transactional API for the filesystem. Any pointers to how to write files 'safely' on Windows would be much appreciated!

pbsd 11y ago
`ReplaceFile` is supposed to be atomic, at least according to the documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx#applications_updating_a_single_file_with_document-like_data https://msdn.microsoft.com/en-us/library/windows/desktop/hh8...

jakub_g 11y ago

Related link with a very detailed writeup: http://danluu.com/file-consistency/ http://danluu.com/file-consistency/ (surprised no one posted it yet). TL;DR: files are hard; filesystems differ a lot; sqlite is very robust, most other software (git and mercurial including) not that much.

pgaddict 11y ago

FWIW this uncertainty in fsync behavior is essentially the cause of the recent PostgreSQL bug on ext4 (fixed in the last round of minor releases). Basically, on XFS and all other tested filesystems is seems to work just fine, on EXT4 we may loose the last rename() effects. So this makes it difficult to get the durability right, even when the project is as careful about it as PostgreSQL.

amelius 11y ago

But what if, behind the scenes, the hardware is performing similar tricks as the OS?

colin_mccabe 11y ago

UNIX has "a standard way to deal with the file durability problem." If you fsync the file, then fsync the containing directory, that should be durable on all properly behaved setups. Of course this comes with a lot of footnotes. There was a bug on some older Linux kernels where they didn't tell the hard drive to flush the data after an fsync. This is why a lot of people still incorrectly believe that enabling hard disk write cache is unsafe. But this was a bug, and the bug was fixed. There are also reports of hard drives that don't honor flush requests. There's not much UNIX or any other OS can do about this-- if the hardware lies, you are in trouble. You could also imagine a much richer durability API. Empirically, databases need such a richer API rather than simple fsync. The POSIX async I/O standard was supposed to standardize all this, but Linux's glibc just implements it as a thread pool making blocking system calls. If you want real async I/O on Linux, you need to use an OS-specific API.