3 ms·

At the point where it disappears for seconds it's a distributed system not an attached disk. At this point you have to start applying the CAP theorem. At least

At the point where it disappears for seconds it's a distributed system not an attached disk. At this point you have to start applying the CAP theorem. At least in Unix the assumption is that disks are always attached (and reliable...) so write errors don't typically bubble up to the application layer. This is why losing an NFS mount typically just hangs the system until it recovers.

deathanatos 3y ago
> At least in Unix the assumption is that disks are always attached (and reliable...) I want to say a physical disk being yanked from the system (essentially what was happening, as far as the OS could tell) does cause I/O errors in Linux? I could be wrong though, this isn't exactly something I try to exercise. As for it being a distributed system … I suppose? But that's what an FS's log is for: when the drive reconnects, there will either be a pending WAL entry, or not. If there is, the write can be persisted, otherwise, it is lost. But consistency should still happen. Now, an app might not be ready for that, but that's an app bug. But it can always happen that the power goes out, which in my situation is equivalent to a disk yank. There's also small children, the SO tripping over a cable, etc. But these are different failure modes from some of what the above post listed, such as disks undoing acknowledged writes, or lying about having persisted the write. (Some of the examples are byzantine, some are not.)
- LtdJorge 3y ago
  Ackchually, the journal is not a WAL (a journal is a log of intended and completed actions, while the WAL actually stores the data before committing it to the data store, it's more advanced and can recover all data ). But yeah, I agree that is what the journal is for. However, I don't know if the journal in "modern" filesystems works the same when a drive reconnects than when the power is cut out (the OS still being alive and all in the first case).
  - deathanatos 3y ago
    I would expect that it must: the OS cannot assume that the newly connected drive wasn't altered by another system in the meantime. (Or that it's the same FS, even.) (although I grant that, a.) the FS probably has a UUID that idents it, and b.) it's only gone for a few ms — but those aren't things that, reasonably, the OS is going to look at to go "eh, we don't need to recover this newly attached disk".) (I'm not sure I'm still grokking the differences between a WAL & a journal, to you.)
- mprovost 3y ago
  The OS notices when disks or NFS servers ("NFS server not responding") go missing, but there's no way to generally broadcast this information to every process. And then you would have to build in error handling everywhere. What does `cat` do when it's informed that the file it's reading is temporarily unavailable? You can either block the next read() call and hang the process until the disk comes back, or return a hard error that it's gone forever. Otherwise each application has to have its own wait/timeout code. (Which is what applications dealing with HTTP have to deal with but at least when you make an HTTP connection you're aware that something could go wrong.)
  - deathanatos 3y ago
    NFS is a bit of a pathological case. > but there's no way to generally broadcast this information to every process. Sure there is: pending I/O returns EIO. > What does `cat` do when it's informed that the file it's reading is temporarily unavailable? Let's test it. I inserted a USB disk, mounted it, started `cat`, and yanked the disk. `cat`'s read(2) call returns EIO, and cat dies with an error. I'd expect similar results from most processes: for applications, there is no sane, in-process way to recover from such a situation; either you can rely on the FS's capabilities, if any, or you need to build those into your files (e.g., how SQLite does). > And then you would have to build in error handling everywhere. Apps that presume I/O isn't fallible are broken. Disconnection is hardly the only possible error during I/O. NFS is, again, rather pathological: the FS driver there I think just keeps the mount alive, and pends the writes until a connection is re-established. I'd argue this is terrible behavior, too: the probability with which an SRE has to debug a hung, nigh unkillable process when NFS is involved in system design is 100%.

deathanatos 3y ago

> At least in Unix the assumption is that disks are always attached (and reliable...) I want to say a physical disk being yanked from the system (essentially what was happening, as far as the OS could tell) does cause I/O errors in Linux? I could be wrong though, this isn't exactly something I try to exercise. As for it being a distributed system … I suppose? But that's what an FS's log is for: when the drive reconnects, there will either be a pending WAL entry, or not. If there is, the write can be persisted, otherwise, it is lost. But consistency should still happen. Now, an app might not be ready for that, but that's an app bug. But it can always happen that the power goes out, which in my situation is equivalent to a disk yank. There's also small children, the SO tripping over a cable, etc. But these are different failure modes from some of what the above post listed, such as disks undoing acknowledged writes, or lying about having persisted the write. (Some of the examples are byzantine, some are not.)

LtdJorge 3y ago
Ackchually, the journal is not a WAL (a journal is a log of intended and completed actions, while the WAL actually stores the data before committing it to the data store, it's more advanced and can recover all data ). But yeah, I agree that is what the journal is for. However, I don't know if the journal in "modern" filesystems works the same when a drive reconnects than when the power is cut out (the OS still being alive and all in the first case).
- deathanatos 3y ago
  I would expect that it must: the OS cannot assume that the newly connected drive wasn't altered by another system in the meantime. (Or that it's the same FS, even.) (although I grant that, a.) the FS probably has a UUID that idents it, and b.) it's only gone for a few ms — but those aren't things that, reasonably, the OS is going to look at to go "eh, we don't need to recover this newly attached disk".) (I'm not sure I'm still grokking the differences between a WAL & a journal, to you.)
mprovost 3y ago
The OS notices when disks or NFS servers ("NFS server not responding") go missing, but there's no way to generally broadcast this information to every process. And then you would have to build in error handling everywhere. What does `cat` do when it's informed that the file it's reading is temporarily unavailable? You can either block the next read() call and hang the process until the disk comes back, or return a hard error that it's gone forever. Otherwise each application has to have its own wait/timeout code. (Which is what applications dealing with HTTP have to deal with but at least when you make an HTTP connection you're aware that something could go wrong.)
- deathanatos 3y ago
  NFS is a bit of a pathological case. > but there's no way to generally broadcast this information to every process. Sure there is: pending I/O returns EIO. > What does `cat` do when it's informed that the file it's reading is temporarily unavailable? Let's test it. I inserted a USB disk, mounted it, started `cat`, and yanked the disk. `cat`'s read(2) call returns EIO, and cat dies with an error. I'd expect similar results from most processes: for applications, there is no sane, in-process way to recover from such a situation; either you can rely on the FS's capabilities, if any, or you need to build those into your files (e.g., how SQLite does). > And then you would have to build in error handling everywhere. Apps that presume I/O isn't fallible are broken. Disconnection is hardly the only possible error during I/O. NFS is, again, rather pathological: the FS driver there I think just keeps the mount alive, and pends the writes until a connection is re-established. I'd argue this is terrible behavior, too: the probability with which an SRE has to debug a hung, nigh unkillable process when NFS is involved in system design is 100%.