4 ms·

I think the only real defense of the linux et al behaviour here is that: a) there's no standardized way to recover from write errors when not removing dirty da

I think the only real defense of the linux et al behaviour here is that: a) there's no standardized way to recover from write errors when not removing dirty data. I personally find that more than an acceptable tradeoff, but obviously not everybody considers it that way. b) If IO errors, especially when triggered by writeback, cause kernel resources to be retained (both memory for dirty data, but even just a per-file data to stash a 'failed' bit), it's possible to get the kernel stuck in a way it needs memory without a good way to recover from that. I personally think that can be handled in smarter ways by escalating per-file flags to per-fs flags if there's too many failures, and remounting ro, but not everybody sees it that way...

loeg 8y ago
That sounds reasonable to me. > b) If IO errors, especially when triggered by writeback, cause kernel resources to be retained (both memory for dirty data, but even just a per-file data to stash a 'failed' bit), it's possible to get the kernel stuck in a way it needs memory without a good way to recover from that. To put it more succinctly: userspace is not allowed to leak kernel resources. (The page cache, clean and dirty, is only allowed to persist outside the lifetime of user programs because the kernel is free to reclaim it at will — writing out dirty pages to clean them.) > I personally think that can be handled in smarter ways by escalating per-file flags to per-fs flags if there's too many failures, and remounting ro, but not everybody sees it that way... Yeah, I think that's the only real sane option. If writes start erroring and continue to error on a rewrite attempt, the disk is failing or gone. The kernel has different error numbers for these (EIO or ENXIO). If failing, the filesystem must either re-layout the file and mark the block as bad (if it supports such a concept), or fail the whole device by with a per-fs flag. A failed filesystem should probably be RO and fail any write or fsync operation with EROFS or EIO. If the device has been failed, it's ok to clean the lost write and discard it, releasing kernel resources, because we know all future writes and fsyncs to that file / block will also report failure. This model isn't super difficult to understand or implement and it's easier for userspace applications to understand. The only "con" argument might be that users are prevented from writing to drives that have "only a few" bad blocks (if the FS isn't bad block aware). I don't find that argument persuasive — once a drive develops some bad sectors, they tend to develop more rapidly. And allowing further buffered writes can really exacerbate the amount of data lost, for example, when an SSD reaches its lifetime write capacity.
pgaddict 8y ago
Isn't one of the difficulties the inability to decide when the I/O error is transient (e.g. running out of space with thin provisioning) vs. permanent (e.g. drive eaten by a velociraptor)? Also, isn't it possible to use multipath to queue writes in case of error? I wonder if that's safe, though, because it will keep it in memory only and make it look OK to the caller.
- anarazel 8y ago
  > Isn't one of the difficulties the inability to decide when the I/O error is transient (e.g. running out of space with thin provisioning) vs. permanent (e.g. drive eaten by a velociraptor)? I don't really think so. The thin provisioning thing really is just a series of bugs in the kernel, because the thin provisioning code just didn't (and maybe doesn't), bubble up errors with enough information. Even if errors are transient, at some point the decision to shut down the FS on a device with enough transient errors is a much better fix than throwing away data.

loeg 8y ago

That sounds reasonable to me. > b) If IO errors, especially when triggered by writeback, cause kernel resources to be retained (both memory for dirty data, but even just a per-file data to stash a 'failed' bit), it's possible to get the kernel stuck in a way it needs memory without a good way to recover from that. To put it more succinctly: userspace is not allowed to leak kernel resources. (The page cache, clean and dirty, is only allowed to persist outside the lifetime of user programs because the kernel is free to reclaim it at will — writing out dirty pages to clean them.) > I personally think that can be handled in smarter ways by escalating per-file flags to per-fs flags if there's too many failures, and remounting ro, but not everybody sees it that way... Yeah, I think that's the only real sane option. If writes start erroring and continue to error on a rewrite attempt, the disk is failing or gone. The kernel has different error numbers for these (EIO or ENXIO). If failing, the filesystem must either re-layout the file and mark the block as bad (if it supports such a concept), or fail the whole device by with a per-fs flag. A failed filesystem should probably be RO and fail any write or fsync operation with EROFS or EIO. If the device has been failed, it's ok to clean the lost write and discard it, releasing kernel resources, because we know all future writes and fsyncs to that file / block will also report failure. This model isn't super difficult to understand or implement and it's easier for userspace applications to understand. The only "con" argument might be that users are prevented from writing to drives that have "only a few" bad blocks (if the FS isn't bad block aware). I don't find that argument persuasive — once a drive develops some bad sectors, they tend to develop more rapidly. And allowing further buffered writes can really exacerbate the amount of data lost, for example, when an SSD reaches its lifetime write capacity.

pgaddict 8y ago

Isn't one of the difficulties the inability to decide when the I/O error is transient (e.g. running out of space with thin provisioning) vs. permanent (e.g. drive eaten by a velociraptor)? Also, isn't it possible to use multipath to queue writes in case of error? I wonder if that's safe, though, because it will keep it in memory only and make it look OK to the caller.

anarazel 8y ago
> Isn't one of the difficulties the inability to decide when the I/O error is transient (e.g. running out of space with thin provisioning) vs. permanent (e.g. drive eaten by a velociraptor)? I don't really think so. The thin provisioning thing really is just a series of bugs in the kernel, because the thin provisioning code just didn't (and maybe doesn't), bubble up errors with enough information. Even if errors are transient, at some point the decision to shut down the FS on a device with enough transient errors is a much better fix than throwing away data.