7 ms·

It's pretty easy at first order: features that users want are good ideas. I agree at second order users want a lot of features that solve their problems in ways

It's pretty easy at first order: features that users want are good ideas. I agree at second order users want a lot of features that solve their problems in ways that are less clever than one might like, but in the vast majority of cases the right answer from development is, "Users know what they need better than I do, so if they ask for something, I'll do my best to implement it even if I don't totally understand why." That answer exposes the deeper answer, too: "My job as a developer is to understand user needs so that our software can help them fulfill them, so if I don't understand why someone needs a feature I should dig in further before implementing it so I don't implement the wrong thing or the right thing in the wrong way." Not always possible given shedule constraints, though. Admitedly, those rules don't seem to apply in this case, since if the OS allows you to do something that corrupts data, that's a problem no matter why the user wants to do it. If it can corrupt data, the OS shouldn't allow you to do it, end of story.

nathanb 11y ago
It's not that simple (speaking as someone who works on OS SCSI driver code for a living...). I will amend your statement to say that the OS itself should never corrupt your data, which I agree with entirely. The post was not super clear about exactly what was happening (or it may be that my limited knowledge of Windows storage internals is keeping me from understanding it), but it sounds like the NTFS client requested a cache flush and then was issuing writes during the flush. I don't know what contract these operations have, but it may very well be the case that the user was violating the contract. If Microsoft responded with "don't do that", this may be the case. But wait! Shouldn't Windows prevent the data from being corrupted? Or shouldn't NTFS fail the writes in this case? Possibly. And most likely inserting the checks to make this happen would increase write latency for every NTFS client, even the ones that don't behave in this way. This reminds me of another scenario I encountered, with Veritas VxFS running on top of AIX. The user initiated a space reclamation, which was sending what you can think of as a delete to the storage array. And the user was also writing data to the device at the same time. Due to a race condition (which I can describe for you if you really care), the legitimate user data would sometimes be deleted. Should VxFS have protected users against this case? Yes. Was VxFS violating the SCSI protocol? No. Was the storage array violating the SCSI protocol? No. (Has VxFS fixed this bug, almost three years after I discovered it? No comment.) It's always a lot more complicated than it seems on the outside.
- chetanahuja 11y ago
  Sorry I don't buy that. An operating system's "contract" with the user is the syscall API. There's no room for argument there. Calling a write while flushing the same data from another process (as the OP reported) or thread is perfectly legitimate set of operations. If these operations are not supposed to be running simultaneously, that has to be enforced by the OS kernel. The "right" way would be for the OS to serialize these operations internally. But even returning an error for the write might be an acceptable (though not nice) way to handle it. What's absolutely not acceptable is randomly corrupting data.
  - deleted 11y ago
    [deleted]
  - nathanb 11y ago
    I don't know anything about Windows driver programming so I don't know what the contract of the buffer flush operation is. Obviously, "sometimes fills your buffer with random data from memory pages owned by the operating system" is not part of that contract. This is a bug. I'm not trying to argue that it isn't. All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. What if you have an API that takes an out pointer. I pass in data that I own, then I free that pointer in another thread. If I'm in userspace, I can blow up with SIGSEGV. If I'm in the kernel, maybe now you've just scribbled all over somebody else's memory. Shame on you for corrupting data. APIs can always be abused. All the API developer can do is try to protect against obvious forms of abuse. I guarantee you every operating system that supports simultaneous multiprocess execution has some series of APIs that, when called in parallel, will corrupt your data.
    - Dylan16807 11y ago
      >All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. Only for a wide interpretation of 'misuse'. In the case you described, the kernel is still doing what it was asked. In the article, it wasn't. Situations where one command invalidates another should be very uncommon and very documented. A real, sane OS should be resilient to any syscalls in any order without triggering internal bugs.

nathanb 11y ago

It's not that simple (speaking as someone who works on OS SCSI driver code for a living...). I will amend your statement to say that the OS itself should never corrupt your data, which I agree with entirely. The post was not super clear about exactly what was happening (or it may be that my limited knowledge of Windows storage internals is keeping me from understanding it), but it sounds like the NTFS client requested a cache flush and then was issuing writes during the flush. I don't know what contract these operations have, but it may very well be the case that the user was violating the contract. If Microsoft responded with "don't do that", this may be the case. But wait! Shouldn't Windows prevent the data from being corrupted? Or shouldn't NTFS fail the writes in this case? Possibly. And most likely inserting the checks to make this happen would increase write latency for every NTFS client, even the ones that don't behave in this way. This reminds me of another scenario I encountered, with Veritas VxFS running on top of AIX. The user initiated a space reclamation, which was sending what you can think of as a delete to the storage array. And the user was also writing data to the device at the same time. Due to a race condition (which I can describe for you if you really care), the legitimate user data would sometimes be deleted. Should VxFS have protected users against this case? Yes. Was VxFS violating the SCSI protocol? No. Was the storage array violating the SCSI protocol? No. (Has VxFS fixed this bug, almost three years after I discovered it? No comment.) It's always a lot more complicated than it seems on the outside.

chetanahuja 11y ago
Sorry I don't buy that. An operating system's "contract" with the user is the syscall API. There's no room for argument there. Calling a write while flushing the same data from another process (as the OP reported) or thread is perfectly legitimate set of operations. If these operations are not supposed to be running simultaneously, that has to be enforced by the OS kernel. The "right" way would be for the OS to serialize these operations internally. But even returning an error for the write might be an acceptable (though not nice) way to handle it. What's absolutely not acceptable is randomly corrupting data.
- deleted 11y ago
  [deleted]
- nathanb 11y ago
  I don't know anything about Windows driver programming so I don't know what the contract of the buffer flush operation is. Obviously, "sometimes fills your buffer with random data from memory pages owned by the operating system" is not part of that contract. This is a bug. I'm not trying to argue that it isn't. All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. What if you have an API that takes an out pointer. I pass in data that I own, then I free that pointer in another thread. If I'm in userspace, I can blow up with SIGSEGV. If I'm in the kernel, maybe now you've just scribbled all over somebody else's memory. Shame on you for corrupting data. APIs can always be abused. All the API developer can do is try to protect against obvious forms of abuse. I guarantee you every operating system that supports simultaneous multiprocess execution has some series of APIs that, when called in parallel, will corrupt your data.
  - Dylan16807 11y ago
    >All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. Only for a wide interpretation of 'misuse'. In the case you described, the kernel is still doing what it was asked. In the article, it wasn't. Situations where one command invalidates another should be very uncommon and very documented. A real, sane OS should be resilient to any syscalls in any order without triggering internal bugs.