5 ms·

It's not that simple (speaking as someone who works on OS SCSI driver code for a living...). I will amend your statement to say that the OS itself should never

by nathanb 11y ago

It's not that simple (speaking as someone who works on OS SCSI driver code for a living...). I will amend your statement to say that the OS itself should never corrupt your data, which I agree with entirely. The post was not super clear about exactly what was happening (or it may be that my limited knowledge of Windows storage internals is keeping me from understanding it), but it sounds like the NTFS client requested a cache flush and then was issuing writes during the flush. I don't know what contract these operations have, but it may very well be the case that the user was violating the contract. If Microsoft responded with "don't do that", this may be the case. But wait! Shouldn't Windows prevent the data from being corrupted? Or shouldn't NTFS fail the writes in this case? Possibly. And most likely inserting the checks to make this happen would increase write latency for every NTFS client, even the ones that don't behave in this way. This reminds me of another scenario I encountered, with Veritas VxFS running on top of AIX. The user initiated a space reclamation, which was sending what you can think of as a delete to the storage array. And the user was also writing data to the device at the same time. Due to a race condition (which I can describe for you if you really care), the legitimate user data would sometimes be deleted. Should VxFS have protected users against this case? Yes. Was VxFS violating the SCSI protocol? No. Was the storage array violating the SCSI protocol? No. (Has VxFS fixed this bug, almost three years after I discovered it? No comment.) It's always a lot more complicated than it seems on the outside.

chetanahuja 11y ago
Sorry I don't buy that. An operating system's "contract" with the user is the syscall API. There's no room for argument there. Calling a write while flushing the same data from another process (as the OP reported) or thread is perfectly legitimate set of operations. If these operations are not supposed to be running simultaneously, that has to be enforced by the OS kernel. The "right" way would be for the OS to serialize these operations internally. But even returning an error for the write might be an acceptable (though not nice) way to handle it. What's absolutely not acceptable is randomly corrupting data.
- deleted 11y ago
  [deleted]
- nathanb 11y ago
  I don't know anything about Windows driver programming so I don't know what the contract of the buffer flush operation is. Obviously, "sometimes fills your buffer with random data from memory pages owned by the operating system" is not part of that contract. This is a bug. I'm not trying to argue that it isn't. All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. What if you have an API that takes an out pointer. I pass in data that I own, then I free that pointer in another thread. If I'm in userspace, I can blow up with SIGSEGV. If I'm in the kernel, maybe now you've just scribbled all over somebody else's memory. Shame on you for corrupting data. APIs can always be abused. All the API developer can do is try to protect against obvious forms of abuse. I guarantee you every operating system that supports simultaneous multiprocess execution has some series of APIs that, when called in parallel, will corrupt your data.
  - Dylan16807 11y ago
    >All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. Only for a wide interpretation of 'misuse'. In the case you described, the kernel is still doing what it was asked. In the article, it wasn't. Situations where one command invalidates another should be very uncommon and very documented. A real, sane OS should be resilient to any syscalls in any order without triggering internal bugs.
    - nathanb 11y ago
      This may come as a shock to you, but operating systems have bugs. And nobody is trying to defend NTFS populating your file with random OS pages. So yes, an OS should allow you to call its API without triggering internal bugs. But that's kind of a straw man. There's an interesting line that you're exploring, though, and I would like to dig deeper. You said "the kernel is still doing what it was asked". Let's ignore the bug described by the article for a moment and look into this. Let's say I have two threads. Each one does a write to the same LBA range. You wind up with a file that doesn't represent the full contents of either write (say it's a 8192-byte write, and your file has 4096 bytes from the first write and 4096 bytes from the second write). Do you consider that to be a bug in the OS or a bug in the application client? (It's obvious that two conflicting writes will result in someone losing their data. The part of the scenario I'm exploring is that in this case everyone lost their data.)
fl0wenol 11y ago
The bug wasn't just corrupting the data in the file after the weird sequence of API calls. It was corrupting it with data from other files on the disk, which could have been sensitive. Where this was particularly troubling was that an unprivileged user could then use this as an exploit/attack on the system to get it to leak pages of system files. This is where fun stuff like pass-the-hash begins.
- deleted 11y ago
  [deleted]
- nathanb 11y ago
  My reading of the bug is that it would write random data from memory owned by the OS, not necessarily other files on disk. Certainly no better (and could very well be worse), but just a clarification. (Going back and reading the post again, it seems that it's even worse than that, since a virtual environment hitting this bug could get data from the host. This opens an attack vector for a guest to bypass the hypervisor and compromise sensitive host data. I don't know if data from other VMs on the system could also be exposed in this way, but that would also be quite bad.) Anyway, I think my rambling caused my point to get lost, because I wasn't trying to argue that Microsoft are justified in their "don't do that" comment. But no sane, realistic OS can prevent against every hare-brained thing a driver developer is going to do.

by nathanb 11y ago

chetanahuja 11y ago

Sorry I don't buy that. An operating system's "contract" with the user is the syscall API. There's no room for argument there. Calling a write while flushing the same data from another process (as the OP reported) or thread is perfectly legitimate set of operations. If these operations are not supposed to be running simultaneously, that has to be enforced by the OS kernel. The "right" way would be for the OS to serialize these operations internally. But even returning an error for the write might be an acceptable (though not nice) way to handle it. What's absolutely not acceptable is randomly corrupting data.

deleted 11y ago
[deleted]
nathanb 11y ago
I don't know anything about Windows driver programming so I don't know what the contract of the buffer flush operation is. Obviously, "sometimes fills your buffer with random data from memory pages owned by the operating system" is not part of that contract. This is a bug. I'm not trying to argue that it isn't. All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. What if you have an API that takes an out pointer. I pass in data that I own, then I free that pointer in another thread. If I'm in userspace, I can blow up with SIGSEGV. If I'm in the kernel, maybe now you've just scribbled all over somebody else's memory. Shame on you for corrupting data. APIs can always be abused. All the API developer can do is try to protect against obvious forms of abuse. I guarantee you every operating system that supports simultaneous multiprocess execution has some series of APIs that, when called in parallel, will corrupt your data.
- Dylan16807 11y ago
  >All I'm saying is that no real, sane OS is going to be capable of protecting itself against every possible misuse. Remember that these are developers coding to an API we're talking about, not end users. Only for a wide interpretation of 'misuse'. In the case you described, the kernel is still doing what it was asked. In the article, it wasn't. Situations where one command invalidates another should be very uncommon and very documented. A real, sane OS should be resilient to any syscalls in any order without triggering internal bugs.
  - nathanb 11y ago
    This may come as a shock to you, but operating systems have bugs. And nobody is trying to defend NTFS populating your file with random OS pages. So yes, an OS should allow you to call its API without triggering internal bugs. But that's kind of a straw man. There's an interesting line that you're exploring, though, and I would like to dig deeper. You said "the kernel is still doing what it was asked". Let's ignore the bug described by the article for a moment and look into this. Let's say I have two threads. Each one does a write to the same LBA range. You wind up with a file that doesn't represent the full contents of either write (say it's a 8192-byte write, and your file has 4096 bytes from the first write and 4096 bytes from the second write). Do you consider that to be a bug in the OS or a bug in the application client? (It's obvious that two conflicting writes will result in someone losing their data. The part of the scenario I'm exploring is that in this case everyone lost their data.)

fl0wenol 11y ago

The bug wasn't just corrupting the data in the file after the weird sequence of API calls. It was corrupting it with data from other files on the disk, which could have been sensitive. Where this was particularly troubling was that an unprivileged user could then use this as an exploit/attack on the system to get it to leak pages of system files. This is where fun stuff like pass-the-hash begins.

deleted 11y ago
[deleted]
nathanb 11y ago
My reading of the bug is that it would write random data from memory owned by the OS, not necessarily other files on disk. Certainly no better (and could very well be worse), but just a clarification. (Going back and reading the post again, it seems that it's even worse than that, since a virtual environment hitting this bug could get data from the host. This opens an attack vector for a guest to bypass the hypervisor and compromise sensitive host data. I don't know if data from other VMs on the system could also be exposed in this way, but that would also be quite bad.) Anyway, I think my rambling caused my point to get lost, because I wasn't trying to argue that Microsoft are justified in their "don't do that" comment. But no sane, realistic OS can prevent against every hare-brained thing a driver developer is going to do.