5 ms·

I come from Linux land so some of the details flew over my head. With that said, is there not a valgrind equivalent for Windows? The sentinel thing seems to a d

I come from Linux land so some of the details flew over my head. With that said, is there not a valgrind equivalent for Windows? The sentinel thing seems to a default functionality to the valgrind virtual machine. I would expect that the initial part of the investigation would be automatically done by valgrind. Of course reading the whole post it looks like it would not get very far. The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? Even if that is possible that sounds like a dangerous game to play, and this class of bugs seems natural. I can imagine the scenario where a thread is cancelled mid system call, but it will be the kernel that will cancel the thread and responsible to know that half way syscalls need to be cleaned up. This way the application does not get system call call leaks? Also if i remember one debugging session correctly, when you have a blocking syscall interrupted you will get a sigabort on linux.

apocalypses 5y ago
There’s address sanitizer on newer versions of visual studio now but in my experience getting it to actually work with all of your projects’ dependencies can be very hit and miss. Windows Debug build config also does a lot more memory checking (part of the reason it’s so slow) so you’re not totally screwed for automated tools.
mike_hock 5y ago
> The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? Even if that is possible that sounds like a dangerous game to play, and this class of bugs seems natural. I mean, signal handlers exist on Linux, too ... > when you have a blocking syscall interrupted you will get a sigabort on linux Doesn't the syscall just return EINTR?
- nyc_pizzadev 5y ago
  The select() was not interrupted in a traditional sense. It looks like a special windows API was used (QueueUserAPC) which allows a callback to be called in user space while blocked in certain kernel calls. My guess is that when used normally, the callback returns and control goes back to select() being blocked, but in this case an exception was thrown which unrolled the stack and left the select() call holding onto an invalid stack address. Jumping out of a syscall using an exception like this will probably cause all kinds of problems and my guess this is completely unsupported. https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-queueuserapc#remarks https://docs.microsoft.com/en-us/windows/win32/api/processth...
  - mike_hock 5y ago
    Throwing from a signal handler on Linux is also UB. Or rather, it will blow up on Linux too, and that's why it's UB in the standard, because it'll blow up on any platform.
    - ptsneves 5y ago
      With the understanding that somebody had complex logic in the equivalent of a signal handler in linux, then i am not sure this bug is so fantastic after all. I have signal handlers so high on my danger zone that i do not recall ever coding any. If I am in a sigabrt or sigsegv i hope i die as cleanly as possible, so that a core dump is useful. No wonder he needed kernel level development.
      - mike_hock 5y ago
        No, absolutely not. It just illustrates the problems that obviously insane code can lead to and why we call it "insane" in the first place. But it was still a fantastic read following him on his journey from "huh, that's odd" to "I had to use kernel-level debugging to find out they were ~~throwing from an APC~~ using APC." Imagine how much he could have achieved in those five days if he had just unfucked obvious bug magnets proactively without first painstakingly proving that they were really manifesting as bugs, and how.
CodesInChaos 5y ago
The syscall writing the data is async, which requires the memory it uses to be available until until it completes. Since this was on the stack here, it means that the function that contains this memory on its stack must not return until IO completes But the code that was supposed to wait until completion was interrupted by an exception, which unrolled the stack beyond the function owning the relevant memory. So the completing async operation writes into memory owned by somebody else now. This is a general complication with completion based async IO. You could get the same problems without any interrupts, if you just don't wait for async IO to complete in some circumstances.
- CamperBob2 5y ago
  More like a general complication with SEH. One of several reasons why I don't allow it in any code I touch. Anything that obscures the flow of control is a violation of literate programming principles, and exceptions are pretty high on that particular shit list.
- _nalply 5y ago
  Thank you. Now I understand why Rust async is so complicated. It's about lifetimes. It's dangerous to write data asynchronously because... eh... that location might be already invalid. I know this is not about Rust. Forgive me that I mentally connect this story with async Rust.
  - CodesInChaos 5y ago
    There are two ways how async IO can work: * Readiness based: You use `select` (or preferably its successors) to figure out which file descriptor has data available. Then you synchronously read the data, which completes instantly because there is data available. This is the traditional way on Linux, and also the way Rust's async works. This approach doesn't run into any lifetime issues, because you only need to provide the memory (`&mut [u8]`) to a sync call once data is available. This approach works well from sequential streams, like sockets or pipes, but isn't a natural match for random access file IO. * Completion based: You trigger an IO operation specifying the target memory. The system notifies you when it completes. This approach needs you to keep the memory available until the operation finishes. This is the traditional way on Windows (IO completion ports) and also used by the more recent io_uring on Linux. This one runs into the lifetime issues that caused the bug in the linked article. Thus in Rust it can't safely be used with a `&mut [u8]`, you have to transfer ownership of the buffer to the IO library, so it can keep it alive until the operation completes. The ringbahn crate is an example of this approach in Rust.
fxtentacle 5y ago
There's IBM Rational PurifyPlus, which is kinda the same as valgrind, but with a Visual Studio plugin and fancy GUI. But when we bought it, we only got one license for the entire company because it cost around $10k per seat.
- dzdt 5y ago
  PurifyPlus was an old enterprise software similar to valgrind for Sun/Solaris. It eventually got Linux and Windows support, but for more than a decade has been one of those undeveloped systems milking legacy enterprise subscriptions as it slowly bitrots into uselessness. It was bought by IBM and then passed on to Unicom. These days it is not a reasonable choice for anyone not a legacy user.
  - randomswede 5y ago
    Not sure if PurifyPlus existed for HP-UX, but I know Purify did (at a very previous job, we even managed to find a small memory leak in the standard libc, which we filed with HP; unfortunately, a repro required having a Purify license).
anaisbetts 5y ago
> With that said, is there not a valgrind equivalent for Windows? Yep, it's called Application Verifier and it will check all kinds of system calls and make the memory allocator incredibly aggro towards checking validity > The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? When you call select(), your thread is in a special state called Alertable Wait; one way to get out of it is to have an APC queued to your thread, which has a similar vibe to a UNIX signal (with all of the similar caveats about how easy it is to fuck things up in it). Basically, OP's scenario is similar to corruption caused by a badly written signal handler

apocalypses 5y ago

There’s address sanitizer on newer versions of visual studio now but in my experience getting it to actually work with all of your projects’ dependencies can be very hit and miss. Windows Debug build config also does a lot more memory checking (part of the reason it’s so slow) so you’re not totally screwed for automated tools.

mike_hock 5y ago

> The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? Even if that is possible that sounds like a dangerous game to play, and this class of bugs seems natural. I mean, signal handlers exist on Linux, too ... > when you have a blocking syscall interrupted you will get a sigabort on linux Doesn't the syscall just return EINTR?

nyc_pizzadev 5y ago
The select() was not interrupted in a traditional sense. It looks like a special windows API was used (QueueUserAPC) which allows a callback to be called in user space while blocked in certain kernel calls. My guess is that when used normally, the callback returns and control goes back to select() being blocked, but in this case an exception was thrown which unrolled the stack and left the select() call holding onto an invalid stack address. Jumping out of a syscall using an exception like this will probably cause all kinds of problems and my guess this is completely unsupported. https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-queueuserapc#remarks https://docs.microsoft.com/en-us/windows/win32/api/processth...
- mike_hock 5y ago
  Throwing from a signal handler on Linux is also UB. Or rather, it will blow up on Linux too, and that's why it's UB in the standard, because it'll blow up on any platform.
  - ptsneves 5y ago
    With the understanding that somebody had complex logic in the equivalent of a signal handler in linux, then i am not sure this bug is so fantastic after all. I have signal handlers so high on my danger zone that i do not recall ever coding any. If I am in a sigabrt or sigsegv i hope i die as cleanly as possible, so that a core dump is useful. No wonder he needed kernel level development.
    - mike_hock 5y ago
      No, absolutely not. It just illustrates the problems that obviously insane code can lead to and why we call it "insane" in the first place. But it was still a fantastic read following him on his journey from "huh, that's odd" to "I had to use kernel-level debugging to find out they were ~~throwing from an APC~~ using APC." Imagine how much he could have achieved in those five days if he had just unfucked obvious bug magnets proactively without first painstakingly proving that they were really manifesting as bugs, and how.

CodesInChaos 5y ago

The syscall writing the data is async, which requires the memory it uses to be available until until it completes. Since this was on the stack here, it means that the function that contains this memory on its stack must not return until IO completes But the code that was supposed to wait until completion was interrupted by an exception, which unrolled the stack beyond the function owning the relevant memory. So the completing async operation writes into memory owned by somebody else now. This is a general complication with completion based async IO. You could get the same problems without any interrupts, if you just don't wait for async IO to complete in some circumstances.

CamperBob2 5y ago
More like a general complication with SEH. One of several reasons why I don't allow it in any code I touch. Anything that obscures the flow of control is a violation of literate programming principles, and exceptions are pretty high on that particular shit list.
_nalply 5y ago
Thank you. Now I understand why Rust async is so complicated. It's about lifetimes. It's dangerous to write data asynchronously because... eh... that location might be already invalid. I know this is not about Rust. Forgive me that I mentally connect this story with async Rust.
- CodesInChaos 5y ago
  There are two ways how async IO can work: * Readiness based: You use `select` (or preferably its successors) to figure out which file descriptor has data available. Then you synchronously read the data, which completes instantly because there is data available. This is the traditional way on Linux, and also the way Rust's async works. This approach doesn't run into any lifetime issues, because you only need to provide the memory (`&mut [u8]`) to a sync call once data is available. This approach works well from sequential streams, like sockets or pipes, but isn't a natural match for random access file IO. * Completion based: You trigger an IO operation specifying the target memory. The system notifies you when it completes. This approach needs you to keep the memory available until the operation finishes. This is the traditional way on Windows (IO completion ports) and also used by the more recent io_uring on Linux. This one runs into the lifetime issues that caused the bug in the linked article. Thus in Rust it can't safely be used with a `&mut [u8]`, you have to transfer ownership of the buffer to the IO library, so it can keep it alive until the operation completes. The ringbahn crate is an example of this approach in Rust.

fxtentacle 5y ago

There's IBM Rational PurifyPlus, which is kinda the same as valgrind, but with a Visual Studio plugin and fancy GUI. But when we bought it, we only got one license for the entire company because it cost around $10k per seat.

dzdt 5y ago
PurifyPlus was an old enterprise software similar to valgrind for Sun/Solaris. It eventually got Linux and Windows support, but for more than a decade has been one of those undeveloped systems milking legacy enterprise subscriptions as it slowly bitrots into uselessness. It was bought by IBM and then passed on to Unicom. These days it is not a reasonable choice for anyone not a legacy user.
- randomswede 5y ago
  Not sure if PurifyPlus existed for HP-UX, but I know Purify did (at a very previous job, we even managed to find a small memory leak in the standard libc, which we filed with HP; unfortunately, a repro required having a Purify license).

anaisbetts 5y ago

> With that said, is there not a valgrind equivalent for Windows? Yep, it's called Application Verifier and it will check all kinds of system calls and make the memory allocator incredibly aggro towards checking validity > The other thing is, how can a system call block a thread but somehow that same thread runs other stuff in between that blocking call? When you call select(), your thread is in a special state called Alertable Wait; one way to get out of it is to have an APC queued to your thread, which has a similar vibe to a UNIX signal (with all of the similar caveats about how easy it is to fuck things up in it). Basically, OP's scenario is similar to corruption caused by a badly written signal handler