7 ms·

Would memory safe languages avoid these kinds of problems? It seems like a good example of a nightmare bug from memory corruption - 5 days to fix and the author

Would memory safe languages avoid these kinds of problems? It seems like a good example of a nightmare bug from memory corruption - 5 days to fix and the author alludes to it keeping them up at night is a pretty strong motivation to avoid memory unsafety IMO.

saagarjha 2y ago
No*. This is one of the bugs that traditional memory safety would not fix, because the issue crosses privilege boundaries in a way that the language can't protect against. *This could, in theory, be caught by fancy hardware strategies like capabilities. But those are somewhat more esoteric.
- quotemstr 2y ago
  Safe code definitely won't have this sort of problem. Any code that could invoke a system call to scribble on arbitrary memory is by definition unsafe.
  - saagarjha 2y ago
    That's basically all code
    - quotemstr 2y ago
      No it isn't. You can write safe file IO in Rust despite the read and write system calls being unsafe.
      - saagarjha 2y ago
        I take it you are not familiar with the classic Rust meme of opening /proc/self/mem and using it to completely wreck your program?
        IshKebab 2y ago
        That's obviously outside the scope of the language's safety model, and it would be quite hard to do that accidentally.
        saagarjha 2y ago
        That is exactly my point, though: system calls are completely outside the scope of a language's safety model. You can say, well /proc/self/mem is stupid (it is) and our file wrappers for read and write are safe (…most languages have at least one), but the fundamental problem remains that you can't just expect to make system calls without that being implicitly unsafe. In the extreme the syscall itself cannot be done safely, with no possible safe wrapper around it. My point is that if you are calling these Windows APIs you can't do it safely from any language; Rust won't magically start yelling at you that the kernel still expects you to keep the buffer alive. You can design your own wrapper around it and try to match the kernel's requirements but you can do that in a lot of languages, and that's kind of missing the point.
        loeg 2y ago
        Right. And of course, it's not just Windows. For example the Linux syscall aio_read() similarly registers a user address with the kernel for later, asynchronous writing (by the kernel). (And I'm sure you get similar lifetime issues with io_uring operations.)
        ryao 2y ago
        While I am not aware of a Linux syscall that would be equivalent to QueueUserAPC() to allow this to happen, the kernel writing to stack memory is not the problem here. The problem is that a C++ exception was invoked and it unwound a C stack frame. C++ exceptions that unwind C stack frames invoke undefined behavior, so the real solution is to avoid passing function pointers to C++ functions not marked noexcept to C functions as callbacks. It is rather unusual that Windows permits execution on the thread while the kernel is supposed to give it a return value. Writing to the stack is not how I would expect a return value to be passed. Presumably, had the stack frame not been unwound, things would have been fine, unless there is a horrific bug in Windows that should have been obvious when QueueUserAPC() was first implemented. Anyway, it is a shame that the compiler does not issue a warning when you do this. I filed bug reports with both GCC and LLVM requesting that they issue warnings, which should be able to avoid this mess if the compilers issue them and developers heed them: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://github.com/llvm/llvm-project/issues/121427 https://github.com/llvm/llvm-project/issues/121427
      - deleted 2y ago
        [deleted]
- kibwen 2y ago
  To elaborate, the problem here is that it looks like the OS API itself is fundamentally unsafe: it's taking a pointer to a memory location and then blindly writing into it, expecting that it's still valid without actually doing any sort of verification. You could imagine an OS providing a safe API instead (with possible performance implications depending on the exact approach used), and if your OS API was written in e.g. Rust then this unsafe version of the API would be marked as `unsafe` with the documented invariant "the caller must ensure that the pointer remains valid".
  - jpc0 2y ago
    Seeing as rust has no stable ABI and likely never will. How would you provide the API in rust, also in golang, also in .NET, and swift, and Java, and whatever other language you add without doing exactly what Win32 does and go to C which has a stable ABI to tie into all those other languages?
    - pornel 2y ago
      Rust ecosystem solves that by providing packages that are thin wrappers around underlying APIs. It's very similar to providing an .h file with extra type information, except it's an .rs file. Correctness of the Rust wrapper can't be checked by the compiler, just like correctness of C headers is unchecked, and it just has to match the actual underlying ABI. The task of making a safe API wrapper can be relatively simple, because you don't have to take into consideration safety an application as a whole, you only need to translate requirements of individual APIs to Rust's safety requirements, function by function. In this case you would need to be aware that the function call may unwind, so whether someone making a dedicated safe API for it would think of it or not, is only a speculation.
      - jpc0 2y ago
        I seem to remember a linux kernel dev quiting and not being able to specify exactly what you say this wrapper should abide by as being a contributing factor. If those specifications were written down clearly enough then this dev wouldn't have needed to spend 5 days debugging this since he spent a significant amount of time reading the documentation to find any errors they are making that is mentioned in the documentation. And don't say that they can actually just read the rust code and check that since well, I can't read low level rust code and how any of the annotations ca interact with each other. A single line of rust code could easily need several paragraphs of written documentation so that someone not familier with what rust is specifying will actually understand what that entails. This is part of why Rust is difficult, you have to nail down the specification and a small change to the specification causes broad changes to the codebase. The same might need to happen in C, but many times it doesn't.
bru3s 2y ago
[flagged]
IshKebab 2y ago
Yes memory safe languages would absolutely help here. In Rust you would get a compile time error about the destination variable not living long enough. This sort of stuff is why any productivity arguments for C++ over Rust are bullshit. Sure you spend a little more time writing lifetime annotations, but in return you avoid spending 5 days debugging one memory corruption bug.
- ryao 2y ago
  This is not a memory corruption bug. It is an undefined behavior bug and it also affected Rust until 1.81.0 as per comments from others: https://blog.rust-lang.org/2024/09/05/Rust-1.81.0.html#abort-on-uncaught-panics-in-extern-c-functions https://blog.rust-lang.org/2024/09/05/Rust-1.81.0.html#abort...
layer8 2y ago
Depends. The underlying issue for this bug is that the code involved crosses language boundaries (the Windows kernel and win32 libraries written in C and the application in C++). The code where the lifetime failure occurs is Windows code, not application code. However, the Windows code is correct in the context of the C language. The error is caused by an APC that calls exception-throwing C++ code, being pushed onto the waiting-in-C thread. This is a case of language-agnostic OS mechanisms conflicting with language-specific stack unwinding mechanisms. This could only be made safe by the OS somehow imposing safety mechanisms on the binary level, or by wrapping all OS APIs into APIs of the safe language, where the wrappers have to take care to ensure both the guarantees implied by the language and the assumptions made by the OS APIs. (Writing the OS itself in a memory-safe language isn’t sufficient, for one because it very likely will still require some amount of “unsafe” code, and furthermore because one would still want to allow applications written in a different language, which even if it also is memory-safe, would need memory-correct wrappers/adapters.) This is similar to the distinction between memory-safe languages like Rust where the safety is established primarily on the source level, not on the binary level, and memory-safe runtimes like the CLR (.NET) and the JVM.
- jpc0 2y ago
  > the Windows kernel and win32 libraries written in C and the application in C++ To my knowledge the kernel and win32 is in fact written in C++ and only the interface has C linkage and follows C norms. So this error occurred going C++ > C > C++ never mind languages with different memory protection mechanisms like Rust > C > C++.
  - rurban 2y ago
    No, the windows kernel is written in pure C.
    - cryptonector 2y ago
      I believe it's C++, but not allowed to use exceptions.
muststopmyths 2y ago
In this specific type of Win32 API case, I can think of a way to make this safe. It would involve looking at the function pointer in QueueUserAPC and making sure the function being called doesn't mess with the stack frame being executed on. This function will run in the context of the called thread, in that thread's stack. NOT in the calling thread. It's a weird execution mode where you're allowed to hijack a blocked thread and run some code in its context. Don't know enough about Rust or the like to say if that's something that could be done in the language with attributes/annotations for a function, but it seems plausible.
- LegionMammal978 2y ago
  Nothing in C can prevent your function from being abnormally unwound through (whether it's via C++ exceptions or via C longjmp()). The only real fix is "don't use C++ exceptions unless you're 100% sure that the code in between is exception-safe (and don't use C longjmp() at all outside of controlled scenarios)".
  - ryao 2y ago
    A better fix is to avoid passing pointers to C++ functions that can throw exceptions to C functions. This theoretically can be enforced by the compiler by requiring the C++ function pointers be marked noexcept. I filed bugs against both GCC and LLVM requesting warnings: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://github.com/llvm/llvm-project/issues/121427 https://github.com/llvm/llvm-project/issues/121427
- loeg 2y ago
  Perhaps simpler would be to just not unwind C++ exceptions through non-C++ stack frames and abort instead. (You'd run into these crashes at development time, debugging them would be pretty obvious, and it'd never release like this.) This might not be viable on Windows, though, where there is a lot of both C++ and legacy C code.
  - charrondev 2y ago
rramadass 2y ago
No. The problem was in the architecture of the asynchronous api w.r.t. the kernel. The last line of the article states; Lesson learnt, folks: do not throw exceptions out of asynchronous procedures if you’re inside a system call!
- LorenPechtel 2y ago
  More generally: 1) The top level of an async routine should have a handler that catches all exceptions and dies if it catches one. 2) If you have a resource you have a cleanup routine for it.
  - rramadass 2y ago
    It is even more fundamental. People are focusing wrongly on the mention of exceptions here (most obvious) but what is crucial is to understand how Async callbacks registered with a Kernel work on all OSes. The limitations/caveats imposed on these routines (they are akin to interrupts) are given in their respective documentations and one has to be careful to understand and use them appropriately; eg. what is the stack used by these handlers? The article though detailed in the beginning sort of glosses over all this in the final paragraphs and hence we have to link the dots ourselves.
    - LegionMammal978 2y ago
      It's not really about asynchronous callbacks or their equivalents. (In this case, the thread running it is otherwise meant to be blocked in a safe state, so that there's none of the usual dangers of interrupting arbitrary code.) Instead, it's about any callbacks coming out of C code, even something as trivial as qsort(). If you pass a C library your C++ callback, and your callback runs back through it with an exception, then 9 times out of 10, the C library will leak some resources at best, or reach an unstable state at worst. C just doesn't have any portable 'try/finally' construct that can help deal with it. So I'd say it's more about the basic expectations of a function called from C, which includes a million other trivial things like "don't write beyond the bounds of buffers you're given" and "don't clobber your caller's stack frame" and "don't spawn another thread just to write to output pointers after your function returns" (not that any of these is the issue here).

saagarjha 2y ago

No*. This is one of the bugs that traditional memory safety would not fix, because the issue crosses privilege boundaries in a way that the language can't protect against. *This could, in theory, be caught by fancy hardware strategies like capabilities. But those are somewhat more esoteric.

quotemstr 2y ago
Safe code definitely won't have this sort of problem. Any code that could invoke a system call to scribble on arbitrary memory is by definition unsafe.
- saagarjha 2y ago
  That's basically all code
  - quotemstr 2y ago
    No it isn't. You can write safe file IO in Rust despite the read and write system calls being unsafe.
    - saagarjha 2y ago
      I take it you are not familiar with the classic Rust meme of opening /proc/self/mem and using it to completely wreck your program?
      - IshKebab 2y ago
        That's obviously outside the scope of the language's safety model, and it would be quite hard to do that accidentally.
        saagarjha 2y ago
        That is exactly my point, though: system calls are completely outside the scope of a language's safety model. You can say, well /proc/self/mem is stupid (it is) and our file wrappers for read and write are safe (…most languages have at least one), but the fundamental problem remains that you can't just expect to make system calls without that being implicitly unsafe. In the extreme the syscall itself cannot be done safely, with no possible safe wrapper around it. My point is that if you are calling these Windows APIs you can't do it safely from any language; Rust won't magically start yelling at you that the kernel still expects you to keep the buffer alive. You can design your own wrapper around it and try to match the kernel's requirements but you can do that in a lot of languages, and that's kind of missing the point.
        loeg 2y ago
        Right. And of course, it's not just Windows. For example the Linux syscall aio_read() similarly registers a user address with the kernel for later, asynchronous writing (by the kernel). (And I'm sure you get similar lifetime issues with io_uring operations.)
        ryao 2y ago
        While I am not aware of a Linux syscall that would be equivalent to QueueUserAPC() to allow this to happen, the kernel writing to stack memory is not the problem here. The problem is that a C++ exception was invoked and it unwound a C stack frame. C++ exceptions that unwind C stack frames invoke undefined behavior, so the real solution is to avoid passing function pointers to C++ functions not marked noexcept to C functions as callbacks. It is rather unusual that Windows permits execution on the thread while the kernel is supposed to give it a return value. Writing to the stack is not how I would expect a return value to be passed. Presumably, had the stack frame not been unwound, things would have been fine, unless there is a horrific bug in Windows that should have been obvious when QueueUserAPC() was first implemented. Anyway, it is a shame that the compiler does not issue a warning when you do this. I filed bug reports with both GCC and LLVM requesting that they issue warnings, which should be able to avoid this mess if the compilers issue them and developers heed them: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://github.com/llvm/llvm-project/issues/121427 https://github.com/llvm/llvm-project/issues/121427
    - deleted 2y ago
      [deleted]
kibwen 2y ago
To elaborate, the problem here is that it looks like the OS API itself is fundamentally unsafe: it's taking a pointer to a memory location and then blindly writing into it, expecting that it's still valid without actually doing any sort of verification. You could imagine an OS providing a safe API instead (with possible performance implications depending on the exact approach used), and if your OS API was written in e.g. Rust then this unsafe version of the API would be marked as `unsafe` with the documented invariant "the caller must ensure that the pointer remains valid".
- jpc0 2y ago
  Seeing as rust has no stable ABI and likely never will. How would you provide the API in rust, also in golang, also in .NET, and swift, and Java, and whatever other language you add without doing exactly what Win32 does and go to C which has a stable ABI to tie into all those other languages?
  - pornel 2y ago
    Rust ecosystem solves that by providing packages that are thin wrappers around underlying APIs. It's very similar to providing an .h file with extra type information, except it's an .rs file. Correctness of the Rust wrapper can't be checked by the compiler, just like correctness of C headers is unchecked, and it just has to match the actual underlying ABI. The task of making a safe API wrapper can be relatively simple, because you don't have to take into consideration safety an application as a whole, you only need to translate requirements of individual APIs to Rust's safety requirements, function by function. In this case you would need to be aware that the function call may unwind, so whether someone making a dedicated safe API for it would think of it or not, is only a speculation.
    - jpc0 2y ago
      I seem to remember a linux kernel dev quiting and not being able to specify exactly what you say this wrapper should abide by as being a contributing factor. If those specifications were written down clearly enough then this dev wouldn't have needed to spend 5 days debugging this since he spent a significant amount of time reading the documentation to find any errors they are making that is mentioned in the documentation. And don't say that they can actually just read the rust code and check that since well, I can't read low level rust code and how any of the annotations ca interact with each other. A single line of rust code could easily need several paragraphs of written documentation so that someone not familier with what rust is specifying will actually understand what that entails. This is part of why Rust is difficult, you have to nail down the specification and a small change to the specification causes broad changes to the codebase. The same might need to happen in C, but many times it doesn't.

bru3s 2y ago

[flagged]

IshKebab 2y ago

Yes memory safe languages would absolutely help here. In Rust you would get a compile time error about the destination variable not living long enough. This sort of stuff is why any productivity arguments for C++ over Rust are bullshit. Sure you spend a little more time writing lifetime annotations, but in return you avoid spending 5 days debugging one memory corruption bug.

ryao 2y ago
This is not a memory corruption bug. It is an undefined behavior bug and it also affected Rust until 1.81.0 as per comments from others: https://blog.rust-lang.org/2024/09/05/Rust-1.81.0.html#abort-on-uncaught-panics-in-extern-c-functions https://blog.rust-lang.org/2024/09/05/Rust-1.81.0.html#abort...

layer8 2y ago

Depends. The underlying issue for this bug is that the code involved crosses language boundaries (the Windows kernel and win32 libraries written in C and the application in C++). The code where the lifetime failure occurs is Windows code, not application code. However, the Windows code is correct in the context of the C language. The error is caused by an APC that calls exception-throwing C++ code, being pushed onto the waiting-in-C thread. This is a case of language-agnostic OS mechanisms conflicting with language-specific stack unwinding mechanisms. This could only be made safe by the OS somehow imposing safety mechanisms on the binary level, or by wrapping all OS APIs into APIs of the safe language, where the wrappers have to take care to ensure both the guarantees implied by the language and the assumptions made by the OS APIs. (Writing the OS itself in a memory-safe language isn’t sufficient, for one because it very likely will still require some amount of “unsafe” code, and furthermore because one would still want to allow applications written in a different language, which even if it also is memory-safe, would need memory-correct wrappers/adapters.) This is similar to the distinction between memory-safe languages like Rust where the safety is established primarily on the source level, not on the binary level, and memory-safe runtimes like the CLR (.NET) and the JVM.

jpc0 2y ago
> the Windows kernel and win32 libraries written in C and the application in C++ To my knowledge the kernel and win32 is in fact written in C++ and only the interface has C linkage and follows C norms. So this error occurred going C++ > C > C++ never mind languages with different memory protection mechanisms like Rust > C > C++.
- rurban 2y ago
  No, the windows kernel is written in pure C.
  - cryptonector 2y ago
    I believe it's C++, but not allowed to use exceptions.

muststopmyths 2y ago

In this specific type of Win32 API case, I can think of a way to make this safe. It would involve looking at the function pointer in QueueUserAPC and making sure the function being called doesn't mess with the stack frame being executed on. This function will run in the context of the called thread, in that thread's stack. NOT in the calling thread. It's a weird execution mode where you're allowed to hijack a blocked thread and run some code in its context. Don't know enough about Rust or the like to say if that's something that could be done in the language with attributes/annotations for a function, but it seems plausible.

LegionMammal978 2y ago
Nothing in C can prevent your function from being abnormally unwound through (whether it's via C++ exceptions or via C longjmp()). The only real fix is "don't use C++ exceptions unless you're 100% sure that the code in between is exception-safe (and don't use C longjmp() at all outside of controlled scenarios)".
- ryao 2y ago
  A better fix is to avoid passing pointers to C++ functions that can throw exceptions to C functions. This theoretically can be enforced by the compiler by requiring the C++ function pointers be marked noexcept. I filed bugs against both GCC and LLVM requesting warnings: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118263 https://github.com/llvm/llvm-project/issues/121427 https://github.com/llvm/llvm-project/issues/121427
loeg 2y ago
Perhaps simpler would be to just not unwind C++ exceptions through non-C++ stack frames and abort instead. (You'd run into these crashes at development time, debugging them would be pretty obvious, and it'd never release like this.) This might not be viable on Windows, though, where there is a lot of both C++ and legacy C code.
- charrondev 2y ago

rramadass 2y ago

No. The problem was in the architecture of the asynchronous api w.r.t. the kernel. The last line of the article states; Lesson learnt, folks: do not throw exceptions out of asynchronous procedures if you’re inside a system call!

LorenPechtel 2y ago
More generally: 1) The top level of an async routine should have a handler that catches all exceptions and dies if it catches one. 2) If you have a resource you have a cleanup routine for it.
- rramadass 2y ago
  It is even more fundamental. People are focusing wrongly on the mention of exceptions here (most obvious) but what is crucial is to understand how Async callbacks registered with a Kernel work on all OSes. The limitations/caveats imposed on these routines (they are akin to interrupts) are given in their respective documentations and one has to be careful to understand and use them appropriately; eg. what is the stack used by these handlers? The article though detailed in the beginning sort of glosses over all this in the final paragraphs and hence we have to link the dots ourselves.
  - LegionMammal978 2y ago
    It's not really about asynchronous callbacks or their equivalents. (In this case, the thread running it is otherwise meant to be blocked in a safe state, so that there's none of the usual dangers of interrupting arbitrary code.) Instead, it's about any callbacks coming out of C code, even something as trivial as qsort(). If you pass a C library your C++ callback, and your callback runs back through it with an exception, then 9 times out of 10, the C library will leak some resources at best, or reach an unstable state at worst. C just doesn't have any portable 'try/finally' construct that can help deal with it. So I'd say it's more about the basic expectations of a function called from C, which includes a million other trivial things like "don't write beyond the bounds of buffers you're given" and "don't clobber your caller's stack frame" and "don't spawn another thread just to write to output pointers after your function returns" (not that any of these is the issue here).