8 ms·

Segfaults are our friends and teachers

60 points by kalmar 10y ago

userbinator 10y ago
These are called guard pages. Attempts to write there would result in a segmentation fault. ...which are caught by the OS and used to either truly kill the process when the stack overflows, or to dynamically allocate more memory as the stack grows downwards. That's how it works on Windows, at least; I'm not as clear about Linux.
EliRivers 10y ago
The first sample code; "This program segfaults because the entire stack is set to 0 at program start." I'd be surprised; as a strong general rule, the stack does not get zeroed [Edit: see end of thread! It's the OS zeroing everything - learn something very day]. I'd expect it to segfault because the pointer value is whatever leftover non-zero value happens to be in that piece of memory, so it points into random memory the user program shouldn't be messing with (sticking in a printf to output the value of the pointer confirms this on at least one system). Wouldn't be surprised if some implementations took security really seriously and zero everything, or if a debug build was zero happy, but under normal circumstances, the stack doesn't get zeroed.
- dkopi 10y ago
  edit: exDM69 actually makes more sense.
  - EliRivers 10y ago
    Well, I can see the build command on the webpage there, and assuming this is the same gcc we all know and love without any cleverness going on, it's not a debug build of any kind as far as I can see: gcc segfault.c Now that I think about it, I'm not aware that gcc C compiler even offers the option to deliberately zero initialise non-static local variables, so unless I've missed a switch somewhere there is no arrangement of options available for a "debug build" to do this. I recall the GCC Fortran compiler did offer it.
    - exDM69 10y ago
      It's the OS linker/loader that zeros the pages used by a new process to avoid leaking memory contents of previous processes (security risk). It's zero only because this occurs at the very beginning of the program. In general it would be undefined.
      - EliRivers 10y ago
        Nice. Learn something every day. I edited my original post to add this; save anyone else tracing through the whole thread.
      - Buge 10y ago
        The OS will initialize the memory of the stack to 0 before the program starts. But before main is called, the compiler is free to insert other code that runs before main. gcc inserts a function named __libc_start_main that runs before main. This code will modify the content of the stack. So when main is run, the stack where the uninitialized local variable is has a decent chance of not being 0 anymore. This is easily testable. #include <stdio.h> int main(void) { char* pointers[20]; int i; for (i = 0; i < 20; ++i) { printf("%p\n", pointers[i]); } return 0; } And yes when I run it, most of the pointers are not null.
        exDM69 10y ago
        Sure, the C runtime initialization runs before main. Unless you're looking at the stack at _start, it's probably unintialized. And it depends on your libc implementation, etc. It just happened to be zero in the author's case.
        kalmar 10y ago
        Oh this is cool! I should have checked more pointers. On my machine, the single pointer was always null, and I read up on stack being initialized to zero. I didn't realize the things-before-main could mess up the stack so much.
      - kalmar 10y ago
- Annatar 10y ago
  > It's the OS zeroing everything - learn something very day No, the operating system (kernel, actually) does not zero out anything. The runtime linker would be the one initializing static memory declared in the ELF BSS sections at execution time. The rest (including the stack and heap) is setup by the prologue (crti.o, depending on the OS crt1.o, and crts.o).
Szpadel 10y ago
> Curiously, I found that if I had a buffer size of even 1 byte over (8 MB - 8 KB), I still got the segfault. I’m not yet sure what’s going on there! This is because of gcc padding. Programs have to allocate whole page from OS. So if you want just 1 int, you have to get whole page for it (compilers can optimize it in some conditions). This is result of MMU that works for memory block and not for single bytes (performance issue I think) But as I know by default page size have 4KB. Another reason may be that compiler tries to allocate 2^n bytes because of performance. and 8KB is close enough I think.
- dfox 10y ago
  The main problem there is that local non-static variables get placed onto stack. Stack space is allocated by generated code by just decrementing stack pointer without any explicit calls to OS. On typical modern unix, only few pages of stack are actually mapped and kernel handles page faults on neighboring pages by allocating more stack pages. Because it is possible that function needs more than one page of stack space, there is more than one such "magic" stack page, but still there is some finite number of them (othervise there would be no way to distinguish between accesses beyond the end of stack that should grow stack and accesses to random unmapped memory). Thus if you allocate some ridiculously large things on stack and access them in the direction opposite of stack growth, you may get segfault (accessing these structures in "the right order" is by no means sufficient for this to be safe because there are things like red-zones, signals, other local variables...). This is true only for first thread, other threads have fixed stack size specified on thread creation (on Linux it's 8MB by default), but usually even thread's stack pages are really allocated only on first access. On UNIX if you really want to bump stack by arbitrary amounts the most portable way is to preallocate your own stack of sufficient size and then use that (either by abusing sigaltstack() or via makecontext()/setcontext() or possibly by creating new thread). But generally, having large local variables is not exactly good idea.
  - bogomipz 10y ago
    Could you elaborate on "Stack space is allocated by generated code" Are you talking about the C runtime in the binary? Thanks.
Sean1708 10y ago
For those wondering about segfaults specifically in Rust (I know it's not the point of the blog post but it might be interesting to others), this thread talks about why they occur/whether they'll ever be eliminated entirely: https://users.rust-lang.org/t/rust-guarantees-no-segfaults-with-only-safe-code-but-it-segfaults-stack-overflow/4305 https://users.rust-lang.org/t/rust-guarantees-no-segfaults-w...
- steveklabnik 10y ago
  One small update to that thread: hitting the guard page no longer sends SIGSEV, but SIGILL: https://github.com/rust-lang/rust/pull/31333 https://github.com/rust-lang/rust/pull/31333
amelius 10y ago
> Segfaults are our friends and teachers Too bad memory is not better segmented then. For instance, when linking against a library, that library's memory ends up in the same "segment" as the program itself. Therefore, right now, you can totally screw up a library's internal data structures without even causing a segfault directly.
Annatar 10y ago
From the article: > The Rust program got a segmentation fault because it attempted to write to inaccessible memory, but only through the stack pointer. None of the undefined behaviors disallow this, which I think is why it’s ok for this Rust program to segfault. What I got out of that is that Rust does not work as advertised if there are still situations where a program could segfault. The entire premise of Rust, as I understood it at least, is that it does things in a safe manner and the programmer does not have to worry about it. Now I learned that there are undefined behaviors. In my view, for a language that bills itself as safe, there should not exist such things as undefined behaviors. As far as I am concerned, then, based on the advertising of Rust, this is false advertising.
- wofo 10y ago
  Of course there are cases in which you can get segfaults, and not only in Rust. The single fact of having a C FFI is enough to break all safety guarantees of your language. What Rust does promise is that undefined behavior will never happen if you don't use code blocks marked as unsafe (which include the C FFI). In other words, if you are able to trigger undefined behavior without using unsafe code, that is a compiler or library bug. EDIT: as noted by steveklabnik, this is not undefined behavior. Actually you should get a better error message instead of a segmentation fault (see the link to the issue on GitHub in steveklabnik's comment).
  - Ono-Sendai 10y ago
    And if you read the article, that is the case. You can get crashes without using blocks marked as unsafe.
    - wofo 10y ago
      I am not denying that. In fact, there is a GitHub issue intended to solve this (https://github.com/rust-lang/rust/issues/16012 https://github.com/rust-lang/rust/issues/16012).
  - jws 10y ago
    What does Rust promise if I infinitely recursively call a function which can not be tail optimized? Eventually the stack will exceed either available memory or available address space, what then?
catern 10y ago
The robust solution to this problem is not hardcoding the pipe buffer size and changing the size of pipe buffers within your program to match your hard coded value, but rather calling fgetconf to query the pipe buffer size for the pipe FD you are working with.
kibwen 10y ago
To be absolutely clear, the behavior exhibited in the OP is indeed considered a bug by the Rust developers: see https://github.com/rust-lang/rust/issues/16012#issuecomment-165589146 https://github.com/rust-lang/rust/issues/16012#issuecomment-... for the latest discussion. TL;DR: this currently isn't exploitable on Windows, and patches to LLVM adding support for stack probes will ideally allow this to be easily solved for non-Windows platforms as well.

60 points by kalmar 10y ago

These are called guard pages. Attempts to write there would result in a segmentation fault. ...which are caught by the OS and used to either truly kill the process when the stack overflows, or to dynamically allocate more memory as the stack grows downwards. That's how it works on Windows, at least; I'm not as clear about Linux.

EliRivers 10y ago

The first sample code; "This program segfaults because the entire stack is set to 0 at program start." I'd be surprised; as a strong general rule, the stack does not get zeroed [Edit: see end of thread! It's the OS zeroing everything - learn something very day]. I'd expect it to segfault because the pointer value is whatever leftover non-zero value happens to be in that piece of memory, so it points into random memory the user program shouldn't be messing with (sticking in a printf to output the value of the pointer confirms this on at least one system). Wouldn't be surprised if some implementations took security really seriously and zero everything, or if a debug build was zero happy, but under normal circumstances, the stack doesn't get zeroed.

dkopi 10y ago
edit: exDM69 actually makes more sense.
- EliRivers 10y ago
  Well, I can see the build command on the webpage there, and assuming this is the same gcc we all know and love without any cleverness going on, it's not a debug build of any kind as far as I can see: gcc segfault.c Now that I think about it, I'm not aware that gcc C compiler even offers the option to deliberately zero initialise non-static local variables, so unless I've missed a switch somewhere there is no arrangement of options available for a "debug build" to do this. I recall the GCC Fortran compiler did offer it.
  - exDM69 10y ago
    It's the OS linker/loader that zeros the pages used by a new process to avoid leaking memory contents of previous processes (security risk). It's zero only because this occurs at the very beginning of the program. In general it would be undefined.
    - EliRivers 10y ago
      Nice. Learn something every day. I edited my original post to add this; save anyone else tracing through the whole thread.
    - Buge 10y ago
      The OS will initialize the memory of the stack to 0 before the program starts. But before main is called, the compiler is free to insert other code that runs before main. gcc inserts a function named __libc_start_main that runs before main. This code will modify the content of the stack. So when main is run, the stack where the uninitialized local variable is has a decent chance of not being 0 anymore. This is easily testable. #include <stdio.h> int main(void) { char* pointers[20]; int i; for (i = 0; i < 20; ++i) { printf("%p\n", pointers[i]); } return 0; } And yes when I run it, most of the pointers are not null.
      - exDM69 10y ago
        Sure, the C runtime initialization runs before main. Unless you're looking at the stack at _start, it's probably unintialized. And it depends on your libc implementation, etc. It just happened to be zero in the author's case.
      - kalmar 10y ago
        Oh this is cool! I should have checked more pointers. On my machine, the single pointer was always null, and I read up on stack being initialized to zero. I didn't realize the things-before-main could mess up the stack so much.
    - kalmar 10y ago
Annatar 10y ago
> It's the OS zeroing everything - learn something very day No, the operating system (kernel, actually) does not zero out anything. The runtime linker would be the one initializing static memory declared in the ELF BSS sections at execution time. The rest (including the stack and heap) is setup by the prologue (crti.o, depending on the OS crt1.o, and crts.o).

Szpadel 10y ago

> Curiously, I found that if I had a buffer size of even 1 byte over (8 MB - 8 KB), I still got the segfault. I’m not yet sure what’s going on there! This is because of gcc padding. Programs have to allocate whole page from OS. So if you want just 1 int, you have to get whole page for it (compilers can optimize it in some conditions). This is result of MMU that works for memory block and not for single bytes (performance issue I think) But as I know by default page size have 4KB. Another reason may be that compiler tries to allocate 2^n bytes because of performance. and 8KB is close enough I think.

dfox 10y ago
The main problem there is that local non-static variables get placed onto stack. Stack space is allocated by generated code by just decrementing stack pointer without any explicit calls to OS. On typical modern unix, only few pages of stack are actually mapped and kernel handles page faults on neighboring pages by allocating more stack pages. Because it is possible that function needs more than one page of stack space, there is more than one such "magic" stack page, but still there is some finite number of them (othervise there would be no way to distinguish between accesses beyond the end of stack that should grow stack and accesses to random unmapped memory). Thus if you allocate some ridiculously large things on stack and access them in the direction opposite of stack growth, you may get segfault (accessing these structures in "the right order" is by no means sufficient for this to be safe because there are things like red-zones, signals, other local variables...). This is true only for first thread, other threads have fixed stack size specified on thread creation (on Linux it's 8MB by default), but usually even thread's stack pages are really allocated only on first access. On UNIX if you really want to bump stack by arbitrary amounts the most portable way is to preallocate your own stack of sufficient size and then use that (either by abusing sigaltstack() or via makecontext()/setcontext() or possibly by creating new thread). But generally, having large local variables is not exactly good idea.
- bogomipz 10y ago
  Could you elaborate on "Stack space is allocated by generated code" Are you talking about the C runtime in the binary? Thanks.

Sean1708 10y ago

For those wondering about segfaults specifically in Rust (I know it's not the point of the blog post but it might be interesting to others), this thread talks about why they occur/whether they'll ever be eliminated entirely: https://users.rust-lang.org/t/rust-guarantees-no-segfaults-with-only-safe-code-but-it-segfaults-stack-overflow/4305 https://users.rust-lang.org/t/rust-guarantees-no-segfaults-w...

steveklabnik 10y ago
One small update to that thread: hitting the guard page no longer sends SIGSEV, but SIGILL: https://github.com/rust-lang/rust/pull/31333 https://github.com/rust-lang/rust/pull/31333

amelius 10y ago

> Segfaults are our friends and teachers Too bad memory is not better segmented then. For instance, when linking against a library, that library's memory ends up in the same "segment" as the program itself. Therefore, right now, you can totally screw up a library's internal data structures without even causing a segfault directly.

Annatar 10y ago

From the article: > The Rust program got a segmentation fault because it attempted to write to inaccessible memory, but only through the stack pointer. None of the undefined behaviors disallow this, which I think is why it’s ok for this Rust program to segfault. What I got out of that is that Rust does not work as advertised if there are still situations where a program could segfault. The entire premise of Rust, as I understood it at least, is that it does things in a safe manner and the programmer does not have to worry about it. Now I learned that there are undefined behaviors. In my view, for a language that bills itself as safe, there should not exist such things as undefined behaviors. As far as I am concerned, then, based on the advertising of Rust, this is false advertising.

wofo 10y ago
Of course there are cases in which you can get segfaults, and not only in Rust. The single fact of having a C FFI is enough to break all safety guarantees of your language. What Rust does promise is that undefined behavior will never happen if you don't use code blocks marked as unsafe (which include the C FFI). In other words, if you are able to trigger undefined behavior without using unsafe code, that is a compiler or library bug. EDIT: as noted by steveklabnik, this is not undefined behavior. Actually you should get a better error message instead of a segmentation fault (see the link to the issue on GitHub in steveklabnik's comment).
- Ono-Sendai 10y ago
  And if you read the article, that is the case. You can get crashes without using blocks marked as unsafe.
  - wofo 10y ago
    I am not denying that. In fact, there is a GitHub issue intended to solve this (https://github.com/rust-lang/rust/issues/16012 https://github.com/rust-lang/rust/issues/16012).
- jws 10y ago
  What does Rust promise if I infinitely recursively call a function which can not be tail optimized? Eventually the stack will exceed either available memory or available address space, what then?

catern 10y ago

The robust solution to this problem is not hardcoding the pipe buffer size and changing the size of pipe buffers within your program to match your hard coded value, but rather calling fgetconf to query the pipe buffer size for the pipe FD you are working with.

kibwen 10y ago

To be absolutely clear, the behavior exhibited in the OP is indeed considered a bug by the Rust developers: see https://github.com/rust-lang/rust/issues/16012#issuecomment-165589146 https://github.com/rust-lang/rust/issues/16012#issuecomment-... for the latest discussion. TL;DR: this currently isn't exploitable on Windows, and patches to LLVM adding support for stack probes will ideally allow this to be easily solved for non-Windows platforms as well.