3 ms·

To rustc not calling stacker enough/at the right times, the behaviour on MSVC/Windows is for the compiler to rely on hitting the OS's guard page to extend the s

To rustc not calling stacker enough/at the right times, the behaviour on MSVC/Windows is for the compiler to rely on hitting the OS's guard page to extend the stack (rather than growing it yourself), but also for the compiler to emit a special routine in any function that uses more than a page of stack frame (to make sure the first thing the function does is poke every page in order, so the OS can grow the stack the right amount).

sunshowers 2y ago
I believe that's how rustc works on all platforms. rustc's soundness story depends on each call stack having one guard page, and ensuring that each time a frame is created every page gets poked at least once.
- dwattttt 2y ago
  Oh, is stacker requesting stack growth that otherwise wouldn't happen? I always just assume address space reservations are cheap/free, and you can just mark a heap (hah) of space for your stacks. I guess it also has the behaviour that you'll find out which programs are stack hogs earlier.
  - sunshowers 2y ago
    Yes, stacker has to be called explicitly, and in the case of an arbitrarily recursive program at each N recursion levels -- it's not something like a signal handler that sits in the background and gets activated when the thread runs out of stack space.
    - dwattttt 2y ago
      It's funny the parallel of "detect when the mapped stack region is exhausted, and map the next page" and "detect when the stack region is/will exhaust, and reserve/allocate another region"
      - sunshowers 2y ago
        Yeah, I was thinking about that myself. This post suggests that actually growing the stack automatically may just be very difficult with standard POSIX APIs: https://news.ycombinator.com/item?id=36020073 https://news.ycombinator.com/item?id=36020073 The Rust runtime installs a SIGSEGV handler but it's purely informative, and it can be overridden without affecting correctness.
        dwattttt 2y ago
        I took a brief look into Windows public API, even with a tight coupling between compiler/toolchain & OS they don't offer a way to increase the region for a stack after a thread has been created.
        ynik 2y ago
        But if you're on 64-bit; you can just create the threads with a huge stack size limit (e.g. 1GB) and let the OS handle automatically growing the actual stack size. No need to reinvent the wheel. Stack size is the one area where Windows does have something like "overcommit" by default, as you can separately configure the reserved and commit sizes of the stack: https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t...
        SkiFire13 2y ago
        This just delays the problem though
        sunshowers 2y ago
        Yes, this approach works on other platforms too. However I believe that this will not deallocate stack space, while stacker will.
        fch42 2y ago
        You can only grow the stack automatically if you "prereserve" the virtual address space for it. Then it becomes just a question of "committing" the memory at time of need. Default options to mmap() should actually give you this (the OS finds/fills the pages for you on first pagefault), but Solaris/Linux differ on overcommit behaviour. To "grow" a stack at runtime, if the virtual address space to just add mappings in the "right" place is unavailable (already used for something else, say), it means a stack switch. That isn't entirely impossible. But if done anywhere else but in the "base thread initialisation" (as seen in that example) it becomes complicated and hazardous. It's more like "coroutines" then, your current function "spawns" a func on a new callstack and on return, undoes that and cleans up. One can imagine things like thread-local signal handlers to "simulate" this transparently but that would become more of a piece of programming performance art. If there ever were an "unbelievable unix underbelly programming contest" (UUUPC) stuff like that might well make it.

sunshowers 2y ago

I believe that's how rustc works on all platforms. rustc's soundness story depends on each call stack having one guard page, and ensuring that each time a frame is created every page gets poked at least once.

dwattttt 2y ago
Oh, is stacker requesting stack growth that otherwise wouldn't happen? I always just assume address space reservations are cheap/free, and you can just mark a heap (hah) of space for your stacks. I guess it also has the behaviour that you'll find out which programs are stack hogs earlier.
- sunshowers 2y ago
  Yes, stacker has to be called explicitly, and in the case of an arbitrarily recursive program at each N recursion levels -- it's not something like a signal handler that sits in the background and gets activated when the thread runs out of stack space.
  - dwattttt 2y ago
    It's funny the parallel of "detect when the mapped stack region is exhausted, and map the next page" and "detect when the stack region is/will exhaust, and reserve/allocate another region"
    - sunshowers 2y ago
      Yeah, I was thinking about that myself. This post suggests that actually growing the stack automatically may just be very difficult with standard POSIX APIs: https://news.ycombinator.com/item?id=36020073 https://news.ycombinator.com/item?id=36020073 The Rust runtime installs a SIGSEGV handler but it's purely informative, and it can be overridden without affecting correctness.
      - dwattttt 2y ago
        I took a brief look into Windows public API, even with a tight coupling between compiler/toolchain & OS they don't offer a way to increase the region for a stack after a thread has been created.
        ynik 2y ago
        But if you're on 64-bit; you can just create the threads with a huge stack size limit (e.g. 1GB) and let the OS handle automatically growing the actual stack size. No need to reinvent the wheel. Stack size is the one area where Windows does have something like "overcommit" by default, as you can separately configure the reserved and commit sizes of the stack: https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size https://learn.microsoft.com/en-us/windows/win32/procthread/t...
        SkiFire13 2y ago
        This just delays the problem though
        sunshowers 2y ago
        Yes, this approach works on other platforms too. However I believe that this will not deallocate stack space, while stacker will.
      - fch42 2y ago
        You can only grow the stack automatically if you "prereserve" the virtual address space for it. Then it becomes just a question of "committing" the memory at time of need. Default options to mmap() should actually give you this (the OS finds/fills the pages for you on first pagefault), but Solaris/Linux differ on overcommit behaviour. To "grow" a stack at runtime, if the virtual address space to just add mappings in the "right" place is unavailable (already used for something else, say), it means a stack switch. That isn't entirely impossible. But if done anywhere else but in the "base thread initialisation" (as seen in that example) it becomes complicated and hazardous. It's more like "coroutines" then, your current function "spawns" a func on a new callstack and on return, undoes that and cleans up. One can imagine things like thread-local signal handlers to "simulate" this transparently but that would become more of a piece of programming performance art. If there ever were an "unbelievable unix underbelly programming contest" (UUUPC) stuff like that might well make it.