7 ms·

Obviously I'm missing something here, because I was under the impression that, regardless of how big the stack is, the pages are going to be mapped (in the data

Obviously I'm missing something here, because I was under the impression that, regardless of how big the stack is, the pages are going to be mapped (in the data structure pointed to by the CR3 register) when accessed and no earlier. Unless the allocation of the stack makes sure that it is mapped in its entirety upfront and stack access cannot incur minor page faults, which may not be so outlandish come to think of it. Can you please provide some clarification here? Thanks!

masklinn 6y ago
I'm not sure what clarification I could provide given I don't know what you're missing. You're correct (AFAIK anyway) that stack pages are mapped lazily. That doesn't change the fact that C stacks are large allocations, while Go will only allocate a very small stack (2k last I checked) per goroutine.
- aduitsis 6y ago
  Yes, sorry, let me be more clear, I don't understand why Go should allocate a small 2k stack (you are correct, the Go stack size has fluctuated through the years, nowadays it's 2k) and if it grows go to the trouble of copying it to a larger structure? Instead of just allocating a big stack that will grow lazily anyway? Obviously the folks that created Go aren't stupid, far from it, so there must be a real and valid reason. But can't easily imagine what it is.
  - JulianMorrison 6y ago
    Go is very, very profligate with threads. When you have 100,000 threads, big stacks add up.
    - masklinn 6y ago
      It's only vmem though is the point. It doesn't really matter that you have 100000 stacks of 8MB when all of that is vmem and you've got a page committed on each.
      - aduitsis 6y ago
        My point exactly, thanks.
      - tuxychandru 6y ago
        This will mean a virtual memory allocation of 800 GB. Such large allocations can fail, even if virtual, depending on over commit settings.
        masklinn 6y ago
        > Such large allocations can fail, even if virtual, depending on over commit settings. Given how Go has no issue being prescriptive as hell on other things, I don't see why they couldn't just go "set vm. overcommit_memory to 1 and fuck off", that's exactly what e.g. Redis tells you.
        pjmlp 6y ago
        Because that is very OS specific.
        masklinn 6y ago
        AFAIK BSDs don't have a concept of overcommit because you can't even turn it off. And again it would hardly be the first time Go makes OS-specific decisions.
      - labawi 6y ago
        Each allocation also needs an entry in the page table (actually a virtual memory tree). On x86/amd64 the fanout is about 1:512. If you have 8MiB stacks, then a minimally allocated stack uses a 4KiB data page, but also 2MiB of address range uses up a full 4KiB bottom level page table, and 8MiB range takes up 4/512 x 4KB of 2nd level page table and so on. So you use about 8.03 KiB RAM if you never touch more than 4KB and your 8MB reservations are mostly grouped. Some architectures have bigger pages, increasing fanout but also the minimum allocation. Contrast to 2KiB stacks without reservations/overcommit - you use about 2KiB of usable RAM + (1/2) x (1/512) of 4KiB 1st level page table + .. , assuming allocation are again mostly grouped. Hence, for up to 2KiB of stack memory you need about¹ 2.005 KiB of RAM. Works the same for 16 KiB and even 64 KiB page sizes. 100000 * (8MiB reserved, 1..4KiB used stack) needs ~784MiB RAM. 100000 * (2KiB reserved, 1..2KiB used stack) needs ~200MiB RAM. Note that, if you actually touch your reserved stack, even once, your allocation can balloon to possibly tens or hundreds of GiB (100k * 8MiB = ~800GiB), unless you do complicated cleanup, while a segmented stack can keep the allocations within reasonable efficiency, freeing any excessive stack allocations in userspace. ¹ ignoring bookkeeping overhead in both cases, to keep the calculation clear. Hopefully it isn't more than a dozen bytes or so.
- simias 6y ago
  Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads. I have no idea about what the Go scheduler looks like though, so I can't say if there's a direct relation between the two (coroutine stack vs. thread stack). Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads, but with pthread_attr_setstacksize you can lower it. I can't seem to find the actual minimal size at the moment but I have a vague memory that you can reduce it to 16kB portably. I'm currently working on a multithreaded Rust program with a bunch of threads and strong memory constraints and I use the runtime to reduce the stack to 64kB, it seems to work just fine. And again, given that the language is garbage collected I can't help but find it a bit amusing that they're being stingy with a few MB of VMEM per thread. I guess that frees virtual memory for use in the balasts!
  - masklinn 6y ago
    > Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads. It matters for goroutines because that means you can't straight call into foreign code from a goroutine's stack, which is why cgo and friends are so problematic. > Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads That's not relevant because it's not vmem is the point. The actual resident size of a 2MB stack is 4k unless you start dirtying more pages. Unless you're running on 32b, VMEM doesn't really matter.

masklinn 6y ago

I'm not sure what clarification I could provide given I don't know what you're missing. You're correct (AFAIK anyway) that stack pages are mapped lazily. That doesn't change the fact that C stacks are large allocations, while Go will only allocate a very small stack (2k last I checked) per goroutine.

aduitsis 6y ago
Yes, sorry, let me be more clear, I don't understand why Go should allocate a small 2k stack (you are correct, the Go stack size has fluctuated through the years, nowadays it's 2k) and if it grows go to the trouble of copying it to a larger structure? Instead of just allocating a big stack that will grow lazily anyway? Obviously the folks that created Go aren't stupid, far from it, so there must be a real and valid reason. But can't easily imagine what it is.
- JulianMorrison 6y ago
  Go is very, very profligate with threads. When you have 100,000 threads, big stacks add up.
  - masklinn 6y ago
    It's only vmem though is the point. It doesn't really matter that you have 100000 stacks of 8MB when all of that is vmem and you've got a page committed on each.
    - aduitsis 6y ago
      My point exactly, thanks.
    - tuxychandru 6y ago
      This will mean a virtual memory allocation of 800 GB. Such large allocations can fail, even if virtual, depending on over commit settings.
      - masklinn 6y ago
        > Such large allocations can fail, even if virtual, depending on over commit settings. Given how Go has no issue being prescriptive as hell on other things, I don't see why they couldn't just go "set vm. overcommit_memory to 1 and fuck off", that's exactly what e.g. Redis tells you.
        pjmlp 6y ago
        Because that is very OS specific.
        masklinn 6y ago
        AFAIK BSDs don't have a concept of overcommit because you can't even turn it off. And again it would hardly be the first time Go makes OS-specific decisions.
    - labawi 6y ago
      Each allocation also needs an entry in the page table (actually a virtual memory tree). On x86/amd64 the fanout is about 1:512. If you have 8MiB stacks, then a minimally allocated stack uses a 4KiB data page, but also 2MiB of address range uses up a full 4KiB bottom level page table, and 8MiB range takes up 4/512 x 4KB of 2nd level page table and so on. So you use about 8.03 KiB RAM if you never touch more than 4KB and your 8MB reservations are mostly grouped. Some architectures have bigger pages, increasing fanout but also the minimum allocation. Contrast to 2KiB stacks without reservations/overcommit - you use about 2KiB of usable RAM + (1/2) x (1/512) of 4KiB 1st level page table + .. , assuming allocation are again mostly grouped. Hence, for up to 2KiB of stack memory you need about¹ 2.005 KiB of RAM. Works the same for 16 KiB and even 64 KiB page sizes. 100000 * (8MiB reserved, 1..4KiB used stack) needs ~784MiB RAM. 100000 * (2KiB reserved, 1..2KiB used stack) needs ~200MiB RAM. Note that, if you actually touch your reserved stack, even once, your allocation can balloon to possibly tens or hundreds of GiB (100k * 8MiB = ~800GiB), unless you do complicated cleanup, while a segmented stack can keep the allocations within reasonable efficiency, freeing any excessive stack allocations in userspace. ¹ ignoring bookkeeping overhead in both cases, to keep the calculation clear. Hopefully it isn't more than a dozen bytes or so.
simias 6y ago
Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads. I have no idea about what the Go scheduler looks like though, so I can't say if there's a direct relation between the two (coroutine stack vs. thread stack). Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads, but with pthread_attr_setstacksize you can lower it. I can't seem to find the actual minimal size at the moment but I have a vague memory that you can reduce it to 16kB portably. I'm currently working on a multithreaded Rust program with a bunch of threads and strong memory constraints and I use the runtime to reduce the stack to 64kB, it seems to work just fine. And again, given that the language is garbage collected I can't help but find it a bit amusing that they're being stingy with a few MB of VMEM per thread. I guess that frees virtual memory for use in the balasts!
- masklinn 6y ago
  > Goroutines don't map directly to OS threads though, do they? So in practice it only matters for "hard" threads. It matters for goroutines because that means you can't straight call into foreign code from a goroutine's stack, which is why cgo and friends are so problematic. > Also the default pthread stack size is (usually) 2MB which is indeed non negligible if you have a ton of threads That's not relevant because it's not vmem is the point. The actual resident size of a 2MB stack is 4k unless you start dirtying more pages. Unless you're running on 32b, VMEM doesn't really matter.