5 ms·

Well, in general, POSIX threads are much more expensive (RAM) than some unit of minimal cooperative concurrency/parallelism, say Erlang "processes." The idea of

Well, in general, POSIX threads are much more expensive (RAM) than some unit of minimal cooperative concurrency/parallelism, say Erlang "processes." The idea of using a threadpool isn't broken because an user-space "scheduler" decides which tasks to run on which threads. It might also decide how to scale or shrink the threadpool. Ultimately, only one thing can run on a processor at a given time, and that typical means a task structure containing at least two items if executing on an interpreted/p-code VM: 0. next unit of work (pointer/counter; instruction, function pointer, etc.) 1. task-local heap (pointer or structure) 2. operand stack (pointer or structure; for stack-oriented VMs only)

pcwalton 7y ago
Most of the memory cost of a POSIX thread is in the stack, and you can customize the stack size to be quite small. Small stacks are properly thought of as a property that GC enables, not a property that M:N threading enables.
- masklinn 7y ago
  > Most of the memory cost of a POSIX thread is in the stack, and you can customize the stack size to be quite small. The problem there is that you need to very carefully size your stack as a mis-sizing will lead to a risky stack overflow. I'm not sure it's necessary either as allocating a "large stack" but using very little of it means most of it is never committed, and thus only costs memory mappings.
  - pcwalton 7y ago
    If you have the runtime infrastructure to grow stacks, then you can use that with POSIX threads too.
    - masklinn 7y ago
      Is there any systems where the C stack is growable? Do stack frames get prefixed with an explicit request for some amount of stack memory, leading to the stack possibly being moved before the funcall happens?
      - pcwalton 7y ago
        We used to have growable C stacks in Rust using that technique. They worked (though were too slow for us). It could have been fixed by using stack copying like Go does.
- sagichmal 7y ago
  Is there a stack size configuration that would enable a program to spawn 1M POSIX threads?
  - pcwalton 7y ago
    As I recall the minimum total user + kernel stack size is 10kB in the Linux kernel, so 1M threads is 10GB of space. It should be doable, though you will probably have to bump up kernel limits. A million threads is an extreme case, though. No system can reliably spawn that many threads that are actually doing something interesting without a very large amount of memory. When you leave the realm of microbenchmarks you have to expect that an unknown quantity of threads will have deep call stacks at any given time, so you really need to give yourself leeway to avoid the risk of OOM.
    - sagichmal 7y ago
      So a big advantage that Go's M:N goroutine model brings to the table is how cheap they are. Cheap enough that tons of concurrency-related stuff you want to do in application code, like implementing a highly-concurrent algorithm, can be done with goroutines directly, without having to think too hard about mechanical sympathy and e.g. translate logical concurrency to physical threading. Go processes commonly have 1M or even 10M active goroutines at once. So I don't think it's fair to say POSIX threads are comparable or whatever if they don't have this property.
      - pcwalton 7y ago
        Go processes do not typically have 1M or 10M active goroutines at once. The initial stack size for goroutines is 2kB, so 10M goroutines would mean 20GB just for stacks, even assuming that the goroutines never grow their stack (which cannot be assumed for anything nontrivial). The 2kB minimum stack size is on the same order of magnitude as the 10kB POSIX thread stack size.
        sagichmal 7y ago
        > Go processes do not typically have 1M or 10M active goroutines at once. It depends on domain, but in my domain of high-RPS network servers, they absolutely do. > 10M goroutines would mean 20GB just for stacks. When I deploy to metal, the average host has ~512GB of RAM, and not much cotenancy. > The 2kB minimum stack size is on the same order of magnitude as the 10kB POSIX thread stack size. There's also a question of cost to create and destroy; it's very common for goroutines to live for O(µs). I don't know how POSIX threads compare here.
masklinn 7y ago
> Well, in general, POSIX threads are much more expensive (RAM) than some unit of minimal cooperative concurrency/parallelism, say Erlang "processes." The vast majority of the "expense" is irrelevant as it's virtual memory and unlikely to ever be touched (and thus committed).
- loeg 7y ago
  Depends on your codebase. Userspace C libraries and programs, including libc, often store surprisingly large buffers on the stack. For example, try setting 'ulimit -s 128' (128kB stack limit) and see how many C programs crash. Then try, say, 16. Go's default is 8 kB, raised from 4 kB in 1.2: https://golang.org/doc/go1.2#stack_size https://golang.org/doc/go1.2#stack_size Linux's default userspace stack limit is 8 megabytes for a reason — programs really do use it.
  - masklinn 7y ago
    > for a reason Not really, the 8MB limit was added back in '95 from a previous limit of "essentially none"[0] with a justification of > Limit the stack by to some sane default: root can always increase this limit if needed.. 8MB seems reasonable. Developers don't generally think about their stack size, especially for single-threaded programs[1] so the defaults need to be a sweet spot of not unnecessarily big (such that you can catch unbounded recursion) but not so small that you'd segfault more than a very small fraction of all programs. [0] https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/diff/include/linux/sched.h?h=1.3.7&id=2d52d174c910808b03474c786fc42c7c7e869bd5 https://git.kernel.org/pub/scm/linux/kernel/git/history/hist... [1] which would be why e.g. OSX has a large main thread stack (8MB) and a relatively puny secondary thread stack (512k).

pcwalton 7y ago

Most of the memory cost of a POSIX thread is in the stack, and you can customize the stack size to be quite small. Small stacks are properly thought of as a property that GC enables, not a property that M:N threading enables.

masklinn 7y ago
> Most of the memory cost of a POSIX thread is in the stack, and you can customize the stack size to be quite small. The problem there is that you need to very carefully size your stack as a mis-sizing will lead to a risky stack overflow. I'm not sure it's necessary either as allocating a "large stack" but using very little of it means most of it is never committed, and thus only costs memory mappings.
- pcwalton 7y ago
  If you have the runtime infrastructure to grow stacks, then you can use that with POSIX threads too.
  - masklinn 7y ago
    Is there any systems where the C stack is growable? Do stack frames get prefixed with an explicit request for some amount of stack memory, leading to the stack possibly being moved before the funcall happens?
    - pcwalton 7y ago
      We used to have growable C stacks in Rust using that technique. They worked (though were too slow for us). It could have been fixed by using stack copying like Go does.
sagichmal 7y ago
Is there a stack size configuration that would enable a program to spawn 1M POSIX threads?
- pcwalton 7y ago
  As I recall the minimum total user + kernel stack size is 10kB in the Linux kernel, so 1M threads is 10GB of space. It should be doable, though you will probably have to bump up kernel limits. A million threads is an extreme case, though. No system can reliably spawn that many threads that are actually doing something interesting without a very large amount of memory. When you leave the realm of microbenchmarks you have to expect that an unknown quantity of threads will have deep call stacks at any given time, so you really need to give yourself leeway to avoid the risk of OOM.
  - sagichmal 7y ago
    So a big advantage that Go's M:N goroutine model brings to the table is how cheap they are. Cheap enough that tons of concurrency-related stuff you want to do in application code, like implementing a highly-concurrent algorithm, can be done with goroutines directly, without having to think too hard about mechanical sympathy and e.g. translate logical concurrency to physical threading. Go processes commonly have 1M or even 10M active goroutines at once. So I don't think it's fair to say POSIX threads are comparable or whatever if they don't have this property.
    - pcwalton 7y ago
      Go processes do not typically have 1M or 10M active goroutines at once. The initial stack size for goroutines is 2kB, so 10M goroutines would mean 20GB just for stacks, even assuming that the goroutines never grow their stack (which cannot be assumed for anything nontrivial). The 2kB minimum stack size is on the same order of magnitude as the 10kB POSIX thread stack size.
      - sagichmal 7y ago
        > Go processes do not typically have 1M or 10M active goroutines at once. It depends on domain, but in my domain of high-RPS network servers, they absolutely do. > 10M goroutines would mean 20GB just for stacks. When I deploy to metal, the average host has ~512GB of RAM, and not much cotenancy. > The 2kB minimum stack size is on the same order of magnitude as the 10kB POSIX thread stack size. There's also a question of cost to create and destroy; it's very common for goroutines to live for O(µs). I don't know how POSIX threads compare here.

masklinn 7y ago

> Well, in general, POSIX threads are much more expensive (RAM) than some unit of minimal cooperative concurrency/parallelism, say Erlang "processes." The vast majority of the "expense" is irrelevant as it's virtual memory and unlikely to ever be touched (and thus committed).

loeg 7y ago
Depends on your codebase. Userspace C libraries and programs, including libc, often store surprisingly large buffers on the stack. For example, try setting 'ulimit -s 128' (128kB stack limit) and see how many C programs crash. Then try, say, 16. Go's default is 8 kB, raised from 4 kB in 1.2: https://golang.org/doc/go1.2#stack_size https://golang.org/doc/go1.2#stack_size Linux's default userspace stack limit is 8 megabytes for a reason — programs really do use it.
- masklinn 7y ago
  > for a reason Not really, the 8MB limit was added back in '95 from a previous limit of "essentially none"[0] with a justification of > Limit the stack by to some sane default: root can always increase this limit if needed.. 8MB seems reasonable. Developers don't generally think about their stack size, especially for single-threaded programs[1] so the defaults need to be a sweet spot of not unnecessarily big (such that you can catch unbounded recursion) but not so small that you'd segfault more than a very small fraction of all programs. [0] https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/diff/include/linux/sched.h?h=1.3.7&id=2d52d174c910808b03474c786fc42c7c7e869bd5 https://git.kernel.org/pub/scm/linux/kernel/git/history/hist... [1] which would be why e.g. OSX has a large main thread stack (8MB) and a relatively puny secondary thread stack (512k).