4 ms·

Hi Colin! I don't necessarily disagree with you, but I feel like your arguments address semantics/linguistics and not the actual issue. First, this limit isn't

Hi Colin! I don't necessarily disagree with you, but I feel like your arguments address semantics/linguistics and not the actual issue. First, this limit isn't set by his computer. Obviously the size of the stack depends on a number of variables (including the hardware architecture, the kernel, how the kernel was compiled, various settings, etc.), but on Dan's configuration it's roughly 10MB, as verified by the program at the end of the blog post. Here is a program that confirms it by computing the number in a different way: #include <pthread.h> #include <stdio.h> int main(int argc, char *argv[]) { size_t stacksize; pthread_attr_t attr; pthread_attr_init(&attr); pthread_attr_getstacksize (&attr, &stacksize); printf("Default stack size = %lu\n", stacksize); } On our configuration (which is roughly what most customers are expected to run) the output is 8388608, or 8MB. According to the manual, pthread_attr_getstacksize returns the minimum size (in bytes) that will be allocated for threads created using the thread attributes object attr. Naturally the size can be changed manually (in the code or with ulimit), but it is the default minimum for newly created threads according to the manual. The point wasn't that you couldn't change this value, it's that for our purposes (lots of green threads, shallow stacks) a much smaller default stack size is more sensible. Second, processes don't have stacks. Threads have stacks. Certainly, but this is a question of terminology. On Linux threads are special-case processes (with address-space sharing and a few other changes). For example, when calling strace you have to pass -f to see syscalls from all the threads instead of just the main one. Third, the stack starts out small -- like, a few hundred bytes or so. Again, not according to the manual (see above). You guys have heard of userland threads, right? When we get the chance we'll post benchmarks between GNU Pth, NPTL, and our coroutine implementation. I can assure you that NPTL is much slower than what we're doing. GNU Pth is a real contender, but there are some issues associated with using it (mainly the fact that we need to be able to move coroutines between native kernel threads, and it's not clear how well GNU Pth can handle that). The coroutines implementation won't be making it into the first release - ultimately we'll test all three alternatives and pick the best one, but it will take some time before that happens.

cperciva 16y ago
I don't necessarily disagree with you, but I feel like your arguments address semantics/linguistics and not the actual issue. It might be that, yes. But as a philosophy professor of my acquaintance likes to remark, "clear thinking requires clear language" (the irony of this phrase coming from a philosopher's mouth seems to be lost on him). Using unclear language both misleads the audience and makes it unclear whether the speaker understands what he's talking about. (I find the same is true of code -- the clarity of some code seems very highly correlated with how well its author understands the problem space in question.) Between us we're using three different definitions of "stack size" -- the size of the actual stack, the amount of RAM allocated, and the amount of VM space allocated; needless to say, I think the first definition makes the most sense. ;-) When we get the chance we'll post benchmarks between GNU Pth, NPTL, and our coroutine implementation. I recommend trying other operating systems too. Linux is famously bad at threading, and only survives in benchmarks because most applications which run on Linux are equally bad in how they use threads. What you really want (and seem to be implementing) is an M:N threading system, which is considerably more performant than either 1:1 or M:1.
- coffeemug 16y ago
  But as a philosophy professor of my acquaintance likes to remark, "clear thinking requires clear language" I think you need to balance that with taking some shortcuts, or you'll never get to the essence (or at least your readers won't).
leif 16y ago
> The point wasn't that you couldn't change this value, it's that for our purposes (lots of green threads, shallow stacks) a much smaller default stack size is more sensible. Slava, you're better than that. WHY is a smaller default stack size more sensible? If this is all you need, and you can change it manually, do that!
- coffeemug 16y ago
  WHY is a smaller default stack size more sensible? Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance. If this is all you need, and you can change it manually, do that! This needs to be understood in the context of the previous posts. The issue with NPTL is significantly lower performance on large number of threads due to context switching. Dan implemented a solution that uses coroutines to address that problem. This post deals with the over-provisioning issue described above, and it starts out by describing the default configuration for pthreads. Adjusting this to a smaller number on the NPTL implementation isn't enough, we needed to adjust it for the coroutines implementation.
  - cperciva 16y ago
    Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance. Say again? Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space.
    - coffeemug 16y ago
      Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space. For various reasons we don't use the kernel's cache, we have our own cache implementation that's in the same address space as the rest of the process.

cperciva 16y ago

I don't necessarily disagree with you, but I feel like your arguments address semantics/linguistics and not the actual issue. It might be that, yes. But as a philosophy professor of my acquaintance likes to remark, "clear thinking requires clear language" (the irony of this phrase coming from a philosopher's mouth seems to be lost on him). Using unclear language both misleads the audience and makes it unclear whether the speaker understands what he's talking about. (I find the same is true of code -- the clarity of some code seems very highly correlated with how well its author understands the problem space in question.) Between us we're using three different definitions of "stack size" -- the size of the actual stack, the amount of RAM allocated, and the amount of VM space allocated; needless to say, I think the first definition makes the most sense. ;-) When we get the chance we'll post benchmarks between GNU Pth, NPTL, and our coroutine implementation. I recommend trying other operating systems too. Linux is famously bad at threading, and only survives in benchmarks because most applications which run on Linux are equally bad in how they use threads. What you really want (and seem to be implementing) is an M:N threading system, which is considerably more performant than either 1:1 or M:1.

coffeemug 16y ago
But as a philosophy professor of my acquaintance likes to remark, "clear thinking requires clear language" I think you need to balance that with taking some shortcuts, or you'll never get to the essence (or at least your readers won't).

leif 16y ago

> The point wasn't that you couldn't change this value, it's that for our purposes (lots of green threads, shallow stacks) a much smaller default stack size is more sensible. Slava, you're better than that. WHY is a smaller default stack size more sensible? If this is all you need, and you can change it manually, do that!

coffeemug 16y ago
WHY is a smaller default stack size more sensible? Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance. If this is all you need, and you can change it manually, do that! This needs to be understood in the context of the previous posts. The issue with NPTL is significantly lower performance on large number of threads due to context switching. Dan implemented a solution that uses coroutines to address that problem. This post deals with the over-provisioning issue described above, and it starts out by describing the default configuration for pthreads. Adjusting this to a smaller number on the NPTL implementation isn't enough, we needed to adjust it for the coroutines implementation.
- cperciva 16y ago
  Because if you have a large number of threads on an architecture with a small address space, you take away memory space from the buffer cache which can have catastrophic effects on performance. Say again? Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space.
  - coffeemug 16y ago
    Address space exhaustion is a problem, sure, but it has nothing to do with the buffer cache. The kernel's address space is completely separate from any particular process' address space. For various reasons we don't use the kernel's cache, we have our own cache implementation that's in the same address space as the rest of the process.