7 ms·

No, it's a lot messier than C. In a regular C program (not using any special libraries for M:N threading) you wouldn't have to spawn an entirely separate proces

No, it's a lot messier than C. In a regular C program (not using any special libraries for M:N threading) you wouldn't have to spawn an entirely separate process. This issue is one of the downsides of Go's M:N scheduling. The OS is simply not aware of what the Go runtime is doing, and as a result you get impedance mismatches like this. It "raises a few eyebrows" because M:N scheduling is unpopular outside of Go and Erlang. It was tried early on in the Linux world and abandoned precisely because of issues like this. Go has repopularized M:N lately, and it's proof that such a system can work for lots of apps, but the downsides of that decision are every bit as real today as they were in the early days of NPTL.

SwellJoe 9y ago
Historically, much (I would probably argue most) C concurrency has been implemented with fork. Certainly not all, and there are many ways to handle it in C...but fork is really common, and I don't think it's considered all that big of a deal to do so. It is idiomatic (at least historically and across maybe billions of lines of C code), and not much harder to reason about than many kinds of thread implementation in C; in fact, it's easier to reason about fork than something like POSIX threads, IMHO. But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency (because you can determine when things will happen with precision in C)? I don't know. Again, this is beyond my understanding of Go right now. I may just not be understanding the implications of this. The code to deal with it looks reasonable enough to me; it's a smallish function, easily isolated. I think the author did a great job explaining the problem, the troubleshooting, and the solution. I just didn't see the problem as being all that damning of Go...but, that may be a reflection of my shallow understanding of the problem, or of the implications of the cost spawning a new process (to me, I always think back to the old adage "fork is cheap on Linux").
- cyphar 9y ago
  > C concurrency has been implemented with fork. Arguably it has been implemented with clone(2) which has flags. > in fact, it's easier to reason about fork than something like POSIX threads, IMHO. Not if you call fork() in a multi-threaded program. That ends _exceptionally_ badly (let's just say there's a reason Go doesn't expose syscall.Fork and it has to do with horrific deadlocks). > I just didn't see the problem as being all that damning of Go. The problem is more subtle, and it comes down to maintainability and understandably. If you ever decide to read the runc codebase, I apologise. One of the reasons the codebase is so scattered is because of these sorts of hacks where you have to work around issues in the Go runtime (because it doesn't give you enough control). In the article, whenever you read the small function you have to keep in mind that it's actually spawning a subprocess (which then means you have to think about what namespaces had the process joined and so on). Go is an okay language, but it simply wasn't designed for stuff this low-level. We would be much better served with Rust in my opinion. > But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency In C you don't need to fork to switch namespaces, you just call setns(...). For the PID namespace you need to fork, but that's just a quirk of the interface. In Go, you theoretically don't need to fork either (syscall.Setns is available). However, there is no real way to safely use it. First of all, the namespace interfaces in Linux are quite fragile when it comes to multi-threaded processes, but combine that with a runtime that will switch you between OS threads at random. And while the documentation on runtime.GOMAXPROC and runtime.LockOSThread might trick you into believing it's possible to stop the Go runtime from doing a clone(CLONE_THREAD|CLONE_PARENT), you can't.
  - jankedeen 9y ago
    No. You can call fork pretty cleanly in a pthread based program. This post is not informed criticism. There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Namespace based code while doing the same is like setting your hair on fire while running through the gasoline forest..but caveat emptor is usually spelled out pretty critically in the API and docs. Oh, btw: Fuck RUST. For once and for all fuck this tyranny of coercion by potential IP holders. It is a shit language.
    - klodolph 9y ago
      > There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Actually, that's not true. Perhaps it's theoretically possible, in some cases, to make your threaded code work with fork with the appropriate pthread_atfork() handlers, but in general, it's a total mess. Thread A has a lock, thread B calls fork(), thread B tries to acquire the lock. The lock is held, but thread A is not running in the child. That much should be obvious. The usual fix is to write a pthread_atfork() handler for every lock in your program. Before forking, it locks all of your mutexes. After forking, in both the child and parent, it unlocks them. You can see the holes in this. One simple hole is that fork() is async signal safe but pthread_mutex_lock() is not, but most people won't fork() in a signal anyway. The big problem is that you now need to make all of your locks recursive and figure out, globally, what order to lock them in. This means registering your atfork() handlers in exactly the right order, something which is more difficult than it sounds. Another problem is the performance implications of acquiring all the locks in your program. Namespaces + threads are fine, the only problem here is namespaces + M:N threads + no method for pinning a green thread to an OS thread, like forkOS in Haskell.
      - jankedeen 9y ago
        *...it's a total mess...theoretically it's possible....speculation on standard behavior based on platform behavior...namespace(x)<-threads->chosen implementation detail...functional language reference... -- Translation: Design is broken
- dullgiulio 9y ago
  I mostly agree. Of course, for as cheap as a fork can be (and in Linux it's very cheap), it's not as cheap as a systemcall. When changing namespace happens often in a hot path, forking might not be fast enough. I disagree with Walton that the blame is on the N:M threading. It's really on the Linux kernel that has never made a clean distinction between threads and processes, both in kernel and in the APIs.
  - YZF 9y ago
    I agree. Bestowing certain permissions/properties on a thread doesn't make sense. The thread shouldn't "enter the namespace", an API should retrieve a handle to that namespace that should then be usable from any thread in the process via some appropriate calls.
    - the8472 9y ago
      On the other hand having individual threads entering namespaces means your broker process does not have to constantly switch namespaces, it can use shared memory between differently privileged threads to do the brokering.
      - YZF 9y ago
        I know nothing of the implementation here but presumably these namespaces can exist side by side in a way that doesn't require any "switching". If switching is expensive that would make context switching between a thread in one namespace and a thread in another namespace just as expensive? If you have one thread in one namespace and another in another you now have to worry about what you can do in the context of a callback. This asymmetry just makes any multi-threaded program more complicated than it needs to be (and already is).
        the8472 9y ago
        Switching is a system call (setns), in principle shared-memory IPC does not involve context switches, just lock-free data structures. I'm not sure how common this is in practice since shared memory also has some downsides if you're doing this for security. But there also are non-security applications of namespaces. And it's not like namespaces are the only per-thread thing in linux. Capabilities, uid and signal handlers come to mind.
- aidenn0 9y ago
  Separate threads for IO was really common in C and C++ in the 90s. In addition fork() style concurrency was essentially nonexistent outside of Unix.
- pyre 9y ago
  > But, are you saying that to switch to a namespace in Go one must fork The article is stating that to control the namespace that the threads execute in, that a separate process must be spawned so that the entire process can be forced into the correct namespace. Otherwise, the runtime can spawn new threads as it sees fit and you don't have control over which namespace they are in. It might be possible to work around this from within the same process if it were possible to force the runtime to not spawn new threads in particular cases. If you could control when the runtime was allowed to spawn new threads, then you could organize the program / threads in a way that would keep the correct code operating in the correct namespace. Unfortunately, you can't. Disclaimer: I don't know Go, but this is my understanding from reading the article.
m0th87 9y ago
This makes sense, and I know it's why Rust abandoned green threading. But I can't help but worry that the focus on async I/O in Rust as a way of avoiding this issue is going to bring the language down a path that isn't as ergonomically pleasant as Go or Erlang's M:N threading for things like highly concurrent web services. Do you share this concern, or do you think Rust can achieve the same level of ergonomics without the impedance mismatch?
- geodel 9y ago
  You can see for yourself [1]. Seems many users are not following the patterns of Async IO in Rust. 1. https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust_need_today_for_server_workloads/ https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust...
  - pcwalton 9y ago
    1:1 can be made compatible with async I/O with a thread pool, though.
- steveklabnik 9y ago
  There's a prototype of async/await that can help a lot with that.
- pcwalton 9y ago
  It's harder for us, but I think we can get to an ergonomic solution with async/await. It's not as easy as threads (M:N vs 1:1 is a red herring as far as this is concerned), but there's no free lunch.
  - pencilhappen 9y ago
    async/await still has the fundamental problem of composability that all the other attempts at sugaring around an event loop have (aka the "functions have colors" problem [1]). It sucks for collaboration, which is one of the most important things for modern software development. 1. http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
_ph_ 9y ago
As far as I know, Solaris threads are actually M:N threads on the system level. M:N threads are extremely efficient. A "native" thread is a heavyweight construct - e.g. by the stack space allocation. So this puts a limit on how many threads you can spawn in a program. For many tasks, this would force you to roll out your own M:N mapping system yourself. Doing it on the language level usually yields the better results. Due to the implicit M:N mapping, Go goroutines are extremely cheap. This allows you to spawn as many, as your algorithm naturally requires. The Go runtime will automatically map them to native threads - typically one per avaliable CPU. As a consequence, a Go program that heavily uses goroutines has a pretty clean code and scales without much overhead across a large variation of number of CPU cores.
- dap 9y ago
  > As far as I know, Solaris threads are actually M:N threads on the system level. Not for a long time, for all the reasons the parent mentioned (and then some).
- pjmlp 9y ago
  Only until Solaris 8, Solaris 9 dropped support for the LWP model.
di4na 9y ago
Just saying, but Erlang do not target that type of "System Programming" and the answer to that problem in Erlang would probably work through totally different way to do it. This namespace thing would not be a problem. This is not a problem of M:N. This is a problem of Go being badly designed. Not new.
- dozzie 9y ago
  More like reification of the very old problem from the times when we got threading API, with thread safety of functions that alter process-wide state (umask(), chdir(), sleep()), except now it's calls that alter some thread-specific thing.
- wbl 9y ago
  It's a problem with Linux not permitting inspection of interfaces through a namespace. If I have a chroot I can look in from outside easily. Apparently not so with container interfaces.
pjmlp 9y ago
Also in Solaris, it was tried and abandoned. Although they have their own set of issues, Windows fibers are also barely used.

SwellJoe 9y ago

Historically, much (I would probably argue most) C concurrency has been implemented with fork. Certainly not all, and there are many ways to handle it in C...but fork is really common, and I don't think it's considered all that big of a deal to do so. It is idiomatic (at least historically and across maybe billions of lines of C code), and not much harder to reason about than many kinds of thread implementation in C; in fact, it's easier to reason about fork than something like POSIX threads, IMHO. But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency (because you can determine when things will happen with precision in C)? I don't know. Again, this is beyond my understanding of Go right now. I may just not be understanding the implications of this. The code to deal with it looks reasonable enough to me; it's a smallish function, easily isolated. I think the author did a great job explaining the problem, the troubleshooting, and the solution. I just didn't see the problem as being all that damning of Go...but, that may be a reflection of my shallow understanding of the problem, or of the implications of the cost spawning a new process (to me, I always think back to the old adage "fork is cheap on Linux").

cyphar 9y ago
> C concurrency has been implemented with fork. Arguably it has been implemented with clone(2) which has flags. > in fact, it's easier to reason about fork than something like POSIX threads, IMHO. Not if you call fork() in a multi-threaded program. That ends _exceptionally_ badly (let's just say there's a reason Go doesn't expose syscall.Fork and it has to do with horrific deadlocks). > I just didn't see the problem as being all that damning of Go. The problem is more subtle, and it comes down to maintainability and understandably. If you ever decide to read the runc codebase, I apologise. One of the reasons the codebase is so scattered is because of these sorts of hacks where you have to work around issues in the Go runtime (because it doesn't give you enough control). In the article, whenever you read the small function you have to keep in mind that it's actually spawning a subprocess (which then means you have to think about what namespaces had the process joined and so on). Go is an okay language, but it simply wasn't designed for stuff this low-level. We would be much better served with Rust in my opinion. > But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency In C you don't need to fork to switch namespaces, you just call setns(...). For the PID namespace you need to fork, but that's just a quirk of the interface. In Go, you theoretically don't need to fork either (syscall.Setns is available). However, there is no real way to safely use it. First of all, the namespace interfaces in Linux are quite fragile when it comes to multi-threaded processes, but combine that with a runtime that will switch you between OS threads at random. And while the documentation on runtime.GOMAXPROC and runtime.LockOSThread might trick you into believing it's possible to stop the Go runtime from doing a clone(CLONE_THREAD|CLONE_PARENT), you can't.
- jankedeen 9y ago
  No. You can call fork pretty cleanly in a pthread based program. This post is not informed criticism. There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Namespace based code while doing the same is like setting your hair on fire while running through the gasoline forest..but caveat emptor is usually spelled out pretty critically in the API and docs. Oh, btw: Fuck RUST. For once and for all fuck this tyranny of coercion by potential IP holders. It is a shit language.
  - klodolph 9y ago
    > There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Actually, that's not true. Perhaps it's theoretically possible, in some cases, to make your threaded code work with fork with the appropriate pthread_atfork() handlers, but in general, it's a total mess. Thread A has a lock, thread B calls fork(), thread B tries to acquire the lock. The lock is held, but thread A is not running in the child. That much should be obvious. The usual fix is to write a pthread_atfork() handler for every lock in your program. Before forking, it locks all of your mutexes. After forking, in both the child and parent, it unlocks them. You can see the holes in this. One simple hole is that fork() is async signal safe but pthread_mutex_lock() is not, but most people won't fork() in a signal anyway. The big problem is that you now need to make all of your locks recursive and figure out, globally, what order to lock them in. This means registering your atfork() handlers in exactly the right order, something which is more difficult than it sounds. Another problem is the performance implications of acquiring all the locks in your program. Namespaces + threads are fine, the only problem here is namespaces + M:N threads + no method for pinning a green thread to an OS thread, like forkOS in Haskell.
    - jankedeen 9y ago
      *...it's a total mess...theoretically it's possible....speculation on standard behavior based on platform behavior...namespace(x)<-threads->chosen implementation detail...functional language reference... -- Translation: Design is broken
dullgiulio 9y ago
I mostly agree. Of course, for as cheap as a fork can be (and in Linux it's very cheap), it's not as cheap as a systemcall. When changing namespace happens often in a hot path, forking might not be fast enough. I disagree with Walton that the blame is on the N:M threading. It's really on the Linux kernel that has never made a clean distinction between threads and processes, both in kernel and in the APIs.
- YZF 9y ago
  I agree. Bestowing certain permissions/properties on a thread doesn't make sense. The thread shouldn't "enter the namespace", an API should retrieve a handle to that namespace that should then be usable from any thread in the process via some appropriate calls.
  - the8472 9y ago
    On the other hand having individual threads entering namespaces means your broker process does not have to constantly switch namespaces, it can use shared memory between differently privileged threads to do the brokering.
    - YZF 9y ago
      I know nothing of the implementation here but presumably these namespaces can exist side by side in a way that doesn't require any "switching". If switching is expensive that would make context switching between a thread in one namespace and a thread in another namespace just as expensive? If you have one thread in one namespace and another in another you now have to worry about what you can do in the context of a callback. This asymmetry just makes any multi-threaded program more complicated than it needs to be (and already is).
      - the8472 9y ago
        Switching is a system call (setns), in principle shared-memory IPC does not involve context switches, just lock-free data structures. I'm not sure how common this is in practice since shared memory also has some downsides if you're doing this for security. But there also are non-security applications of namespaces. And it's not like namespaces are the only per-thread thing in linux. Capabilities, uid and signal handlers come to mind.
aidenn0 9y ago
Separate threads for IO was really common in C and C++ in the 90s. In addition fork() style concurrency was essentially nonexistent outside of Unix.
pyre 9y ago
> But, are you saying that to switch to a namespace in Go one must fork The article is stating that to control the namespace that the threads execute in, that a separate process must be spawned so that the entire process can be forced into the correct namespace. Otherwise, the runtime can spawn new threads as it sees fit and you don't have control over which namespace they are in. It might be possible to work around this from within the same process if it were possible to force the runtime to not spawn new threads in particular cases. If you could control when the runtime was allowed to spawn new threads, then you could organize the program / threads in a way that would keep the correct code operating in the correct namespace. Unfortunately, you can't. Disclaimer: I don't know Go, but this is my understanding from reading the article.

m0th87 9y ago

This makes sense, and I know it's why Rust abandoned green threading. But I can't help but worry that the focus on async I/O in Rust as a way of avoiding this issue is going to bring the language down a path that isn't as ergonomically pleasant as Go or Erlang's M:N threading for things like highly concurrent web services. Do you share this concern, or do you think Rust can achieve the same level of ergonomics without the impedance mismatch?

geodel 9y ago
You can see for yourself [1]. Seems many users are not following the patterns of Async IO in Rust. 1. https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust_need_today_for_server_workloads/ https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust...
- pcwalton 9y ago
  1:1 can be made compatible with async I/O with a thread pool, though.
steveklabnik 9y ago
There's a prototype of async/await that can help a lot with that.
pcwalton 9y ago
It's harder for us, but I think we can get to an ergonomic solution with async/await. It's not as easy as threads (M:N vs 1:1 is a red herring as far as this is concerned), but there's no free lunch.
- pencilhappen 9y ago
  async/await still has the fundamental problem of composability that all the other attempts at sugaring around an event loop have (aka the "functions have colors" problem [1]). It sucks for collaboration, which is one of the most important things for modern software development. 1. http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

_ph_ 9y ago

As far as I know, Solaris threads are actually M:N threads on the system level. M:N threads are extremely efficient. A "native" thread is a heavyweight construct - e.g. by the stack space allocation. So this puts a limit on how many threads you can spawn in a program. For many tasks, this would force you to roll out your own M:N mapping system yourself. Doing it on the language level usually yields the better results. Due to the implicit M:N mapping, Go goroutines are extremely cheap. This allows you to spawn as many, as your algorithm naturally requires. The Go runtime will automatically map them to native threads - typically one per avaliable CPU. As a consequence, a Go program that heavily uses goroutines has a pretty clean code and scales without much overhead across a large variation of number of CPU cores.

dap 9y ago
> As far as I know, Solaris threads are actually M:N threads on the system level. Not for a long time, for all the reasons the parent mentioned (and then some).
pjmlp 9y ago
Only until Solaris 8, Solaris 9 dropped support for the LWP model.

di4na 9y ago

Just saying, but Erlang do not target that type of "System Programming" and the answer to that problem in Erlang would probably work through totally different way to do it. This namespace thing would not be a problem. This is not a problem of M:N. This is a problem of Go being badly designed. Not new.

dozzie 9y ago
More like reification of the very old problem from the times when we got threading API, with thread safety of functions that alter process-wide state (umask(), chdir(), sleep()), except now it's calls that alter some thread-specific thing.
wbl 9y ago
It's a problem with Linux not permitting inspection of interfaces through a namespace. If I have a chroot I can look in from outside easily. Apparently not so with container interfaces.

pjmlp 9y ago

Also in Solaris, it was tried and abandoned. Although they have their own set of issues, Windows fibers are also barely used.