14 ms·

Linux Namespaces and Go Don't Mix

266 points by _lm_ 9y ago

SwellJoe 9y ago
This leads to go code being roughly as messy/clumsy as C (or whatever else) code in the sections that need concurrency and also need to change namespace. That's unfortunate, but I don't know that it really "raises a few eyebrows". I mean, what's the better alternative to Go for this work? Maybe Rust? It is, at least more controllable at a lower level...but, not as easy to pick up for people coming from a C/Python/Perl/Ruby systems and ops background. I'm not saying one should use Go for containers/namespaces programming, but a lot of people are with some success (probably also banging into the namespaces issue now and then), I'm just saying it's not obvious to me what the better alternative would be.
- pcwalton 9y ago
  No, it's a lot messier than C. In a regular C program (not using any special libraries for M:N threading) you wouldn't have to spawn an entirely separate process. This issue is one of the downsides of Go's M:N scheduling. The OS is simply not aware of what the Go runtime is doing, and as a result you get impedance mismatches like this. It "raises a few eyebrows" because M:N scheduling is unpopular outside of Go and Erlang. It was tried early on in the Linux world and abandoned precisely because of issues like this. Go has repopularized M:N lately, and it's proof that such a system can work for lots of apps, but the downsides of that decision are every bit as real today as they were in the early days of NPTL.
  - SwellJoe 9y ago
    Historically, much (I would probably argue most) C concurrency has been implemented with fork. Certainly not all, and there are many ways to handle it in C...but fork is really common, and I don't think it's considered all that big of a deal to do so. It is idiomatic (at least historically and across maybe billions of lines of C code), and not much harder to reason about than many kinds of thread implementation in C; in fact, it's easier to reason about fork than something like POSIX threads, IMHO. But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency (because you can determine when things will happen with precision in C)? I don't know. Again, this is beyond my understanding of Go right now. I may just not be understanding the implications of this. The code to deal with it looks reasonable enough to me; it's a smallish function, easily isolated. I think the author did a great job explaining the problem, the troubleshooting, and the solution. I just didn't see the problem as being all that damning of Go...but, that may be a reflection of my shallow understanding of the problem, or of the implications of the cost spawning a new process (to me, I always think back to the old adage "fork is cheap on Linux").
    - cyphar 9y ago
      > C concurrency has been implemented with fork. Arguably it has been implemented with clone(2) which has flags. > in fact, it's easier to reason about fork than something like POSIX threads, IMHO. Not if you call fork() in a multi-threaded program. That ends _exceptionally_ badly (let's just say there's a reason Go doesn't expose syscall.Fork and it has to do with horrific deadlocks). > I just didn't see the problem as being all that damning of Go. The problem is more subtle, and it comes down to maintainability and understandably. If you ever decide to read the runc codebase, I apologise. One of the reasons the codebase is so scattered is because of these sorts of hacks where you have to work around issues in the Go runtime (because it doesn't give you enough control). In the article, whenever you read the small function you have to keep in mind that it's actually spawning a subprocess (which then means you have to think about what namespaces had the process joined and so on). Go is an okay language, but it simply wasn't designed for stuff this low-level. We would be much better served with Rust in my opinion. > But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency In C you don't need to fork to switch namespaces, you just call setns(...). For the PID namespace you need to fork, but that's just a quirk of the interface. In Go, you theoretically don't need to fork either (syscall.Setns is available). However, there is no real way to safely use it. First of all, the namespace interfaces in Linux are quite fragile when it comes to multi-threaded processes, but combine that with a runtime that will switch you between OS threads at random. And while the documentation on runtime.GOMAXPROC and runtime.LockOSThread might trick you into believing it's possible to stop the Go runtime from doing a clone(CLONE_THREAD|CLONE_PARENT), you can't.
      - jankedeen 9y ago
        No. You can call fork pretty cleanly in a pthread based program. This post is not informed criticism. There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Namespace based code while doing the same is like setting your hair on fire while running through the gasoline forest..but caveat emptor is usually spelled out pretty critically in the API and docs. Oh, btw: Fuck RUST. For once and for all fuck this tyranny of coercion by potential IP holders. It is a shit language.
  - m0th87 9y ago
    This makes sense, and I know it's why Rust abandoned green threading. But I can't help but worry that the focus on async I/O in Rust as a way of avoiding this issue is going to bring the language down a path that isn't as ergonomically pleasant as Go or Erlang's M:N threading for things like highly concurrent web services. Do you share this concern, or do you think Rust can achieve the same level of ergonomics without the impedance mismatch?
    - geodel 9y ago
      You can see for yourself [1]. Seems many users are not following the patterns of Async IO in Rust. 1. https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust_need_today_for_server_workloads/ https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust...
      - pcwalton 9y ago
        1:1 can be made compatible with async I/O with a thread pool, though.
    - steveklabnik 9y ago
      There's a prototype of async/await that can help a lot with that.
    - pcwalton 9y ago
      It's harder for us, but I think we can get to an ergonomic solution with async/await. It's not as easy as threads (M:N vs 1:1 is a red herring as far as this is concerned), but there's no free lunch.
      - pencilhappen 9y ago
        async/await still has the fundamental problem of composability that all the other attempts at sugaring around an event loop have (aka the "functions have colors" problem [1]). It sucks for collaboration, which is one of the most important things for modern software development. 1. http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
  - _ph_ 9y ago
    As far as I know, Solaris threads are actually M:N threads on the system level. M:N threads are extremely efficient. A "native" thread is a heavyweight construct - e.g. by the stack space allocation. So this puts a limit on how many threads you can spawn in a program. For many tasks, this would force you to roll out your own M:N mapping system yourself. Doing it on the language level usually yields the better results. Due to the implicit M:N mapping, Go goroutines are extremely cheap. This allows you to spawn as many, as your algorithm naturally requires. The Go runtime will automatically map them to native threads - typically one per avaliable CPU. As a consequence, a Go program that heavily uses goroutines has a pretty clean code and scales without much overhead across a large variation of number of CPU cores.
    - dap 9y ago
      > As far as I know, Solaris threads are actually M:N threads on the system level. Not for a long time, for all the reasons the parent mentioned (and then some).
    - pjmlp 9y ago
      Only until Solaris 8, Solaris 9 dropped support for the LWP model.
  - di4na 9y ago
    Just saying, but Erlang do not target that type of "System Programming" and the answer to that problem in Erlang would probably work through totally different way to do it. This namespace thing would not be a problem. This is not a problem of M:N. This is a problem of Go being badly designed. Not new.
    - dozzie 9y ago
      More like reification of the very old problem from the times when we got threading API, with thread safety of functions that alter process-wide state (umask(), chdir(), sleep()), except now it's calls that alter some thread-specific thing.
    - wbl 9y ago
      It's a problem with Linux not permitting inspection of interfaces through a namespace. If I have a chroot I can look in from outside easily. Apparently not so with container interfaces.
  - pjmlp 9y ago
    Also in Solaris, it was tried and abandoned. Although they have their own set of issues, Windows fibers are also barely used.
- scriptkiddy 9y ago
  > I mean, what's the better alternative to Go for this work? Maybe Rust? It is, at least more controllable at a lower level...but, not as easy to pick up for people coming from a C/Python/Perl/Ruby systems and ops background. You know, it's interesting. I've been programming with Python for about 6 years now. I've also picked up Javascript, SQL, bash, and PHP along the way. I'm always gaining a little bit of C knowledge here and there when writing C extensions for my Python applications. I'm a fairly experienced programmer at this point. To the point: I tried picking up Go one day because I was hearing so much about how it could replace Python as network glue code with better performance and reliable concurrency. I can't really validate or invalidate those claims. That said, I found Go to be sort of difficult. The syntax is really simple. Compiling is really simple. Concurrency is even simple. However, need to do something in a different way than Go decides is correct? Well, you can't. It won't even compile. The difficulty in Go is in learning about what the compiler thinks is OK. I don't really like that. You don't really know if your code will work until you compile. Basically, I just think Go isn't really flexible enough for modern programming. I find that Nim can do Go's job better than Go can for my use cases anyways.
  - literallycancer 9y ago
    >You don't really know if your code will work until you compile. You are supposed to "compile on save".
    - scriptkiddy 9y ago
      I usually do this, but sometimes I'm writing a really long function or something and I want to save part way through so I don't lose my progress from some unlikely, yet catastrophic, failure.
  - zellyn 9y ago
    > You don't really know if your code will work until you compile. This is true in every text-based programming language. s/compile/execute/ for interpreted or repl-based languages.
    - 9y ago
- ahoka 9y ago
  Better alternative to C with the same low level control? Why not C++?
  - tankenmate 9y ago
    Because C++ is vastly more complex than either C or Go.
    - rkeene2 9y ago
      To go along with this, due to overloading and inheritance rules you cannot look at a small section of C++ code and determine what it does without reading up to the entire program.
      - ahoka 9y ago
        I think that holds for any program written in any language. If the abstractions are confusing then it will be hard to read, be it layers of C macros, over-use of template programming or bad class hierarchy. BTW C++'s standard library is mostly functional (having roots in scheme), which makes it pretty easy to reason about.
    - theamk 9y ago
      In C, I could not find good libraries for basic stuff (dynamic string manipulation, variable-sized arrays, hash tables). I tried glib but it was much worse than C++'s STL. It is entirely possible to write C-style programs in C++ (very few classes, globals all over the place) and so on, and at least for me, C-style software in C++ are much safer/easier to debug than C-style software in C .
      - simias 9y ago
        In my experience the C mindset is to write those yourself. The lack of templates means that it's very difficult to write proper generic containers. You either end up with macro soup, void * soup or code tailored to your needs. Or a lot of the time a combination of all three. But I'm like you, if I know that I'm going to need "advanced" data structures I'll pick C++ over C (or these days more likely Rust).
- cyphar 9y ago
  Rust is much better for this (though I still feel some of the fork/exec interface has similar warts to Go). However, you're wrong in saying that it's "roughly as messy/clumsy" as C. Let me tell you how runc works. runc is written in Go, and we take an OCI configuration file. Because we can't just fork and set up all of the namespaces in Go, we have a C function called nsexec which is specified as __attribute__((constructor)). This ensures that our code will execute before the Go runtime boots. The parent process writes (using netlink as the wire protocol) to a pipe that the child has open and is parsed in C. Then, the child will have to do a series of forks, unshare, setns, {open,read,write} and so on (and the final PID needs to be sent back to the original parent) in order to set up and join all of the necessary namespaces. In C, this code would be _immensely_ easier to read, write and maintain. Just look at LXC. Personally I really wish people had just gone with Rust earlier on rather than implementing everything in Go. I've had nothing but pain from Go.
  - SwellJoe 9y ago
    Thanks for explaining how it's done in runc. That does sound pretty awful. So, even though the initialization can be outsourced to a C function, you still would prefer to be working entirely in C? Are there no advantages to Go for runc? And, would it be possible for someone to write a somewhat standardized Go library for doing this grunt work? Is it merely fear of C that keeps so much of the container infrastructure on Go? I've only spent a couple of weeks looking peripherally at Go, and I already like it better than C (which I've poked at peripherally for ~25 years), but I don't know it well enough to know its warts.
    - cyphar 9y ago
      > you still would prefer to be working entirely in C? No, but there are more options than just Go and C. Rust is an option that I'm shilling at the moment (though I've only started learning it, so take that with a grain of salt). The main reason I would want to write it in C is because there's a lot of string parsing code you have to write in order to make container runtimes work -- and as we all know that's probably the #1 source of security vulnerabilities. > Are there no advantages to Go for runc? There are, mainly due to network effect (everything else is written in Go) and getting contributions from the community (Go is easy to pick up). Unfortunately there are also disadvantages, and quite a few of those disadvantages are present in Go but are not present in other memory-safe and low-level languages (Rust is a good example because to be quite honest it's the only player in this space that doesn't try to do more than necessary). Go is a good language for it's designed for (Web servers and similar things), but from my experience it's not the best choice for low-level tasks. We've seen cases where long-running container daemons (not naming any names) will crash if you run more than 1000 containers on a single system. They don't crash because of the actual daemon code but because of issues with Go's GC (it doesn't actually free memory sanely, it uses MADV_DONTNEED which inflates RSS and causes OOM to kill your daemon). > And, would it be possible for someone to write a somewhat standardized Go library for doing this grunt work? Of course (and you could argue that we have done that in runc with github.com/opencontainers/runc/libcontainer/nsenter), but the thing to note is that in order to get around problems in the Go runtime you have to split out a single piece of code into separate processes and have to now redesign how a single function would work. So moving it to a separate library means that development is even more frustrating (you've created an API around the internal implementation of whatever thing you're working on). > Is it merely fear of C that keeps so much of the container infrastructure on Go? I think the network effect is the main reason. Most of the people I've worked with know quite a lot of C (we do kernel work sometimes) so writing a runtime in C would be frustrating but entirely doable. The problem is that you couldn't just import it into a Go project (and people don't like cgo because it makes binaries harder to build in certain cases). > I've only spent a couple of weeks looking peripherally at Go, and I already like it better than C Go definitely has it's uses, and I still use it for new projects. For example I recently wrote a tool for dealing with OCI container images in Go[1]. The standard library of Go is quite nice (though I found some bugs in archive/tar but let's not go there) and I'm always quite amazed just how much you can do before you have to import external libraries. But I recently started learning Rust and I am _really_ enjoying being able to understand what my program will actually look like when compiled. If you've ever had to strace a Go program, you'll know exactly what I mean. Debugging Go programs is basically fucking impossible. [1]: https://github.com/openSUSE/umoci https://github.com/openSUSE/umoci
- vbernat 9y ago
  In Linux, many properties are task-related and therefore the API is also task-related. Namespaces are not the only problem. See for example https://github.com/golang/go/issues/1435 https://github.com/golang/go/issues/1435. The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. Most other languages would work just fine (C or Python for example).
  - lunixbochs 9y ago
    > The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. runtime.LockOSThread() does exactly that [1]. [1] https://golang.org/pkg/runtime/#LockOSThread https://golang.org/pkg/runtime/#LockOSThread
    - vbernat 9y ago
      No, it doesn't. Go runtime can decide to spawn a new thread from the locked one. The new thread will inherit the characteristics (namespace, uid) of the old one. See https://github.com/docker/libnetwork/issues/1113 https://github.com/docker/libnetwork/issues/1113 for more details.
      - lunixbochs 9y ago
        > Go runtime can decide to spawn a new thread from the locked one That sounds like an implementation issue, why not assume the documentation is the intended behavior and this side effect is a bug? I'd support a CL to fix this behavior or add a block-clone-from-here runtime call (but the end result of that is you want the thread to exit when you're done with it, and not to go back into the pool... which is also new behavior). At the minimum, this behavior of new threads spawning from LockOSThread could be documented. As a workaround, what about CGO -> pthreads -> spawn a control thread free from the Go scheduler -> call back into Golang to run a control loop function? You can do this in init() to ensure it has full control over itself. Or will Golang call clone() from unscheduled code?
        9y ago
- mappu 9y ago
  You can keep using Go. Its behaviour here is actually usually what you want. To avoid the authors' issue, you can write some functions in C. CGo has a very high level of integration (you can mix languages in the same source file) and would be quite simple for the case of a setns/execve wrapper.
  - TheDong 9y ago
    > CGo has a very high level of integration No, no it doesn't. CGo is slow as balls to call into and return from. Types can't fully be shared. It's interacted with via comments. You can't use cgo to control the threading of the Go runtime itself, which is the real problem here. The behavior you want is to be able to call linux syscalls that operate on threads and not be utterly fucked. That behavior cannot be accomplished with go nor go+cgo easily. Threads in cgo are also kinda fucked.
    - mappu 9y ago
      > You can't use cgo to control the threading of the Go runtime itself, which is the real problem here. Go is deliberately opinionated about threading of the runtime. I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS), given the philosophy around e.g. GC tuning. Cgo is a beefed-up runtime.LockOSThread() that could be used to avoid having the author resort to a helper process. > CGo is slow as balls to call into and return from. They're slower than Go function calls, but they still take only nanoseconds. This is negligible on the authors' scale of "launching an entire container". For their case of "it is not possible to guarantee that a new OS process ... will run in a given namespace", you're not even returning from Cgo after exec. > It's interacted with via comments. Do you use build tags? Or go:generate? Like it or not, it's idiomatic. I'm with you on the types, though! `go tool cgo -godefs` helps, but it would be great to see improvements especially in the reverse case of exporting Go buildmode=c-shared for C consumption. Still, a little marshaling seems tidier than a whole helper process.
      - cyphar 9y ago
        > I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS) GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads.
- cat199 9y ago
  > I mean, what's the better alternative to Go for this work? Separate processes, like the post suggests? I have no idea why someone would expect user-level pseudothreads to execute across system-level primitive boundaries.. seems fairly obvious to me. I don't expect a chrooted daemon (e.g. apache, etc) to have access to parent thread contexts.. etc. Fork & pipe IPC shouldn't be too difficult for anyone to understand, beyond that, if you don't understand these things, you probably shouldn't be writing code that complex..
vishvananda 9y ago
Author of the go netlink library here. I've run into this issue a number of times. There has been conversation in the past about adding some kind of new runtime command like LockOSThread to prevent new threads from being spawned, but it didn't gain any momentum. Even though I am a big fan of go, I've personally built two container runtimes in other languages do to the namespace clumsiness. Personally, I think rust is an excellent alternative for namespace utilities. EDIT: there is more information and links in the issue in the netns library: https://github.com/vishvananda/netns/issues/17 https://github.com/vishvananda/netns/issues/17
- cyphar 9y ago
  I completely agree, and I've been coming up with some ideas (based on runc) to see what sort of improvements can you have if you do a full redesign in a language that isn't as painful as Go to use. At the moment I'm incredibly busy, but later this year I might be able to start working on that too.
- bogomipz 9y ago
  >" I've personally built two container runtimes in other languages do to the namespace clumsiness." Can you share any details on those other container runtimes?
  - vishvananda 9y ago
    One was a simplified runtime in c, similar to a stripped down version of systemd-nspawn. I can hopefully share the other one in a few weeks. I'm going through an open source approval process for it.
    - bogomipz 9y ago
      Thanks. It's an interesting topic. I hope you post it on HN then.
- mrjana 9y ago
  One way to alleviate the problem at least for netlink library is to create a function which calls runtimeLockOSThread, sets into the required namespace and then opens a netlink socket using only raw syscall apis. One has to be careful in this code path to not invoke go runtime (i.e both socket and setns should be raw syscall apis) and not even trigger any allocations so that go runtime doesn't get a chance a spin a new OS thread. Once a socket is created in the required namespace you can get back to the caller namespace and return the socket fd. Now this socket fd is bound to that namespace and all netlink operations on that socket will happen in the target namespace. Disclaimer: I am one of the original libnetwork authors and we have been aware of this issue with go for some time now.
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
liveoneggs 9y ago
this is just a more complex version of the issue where go can't safely do the daemonize dance for a privileged port
- brandon 9y ago
  There's a simple-ish workaround for the privileged port issue if you can't just use CAP_NET_ADMIN: http://play.golang.org/p/dXBizm4xl3 http://play.golang.org/p/dXBizm4xl3 The namespace issues are unfortunately a lot tougher to address.
  - vbernat 9y ago
    It's only reliable if none of your dependencies are spawning goroutines during initialization. If this is the case, some goroutines (yours' or the dependencies') can end up with increased privileges.
    - brandon 9y ago
      The workaround functions by opening the privileged socket and re-executing the binary as an unprivileged user with access to the filehandle. Any background goroutines would exit with the privileged parent. Also: please don't spawn goroutines during init(). There's generally a better time and place, and in the event that you justifiably need a package-level background routine you can spawn it on demand with a sync.Once.
  - the8472 9y ago
    But you also want drop that privilege after using it to get the same effect as the "daemonize dance" GP referred to. And that's also per-thread, just like namespaces.
- cyphar 9y ago
  And that's unfortunately only an issue because Linux doesn't implement POSIX's model for thread privileges. In particular, glibc has to implement the nptl in userspace to make Linux threads look like POSIX threads. I get the feeling that the Go runtime doesn't do the same.
ghthor 9y ago
This is discouraging considering go was initially designed as a systems programming language. I wonder if there is another way for go to handle blocking syscall such that this use case would become reliable.
- pritambaral 9y ago
  I think Go was designed as a server programming language more than a systems programming language.
  - stouset 9y ago
    Go was retconned into this role, but originally it was marketed as a systems language.
jankedeen 9y ago
I read this crap in language design and despair. This is why C is my only recourse for systems programming in *nix environments.
- Safety1stClyde 9y ago
  This is what I cannot understand about Go: how did a newly-designed programming language end up as such a muddle?
  - YZF 9y ago
    Because this isn't really a language issue. It's an OS issue. Go assumes that all threads are equal and therefore any Goroutine can run on any thread and threads are all equal. Linux got late to the thread party and its threads are kind of like processes that haven't decided if they want to be processes or threads. It's true the Go runtime could give users more control over goroutine and thread scheduling but that would kind of defeat the purpose of not needing to know about it and having Goroutines as the only flexible unit of concurrency. I think the kludge here is on the Linux side. Having some magic properties bestowed upon threads doesn't make sense. The property should be accessible via a handle that can be shared amongst all threads.
    - the8472 9y ago
      But linux threads predate go, so go is built on assumptions they knew were not true.
      - YZF 9y ago
        It's possible they didn't know different Linux threads can be in different namespaces, it's not exactly something everyone knows. Namespaces in Linux aren't that new (15 years or so). It's possible they knew but decided this wasn't an important enough use case to warrant language features. I've never needed to have half my process in one namespace and half in another. It's a niche application. Green threads/goroutines is something I use extensively and I wouldn't give it up for the ability to have half my process in another namespace. There's probably some middle ground there in giving Go users more control over which thread pools run which goroutines... Someone should file a Go bug and see what the response is...
deleted 9y ago
[deleted]
cat199 9y ago
alternate title: developers who don't understand user level threading don't understand user level threading.
bhawks 9y ago
I think the proper title is: Linux Process and Threads Don't Mix. The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!) This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse. Check out https://ewontfix.com/17/ https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.
- the8472 9y ago
  > The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. Or it could provide another clone flag that indicates that threads spawned that way should share privileges and similar things, then runtimes that need threads to all behave the same way can opt into that. I suspect that some tools do advanced privilege kung-fu that relies on those per-thread properties.
- mrout 9y ago
  The real answer, IMO, is not to use threads. Just use processes instead.
- jeremyjh 9y ago
  The M:N problem in Go is that you cannot control which thread runs which code. So yes, you could wish the OS exposed different APIs, but presently you can manage this situation in languages that let you manage threads.
sreque 9y ago
I've ran into this problem before. IMO this issue has nothing to do with go and the solution is straightforward. Simply create a sub-process whenever entering a new namespace because the operation isn't concurrency safe within a process. Note that you'd run into this bug within any multithreaded process, whether the code was written in go, Java, c, or whatever.

266 points by _lm_ 9y ago

This leads to go code being roughly as messy/clumsy as C (or whatever else) code in the sections that need concurrency and also need to change namespace. That's unfortunate, but I don't know that it really "raises a few eyebrows". I mean, what's the better alternative to Go for this work? Maybe Rust? It is, at least more controllable at a lower level...but, not as easy to pick up for people coming from a C/Python/Perl/Ruby systems and ops background. I'm not saying one should use Go for containers/namespaces programming, but a lot of people are with some success (probably also banging into the namespaces issue now and then), I'm just saying it's not obvious to me what the better alternative would be.

pcwalton 9y ago
No, it's a lot messier than C. In a regular C program (not using any special libraries for M:N threading) you wouldn't have to spawn an entirely separate process. This issue is one of the downsides of Go's M:N scheduling. The OS is simply not aware of what the Go runtime is doing, and as a result you get impedance mismatches like this. It "raises a few eyebrows" because M:N scheduling is unpopular outside of Go and Erlang. It was tried early on in the Linux world and abandoned precisely because of issues like this. Go has repopularized M:N lately, and it's proof that such a system can work for lots of apps, but the downsides of that decision are every bit as real today as they were in the early days of NPTL.
- SwellJoe 9y ago
  Historically, much (I would probably argue most) C concurrency has been implemented with fork. Certainly not all, and there are many ways to handle it in C...but fork is really common, and I don't think it's considered all that big of a deal to do so. It is idiomatic (at least historically and across maybe billions of lines of C code), and not much harder to reason about than many kinds of thread implementation in C; in fact, it's easier to reason about fork than something like POSIX threads, IMHO. But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency (because you can determine when things will happen with precision in C)? I don't know. Again, this is beyond my understanding of Go right now. I may just not be understanding the implications of this. The code to deal with it looks reasonable enough to me; it's a smallish function, easily isolated. I think the author did a great job explaining the problem, the troubleshooting, and the solution. I just didn't see the problem as being all that damning of Go...but, that may be a reflection of my shallow understanding of the problem, or of the implications of the cost spawning a new process (to me, I always think back to the old adage "fork is cheap on Linux").
  - cyphar 9y ago
    > C concurrency has been implemented with fork. Arguably it has been implemented with clone(2) which has flags. > in fact, it's easier to reason about fork than something like POSIX threads, IMHO. Not if you call fork() in a multi-threaded program. That ends _exceptionally_ badly (let's just say there's a reason Go doesn't expose syscall.Fork and it has to do with horrific deadlocks). > I just didn't see the problem as being all that damning of Go. The problem is more subtle, and it comes down to maintainability and understandably. If you ever decide to read the runc codebase, I apologise. One of the reasons the codebase is so scattered is because of these sorts of hacks where you have to work around issues in the Go runtime (because it doesn't give you enough control). In the article, whenever you read the small function you have to keep in mind that it's actually spawning a subprocess (which then means you have to think about what namespaces had the process joined and so on). Go is an okay language, but it simply wasn't designed for stuff this low-level. We would be much better served with Rust in my opinion. > But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency In C you don't need to fork to switch namespaces, you just call setns(...). For the PID namespace you need to fork, but that's just a quirk of the interface. In Go, you theoretically don't need to fork either (syscall.Setns is available). However, there is no real way to safely use it. First of all, the namespace interfaces in Linux are quite fragile when it comes to multi-threaded processes, but combine that with a runtime that will switch you between OS threads at random. And while the documentation on runtime.GOMAXPROC and runtime.LockOSThread might trick you into believing it's possible to stop the Go runtime from doing a clone(CLONE_THREAD|CLONE_PARENT), you can't.
    - jankedeen 9y ago
      No. You can call fork pretty cleanly in a pthread based program. This post is not informed criticism. There is some prep for ugly contingencies(pthread_atfork) but it usually just works. Namespace based code while doing the same is like setting your hair on fire while running through the gasoline forest..but caveat emptor is usually spelled out pretty critically in the API and docs. Oh, btw: Fuck RUST. For once and for all fuck this tyranny of coercion by potential IP holders. It is a shit language.
- m0th87 9y ago
  This makes sense, and I know it's why Rust abandoned green threading. But I can't help but worry that the focus on async I/O in Rust as a way of avoiding this issue is going to bring the language down a path that isn't as ergonomically pleasant as Go or Erlang's M:N threading for things like highly concurrent web services. Do you share this concern, or do you think Rust can achieve the same level of ergonomics without the impedance mismatch?
  - geodel 9y ago
    You can see for yourself [1]. Seems many users are not following the patterns of Async IO in Rust. 1. https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust_need_today_for_server_workloads/ https://www.reddit.com/r/rust/comments/6enj5d/what_does_rust...
    - pcwalton 9y ago
      1:1 can be made compatible with async I/O with a thread pool, though.
  - steveklabnik 9y ago
    There's a prototype of async/await that can help a lot with that.
  - pcwalton 9y ago
    It's harder for us, but I think we can get to an ergonomic solution with async/await. It's not as easy as threads (M:N vs 1:1 is a red herring as far as this is concerned), but there's no free lunch.
    - pencilhappen 9y ago
      async/await still has the fundamental problem of composability that all the other attempts at sugaring around an event loop have (aka the "functions have colors" problem [1]). It sucks for collaboration, which is one of the most important things for modern software development. 1. http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...
- _ph_ 9y ago
  As far as I know, Solaris threads are actually M:N threads on the system level. M:N threads are extremely efficient. A "native" thread is a heavyweight construct - e.g. by the stack space allocation. So this puts a limit on how many threads you can spawn in a program. For many tasks, this would force you to roll out your own M:N mapping system yourself. Doing it on the language level usually yields the better results. Due to the implicit M:N mapping, Go goroutines are extremely cheap. This allows you to spawn as many, as your algorithm naturally requires. The Go runtime will automatically map them to native threads - typically one per avaliable CPU. As a consequence, a Go program that heavily uses goroutines has a pretty clean code and scales without much overhead across a large variation of number of CPU cores.
  - dap 9y ago
    > As far as I know, Solaris threads are actually M:N threads on the system level. Not for a long time, for all the reasons the parent mentioned (and then some).
  - pjmlp 9y ago
    Only until Solaris 8, Solaris 9 dropped support for the LWP model.
- di4na 9y ago
  Just saying, but Erlang do not target that type of "System Programming" and the answer to that problem in Erlang would probably work through totally different way to do it. This namespace thing would not be a problem. This is not a problem of M:N. This is a problem of Go being badly designed. Not new.
  - dozzie 9y ago
    More like reification of the very old problem from the times when we got threading API, with thread safety of functions that alter process-wide state (umask(), chdir(), sleep()), except now it's calls that alter some thread-specific thing.
  - wbl 9y ago
    It's a problem with Linux not permitting inspection of interfaces through a namespace. If I have a chroot I can look in from outside easily. Apparently not so with container interfaces.
- pjmlp 9y ago
  Also in Solaris, it was tried and abandoned. Although they have their own set of issues, Windows fibers are also barely used.
scriptkiddy 9y ago
> I mean, what's the better alternative to Go for this work? Maybe Rust? It is, at least more controllable at a lower level...but, not as easy to pick up for people coming from a C/Python/Perl/Ruby systems and ops background. You know, it's interesting. I've been programming with Python for about 6 years now. I've also picked up Javascript, SQL, bash, and PHP along the way. I'm always gaining a little bit of C knowledge here and there when writing C extensions for my Python applications. I'm a fairly experienced programmer at this point. To the point: I tried picking up Go one day because I was hearing so much about how it could replace Python as network glue code with better performance and reliable concurrency. I can't really validate or invalidate those claims. That said, I found Go to be sort of difficult. The syntax is really simple. Compiling is really simple. Concurrency is even simple. However, need to do something in a different way than Go decides is correct? Well, you can't. It won't even compile. The difficulty in Go is in learning about what the compiler thinks is OK. I don't really like that. You don't really know if your code will work until you compile. Basically, I just think Go isn't really flexible enough for modern programming. I find that Nim can do Go's job better than Go can for my use cases anyways.
- literallycancer 9y ago
  >You don't really know if your code will work until you compile. You are supposed to "compile on save".
  - scriptkiddy 9y ago
    I usually do this, but sometimes I'm writing a really long function or something and I want to save part way through so I don't lose my progress from some unlikely, yet catastrophic, failure.
- zellyn 9y ago
  > You don't really know if your code will work until you compile. This is true in every text-based programming language. s/compile/execute/ for interpreted or repl-based languages.
  - 9y ago
ahoka 9y ago
Better alternative to C with the same low level control? Why not C++?
- tankenmate 9y ago
  Because C++ is vastly more complex than either C or Go.
  - rkeene2 9y ago
    To go along with this, due to overloading and inheritance rules you cannot look at a small section of C++ code and determine what it does without reading up to the entire program.
    - ahoka 9y ago
      I think that holds for any program written in any language. If the abstractions are confusing then it will be hard to read, be it layers of C macros, over-use of template programming or bad class hierarchy. BTW C++'s standard library is mostly functional (having roots in scheme), which makes it pretty easy to reason about.
  - theamk 9y ago
    In C, I could not find good libraries for basic stuff (dynamic string manipulation, variable-sized arrays, hash tables). I tried glib but it was much worse than C++'s STL. It is entirely possible to write C-style programs in C++ (very few classes, globals all over the place) and so on, and at least for me, C-style software in C++ are much safer/easier to debug than C-style software in C .
    - simias 9y ago
      In my experience the C mindset is to write those yourself. The lack of templates means that it's very difficult to write proper generic containers. You either end up with macro soup, void * soup or code tailored to your needs. Or a lot of the time a combination of all three. But I'm like you, if I know that I'm going to need "advanced" data structures I'll pick C++ over C (or these days more likely Rust).
cyphar 9y ago
Rust is much better for this (though I still feel some of the fork/exec interface has similar warts to Go). However, you're wrong in saying that it's "roughly as messy/clumsy" as C. Let me tell you how runc works. runc is written in Go, and we take an OCI configuration file. Because we can't just fork and set up all of the namespaces in Go, we have a C function called nsexec which is specified as __attribute__((constructor)). This ensures that our code will execute before the Go runtime boots. The parent process writes (using netlink as the wire protocol) to a pipe that the child has open and is parsed in C. Then, the child will have to do a series of forks, unshare, setns, {open,read,write} and so on (and the final PID needs to be sent back to the original parent) in order to set up and join all of the necessary namespaces. In C, this code would be _immensely_ easier to read, write and maintain. Just look at LXC. Personally I really wish people had just gone with Rust earlier on rather than implementing everything in Go. I've had nothing but pain from Go.
- SwellJoe 9y ago
  Thanks for explaining how it's done in runc. That does sound pretty awful. So, even though the initialization can be outsourced to a C function, you still would prefer to be working entirely in C? Are there no advantages to Go for runc? And, would it be possible for someone to write a somewhat standardized Go library for doing this grunt work? Is it merely fear of C that keeps so much of the container infrastructure on Go? I've only spent a couple of weeks looking peripherally at Go, and I already like it better than C (which I've poked at peripherally for ~25 years), but I don't know it well enough to know its warts.
  - cyphar 9y ago
    > you still would prefer to be working entirely in C? No, but there are more options than just Go and C. Rust is an option that I'm shilling at the moment (though I've only started learning it, so take that with a grain of salt). The main reason I would want to write it in C is because there's a lot of string parsing code you have to write in order to make container runtimes work -- and as we all know that's probably the #1 source of security vulnerabilities. > Are there no advantages to Go for runc? There are, mainly due to network effect (everything else is written in Go) and getting contributions from the community (Go is easy to pick up). Unfortunately there are also disadvantages, and quite a few of those disadvantages are present in Go but are not present in other memory-safe and low-level languages (Rust is a good example because to be quite honest it's the only player in this space that doesn't try to do more than necessary). Go is a good language for it's designed for (Web servers and similar things), but from my experience it's not the best choice for low-level tasks. We've seen cases where long-running container daemons (not naming any names) will crash if you run more than 1000 containers on a single system. They don't crash because of the actual daemon code but because of issues with Go's GC (it doesn't actually free memory sanely, it uses MADV_DONTNEED which inflates RSS and causes OOM to kill your daemon). > And, would it be possible for someone to write a somewhat standardized Go library for doing this grunt work? Of course (and you could argue that we have done that in runc with github.com/opencontainers/runc/libcontainer/nsenter), but the thing to note is that in order to get around problems in the Go runtime you have to split out a single piece of code into separate processes and have to now redesign how a single function would work. So moving it to a separate library means that development is even more frustrating (you've created an API around the internal implementation of whatever thing you're working on). > Is it merely fear of C that keeps so much of the container infrastructure on Go? I think the network effect is the main reason. Most of the people I've worked with know quite a lot of C (we do kernel work sometimes) so writing a runtime in C would be frustrating but entirely doable. The problem is that you couldn't just import it into a Go project (and people don't like cgo because it makes binaries harder to build in certain cases). > I've only spent a couple of weeks looking peripherally at Go, and I already like it better than C Go definitely has it's uses, and I still use it for new projects. For example I recently wrote a tool for dealing with OCI container images in Go[1]. The standard library of Go is quite nice (though I found some bugs in archive/tar but let's not go there) and I'm always quite amazed just how much you can do before you have to import external libraries. But I recently started learning Rust and I am _really_ enjoying being able to understand what my program will actually look like when compiled. If you've ever had to strace a Go program, you'll know exactly what I mean. Debugging Go programs is basically fucking impossible. [1]: https://github.com/openSUSE/umoci https://github.com/openSUSE/umoci
vbernat 9y ago
In Linux, many properties are task-related and therefore the API is also task-related. Namespaces are not the only problem. See for example https://github.com/golang/go/issues/1435 https://github.com/golang/go/issues/1435. The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. Most other languages would work just fine (C or Python for example).
- lunixbochs 9y ago
  > The problem is Go runtime does not provide a way to exclusively lock a system thread from existing or future routines. runtime.LockOSThread() does exactly that [1]. [1] https://golang.org/pkg/runtime/#LockOSThread https://golang.org/pkg/runtime/#LockOSThread
  - vbernat 9y ago
    No, it doesn't. Go runtime can decide to spawn a new thread from the locked one. The new thread will inherit the characteristics (namespace, uid) of the old one. See https://github.com/docker/libnetwork/issues/1113 https://github.com/docker/libnetwork/issues/1113 for more details.
    - lunixbochs 9y ago
      > Go runtime can decide to spawn a new thread from the locked one That sounds like an implementation issue, why not assume the documentation is the intended behavior and this side effect is a bug? I'd support a CL to fix this behavior or add a block-clone-from-here runtime call (but the end result of that is you want the thread to exit when you're done with it, and not to go back into the pool... which is also new behavior). At the minimum, this behavior of new threads spawning from LockOSThread could be documented. As a workaround, what about CGO -> pthreads -> spawn a control thread free from the Go scheduler -> call back into Golang to run a control loop function? You can do this in init() to ensure it has full control over itself. Or will Golang call clone() from unscheduled code?
      - 9y ago
mappu 9y ago
You can keep using Go. Its behaviour here is actually usually what you want. To avoid the authors' issue, you can write some functions in C. CGo has a very high level of integration (you can mix languages in the same source file) and would be quite simple for the case of a setns/execve wrapper.
- TheDong 9y ago
  > CGo has a very high level of integration No, no it doesn't. CGo is slow as balls to call into and return from. Types can't fully be shared. It's interacted with via comments. You can't use cgo to control the threading of the Go runtime itself, which is the real problem here. The behavior you want is to be able to call linux syscalls that operate on threads and not be utterly fucked. That behavior cannot be accomplished with go nor go+cgo easily. Threads in cgo are also kinda fucked.
  - mappu 9y ago
    > You can't use cgo to control the threading of the Go runtime itself, which is the real problem here. Go is deliberately opinionated about threading of the runtime. I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS), given the philosophy around e.g. GC tuning. Cgo is a beefed-up runtime.LockOSThread() that could be used to avoid having the author resort to a helper process. > CGo is slow as balls to call into and return from. They're slower than Go function calls, but they still take only nanoseconds. This is negligible on the authors' scale of "launching an entire container". For their case of "it is not possible to guarantee that a new OS process ... will run in a given namespace", you're not even returning from Cgo after exec. > It's interacted with via comments. Do you use build tags? Or go:generate? Like it or not, it's idiomatic. I'm with you on the types, though! `go tool cgo -godefs` helps, but it would be great to see improvements especially in the reverse case of exporting Go buildmode=c-shared for C consumption. Still, a little marshaling seems tidier than a whole helper process.
    - cyphar 9y ago
      > I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS) GOMAXPROCS is actually a lie. If you set it to 1, the runtime will still create threads.
cat199 9y ago
> I mean, what's the better alternative to Go for this work? Separate processes, like the post suggests? I have no idea why someone would expect user-level pseudothreads to execute across system-level primitive boundaries.. seems fairly obvious to me. I don't expect a chrooted daemon (e.g. apache, etc) to have access to parent thread contexts.. etc. Fork & pipe IPC shouldn't be too difficult for anyone to understand, beyond that, if you don't understand these things, you probably shouldn't be writing code that complex..

vishvananda 9y ago

Author of the go netlink library here. I've run into this issue a number of times. There has been conversation in the past about adding some kind of new runtime command like LockOSThread to prevent new threads from being spawned, but it didn't gain any momentum. Even though I am a big fan of go, I've personally built two container runtimes in other languages do to the namespace clumsiness. Personally, I think rust is an excellent alternative for namespace utilities. EDIT: there is more information and links in the issue in the netns library: https://github.com/vishvananda/netns/issues/17 https://github.com/vishvananda/netns/issues/17

cyphar 9y ago
I completely agree, and I've been coming up with some ideas (based on runc) to see what sort of improvements can you have if you do a full redesign in a language that isn't as painful as Go to use. At the moment I'm incredibly busy, but later this year I might be able to start working on that too.
bogomipz 9y ago
>" I've personally built two container runtimes in other languages do to the namespace clumsiness." Can you share any details on those other container runtimes?
- vishvananda 9y ago
  One was a simplified runtime in c, similar to a stripped down version of systemd-nspawn. I can hopefully share the other one in a few weeks. I'm going through an open source approval process for it.
  - bogomipz 9y ago
    Thanks. It's an interesting topic. I hope you post it on HN then.
mrjana 9y ago
One way to alleviate the problem at least for netlink library is to create a function which calls runtimeLockOSThread, sets into the required namespace and then opens a netlink socket using only raw syscall apis. One has to be careful in this code path to not invoke go runtime (i.e both socket and setns should be raw syscall apis) and not even trigger any allocations so that go runtime doesn't get a chance a spin a new OS thread. Once a socket is created in the required namespace you can get back to the caller namespace and return the socket fd. Now this socket fd is bound to that namespace and all netlink operations on that socket will happen in the target namespace. Disclaimer: I am one of the original libnetwork authors and we have been aware of this issue with go for some time now.

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

liveoneggs 9y ago

this is just a more complex version of the issue where go can't safely do the daemonize dance for a privileged port

brandon 9y ago
There's a simple-ish workaround for the privileged port issue if you can't just use CAP_NET_ADMIN: http://play.golang.org/p/dXBizm4xl3 http://play.golang.org/p/dXBizm4xl3 The namespace issues are unfortunately a lot tougher to address.
- vbernat 9y ago
  It's only reliable if none of your dependencies are spawning goroutines during initialization. If this is the case, some goroutines (yours' or the dependencies') can end up with increased privileges.
  - brandon 9y ago
    The workaround functions by opening the privileged socket and re-executing the binary as an unprivileged user with access to the filehandle. Any background goroutines would exit with the privileged parent. Also: please don't spawn goroutines during init(). There's generally a better time and place, and in the event that you justifiably need a package-level background routine you can spawn it on demand with a sync.Once.
- the8472 9y ago
  But you also want drop that privilege after using it to get the same effect as the "daemonize dance" GP referred to. And that's also per-thread, just like namespaces.
cyphar 9y ago
And that's unfortunately only an issue because Linux doesn't implement POSIX's model for thread privileges. In particular, glibc has to implement the nptl in userspace to make Linux threads look like POSIX threads. I get the feeling that the Go runtime doesn't do the same.

ghthor 9y ago

This is discouraging considering go was initially designed as a systems programming language. I wonder if there is another way for go to handle blocking syscall such that this use case would become reliable.

pritambaral 9y ago
I think Go was designed as a server programming language more than a systems programming language.
- stouset 9y ago
  Go was retconned into this role, but originally it was marketed as a systems language.

jankedeen 9y ago

I read this crap in language design and despair. This is why C is my only recourse for systems programming in *nix environments.

Safety1stClyde 9y ago
This is what I cannot understand about Go: how did a newly-designed programming language end up as such a muddle?
- YZF 9y ago
  Because this isn't really a language issue. It's an OS issue. Go assumes that all threads are equal and therefore any Goroutine can run on any thread and threads are all equal. Linux got late to the thread party and its threads are kind of like processes that haven't decided if they want to be processes or threads. It's true the Go runtime could give users more control over goroutine and thread scheduling but that would kind of defeat the purpose of not needing to know about it and having Goroutines as the only flexible unit of concurrency. I think the kludge here is on the Linux side. Having some magic properties bestowed upon threads doesn't make sense. The property should be accessible via a handle that can be shared amongst all threads.
  - the8472 9y ago
    But linux threads predate go, so go is built on assumptions they knew were not true.
    - YZF 9y ago
      It's possible they didn't know different Linux threads can be in different namespaces, it's not exactly something everyone knows. Namespaces in Linux aren't that new (15 years or so). It's possible they knew but decided this wasn't an important enough use case to warrant language features. I've never needed to have half my process in one namespace and half in another. It's a niche application. Green threads/goroutines is something I use extensively and I wouldn't give it up for the ability to have half my process in another namespace. There's probably some middle ground there in giving Go users more control over which thread pools run which goroutines... Someone should file a Go bug and see what the response is...

deleted 9y ago

[deleted]

cat199 9y ago

alternate title: developers who don't understand user level threading don't understand user level threading.

bhawks 9y ago

I think the proper title is: Linux Process and Threads Don't Mix. The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!) This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse. Check out https://ewontfix.com/17/ https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.

the8472 9y ago
> The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. Or it could provide another clone flag that indicates that threads spawned that way should share privileges and similar things, then runtimes that need threads to all behave the same way can opt into that. I suspect that some tools do advanced privilege kung-fu that relies on those per-thread properties.
mrout 9y ago
The real answer, IMO, is not to use threads. Just use processes instead.
jeremyjh 9y ago
The M:N problem in Go is that you cannot control which thread runs which code. So yes, you could wish the OS exposed different APIs, but presently you can manage this situation in languages that let you manage threads.

sreque 9y ago

I've ran into this problem before. IMO this issue has nothing to do with go and the solution is straightforward. Simply create a sub-process whenever entering a new namespace because the operation isn't concurrency safe within a process. Note that you'd run into this bug within any multithreaded process, whether the code was written in go, Java, c, or whatever.