7 ms·

NOTE: You can't write an init without threads in Go, as you can in C. (I tried awhile back) At least you can't with its portable interfaces such as exec.Comma

by chubot 11y ago

NOTE: You can't write an init without threads in Go, as you can in C. (I tried awhile back) At least you can't with its portable interfaces such as exec.Command() and cmd.Wait(): http://www.mustafaak.in/2016/02/09/forking-process-in-myinit-go.html http://www.mustafaak.in/2016/02/09/forking-process-in-myinit... You end burning a thread per process, which is pretty lame for an init. The wait() call blocks a goroutine AND OS thread, so the go runtime will have to start a new thread for the next wait(). Whereas in C you can do the entire thing from a simple thread (I call it "async processes"). You set SIGCHLD handler and get async exit notifications, and then process those in a single-threaded loop. I'm pretty sure there is a problem doing this in Go (besides resorting to the syscall module)... I think it has to do with the fact that Go turns signals into messages on a channel, but I don't remember right now... EDIT: Now I remember. The problem with using the syscall module is that Go does not export Fork and Exec. It exports ForkExec, because it needs to ensure they run on the same thread. https://golang.org/pkg/syscall/#ForkExec https://golang.org/pkg/syscall/#ForkExec This severely limits the usefulness of your init program, because in Unix there are all sorts of important things that happen in between. It's how you set the child process state. You basically can't do anything with file descriptors or containers at all with this model.

pcwalton 11y ago
Signal handlers are pretty much impossible to write directly in a managed language (at least without heavy contortions from the runtime), because you could get a signal during a non-GC-safe point and then deadlock or corrupt your program. For example, malloc often takes locks internally; if you get a signal during malloc with that lock held and then try to malloc in your signal handler you can deadlock. Managed languages like Go treat allocation as an implementation detail rather than something the programmer has to explicitly invoke, making them basically incompatible with signal handlers. (BTW, this is mostly a problem with signal handlers as a concept, not with Go.) > The problem with using the syscall module is that Go does not export Fork and Exec. It exports ForkExec, because it needs to ensure they run on the same thread. And because you can't safely allocate after a fork in managed languages, because fork drops your other threads, leading to the same problems above. :)
- chubot 11y ago
  So the way Python does it is that it doesn't call back into the interpreter loop from the signal handler. That would be a really bad idea as you note, because the interpreter can call malloc and do all sorts of other stuff. It queues your Python signal handling function, returns from the signal handler, and then runs it on the main loop at the next tick. As far as I know this works fine. I wrote the same program in Python and it seems to work just fine, and have more functionality than the Go version. I guess one problem is that if you do a long blocking syscall on a tick in the main loop, your signal handler can be delayed indefinitely. There might be some hacks around this in the Python interpreter, but I don't recall offhand.
  - pcwalton 11y ago
    Yeah, that's basically having the runtime advance to the next GC safe point before running the handler. It's not direct control over signal handlers, but it's fairly low overhead. I guess Go could do this too if it wanted to. I don't think it works for fork though.
    - chubot 11y ago
      Yeah the problem with Go and "async processes" doesn't have to do with GC -- it has to do with M:N threading. I'm glad Rust abandoned M:N threading. Though it is definitely a killer feature of Go to write in a blocking style rather than in a state machine style... but I feel that there might have been a way to do it without baking it so hard into the runtime (?). Maybe like coroutines in Python, which run on a single thread and are optional. M:N threading causes a lot of other complications too, like calling from C to Go, etc.
  - d0mine 11y ago
    A signal interrupts a blocking syscall. You can call PyErr_CheckSignals() that may execute Python signal handler before you restart the syscal on EINTR in C code. Look at how os.waitpid() is implemented: https://github.com/python/cpython/blob/master/Modules/posixmodule.c https://github.com/python/cpython/blob/master/Modules/posixm...
deleted 11y ago
[deleted]
- noselasd 11y ago
  Option 1) is not feasible though, as you'll block your one thread. Though I wonder if you can work around all this by using signalfd (http://man7.org/linux/man-pages/man2/signalfd.2.html http://man7.org/linux/man-pages/man2/signalfd.2.html) and have SIGCHLD delivered through normal file descriptors - I don't know the low levels of Go enough to know if you can integrate a raw file descriptor with its runtime, or whether messing with SIGCHLD causes other issues, but it might be worth pursuing.
  - nteon 11y ago
    You could easily do a syscall to get a signalfd file descriptor, then call os.NewFile to turn that numerical FD into a Go *os.File (https://golang.org/pkg/os/#NewFile https://golang.org/pkg/os/#NewFile). It seems like that should work fine!
- chubot 11y ago
  Yes, but if you see my edit, even with GOMAXPROCS=1, you are still constrained to the syscall.ForkExec API, which severely limits the usefulness of an init program. Its entire purpose is to start processes in different states (i.e. with different UID, etc.) Not sure what you mean by 2). If you're suggesting polling, then that sucks, because one important feature of an init system is to restart processes immediately. All in all my point is that Go is a very poor language for this... worse than Python (which I also tried and ended up using). I think it's great that the author is experimenting though -- I think that's the only way to make these issues clear.
- ominous_prime 11y ago
  #1 is not equivalent to being single threaded (every go program is multithreaded), it just means there only 1 thread for user goroutines. Every call to wait will block and spawn a new thread. If you force it to stay on the same thread, you block the runtime, and your entire program. An init system has to call wait, that's one of its primary tasks, especially on orphaned processes it inherits.
sebcat 11y ago
I don't understand. How is using ForkExec, Wait4 forcing you to use threads? What do you mean with "You basically can't do anything with file descriptors or containers at all with this model."? It is possible to pass a ProcAddr containing a SysProcAddr to ForkExec with fds, chroot, namespaces, &c. Attached simple demo code of what I mean with ForkExec, Wait4. EDIT: to clarify, the demo code is not intended to work like init, just to spawn a couple of processes and wait for them to terminate. package main import ( "log" "syscall" ) func main() { cmds := [][]string{ {"/usr/bin/sleep", "4"}, {"/usr/bin/sleep", "5"}, {"/usr/bin/sleep", "1"}, {"/usr/bin/sleep", "2"}, } for _, cmd := range cmds { if len(cmd) > 0 { if _, err := syscall.ForkExec(cmd[0], cmd, nil); err != nil { log.Fatal(err) } } } for nwaited := 0; nwaited < len(cmds); nwaited++ { var w syscall.WaitStatus pid, err := syscall.Wait4(-1, &w, 0, nil) if err != nil { log.Println(err) } else { log.Println("pid", pid, "exited", w.Exited(), "exit status", w.ExitStatus()) } } }
- chubot 11y ago
  OK, I see that there is a lot of stuff like chroot and UIDs in SysProcAttr [1]. I think they have added a bunch since I last tried this. But the point still stands... you get a subset of the OS API. Everything has to be "canned" into that struct and you can't execute arbitrary code between fork and exec. Especially with containers, this is a never-ending target... They apparently have user namespaces, but what about network namespaces or IPC namespaces? [1] https://golang.org/src/syscall/exec_linux.go https://golang.org/src/syscall/exec_linux.go Now, it might be good enough for your application, and it probably IS good enough for everything except a full-fledged production quality init system. For that case I would be worried about running into limitations.
  - sebcat 11y ago
    > Especially with containers, this is a never-ending target... They apparently have user namespaces, but what about network namespaces or IPC namespaces? There's no reason calls to e.g. unshare(2) and setns(2) can't be added to forkAndExecInChild in exec_linux.go like they have been added for the other syscalls needed to set up a child. I guess if anyone is reading this thread and wants that they can follow the guidelines for contributions and look at https://github.com/golang/go/issues/5968 https://github.com/golang/go/issues/5968 EDIT: It's also possible to supply CloneFlags though the SysProcAttr struct, e.g., CLONE_NEWIPC, CLONE_NEWNET.
falcolas 11y ago
So, don't do process management inside init. Spawn a child and check on it after every "wait" to handle the process management. Less opportunity for the kernel panic, less for init to handle (fewer bugs), greater flexibility.
vetrom 11y ago
wait() doesn't need to block at all actually, you can use WNOHANG.
- chubot 11y ago
  Yeah but then you're polling, which isn't good as mentioned elsewhere).

by chubot 11y ago

pcwalton 11y ago

Signal handlers are pretty much impossible to write directly in a managed language (at least without heavy contortions from the runtime), because you could get a signal during a non-GC-safe point and then deadlock or corrupt your program. For example, malloc often takes locks internally; if you get a signal during malloc with that lock held and then try to malloc in your signal handler you can deadlock. Managed languages like Go treat allocation as an implementation detail rather than something the programmer has to explicitly invoke, making them basically incompatible with signal handlers. (BTW, this is mostly a problem with signal handlers as a concept, not with Go.) > The problem with using the syscall module is that Go does not export Fork and Exec. It exports ForkExec, because it needs to ensure they run on the same thread. And because you can't safely allocate after a fork in managed languages, because fork drops your other threads, leading to the same problems above. :)

chubot 11y ago
So the way Python does it is that it doesn't call back into the interpreter loop from the signal handler. That would be a really bad idea as you note, because the interpreter can call malloc and do all sorts of other stuff. It queues your Python signal handling function, returns from the signal handler, and then runs it on the main loop at the next tick. As far as I know this works fine. I wrote the same program in Python and it seems to work just fine, and have more functionality than the Go version. I guess one problem is that if you do a long blocking syscall on a tick in the main loop, your signal handler can be delayed indefinitely. There might be some hacks around this in the Python interpreter, but I don't recall offhand.
- pcwalton 11y ago
  Yeah, that's basically having the runtime advance to the next GC safe point before running the handler. It's not direct control over signal handlers, but it's fairly low overhead. I guess Go could do this too if it wanted to. I don't think it works for fork though.
  - chubot 11y ago
    Yeah the problem with Go and "async processes" doesn't have to do with GC -- it has to do with M:N threading. I'm glad Rust abandoned M:N threading. Though it is definitely a killer feature of Go to write in a blocking style rather than in a state machine style... but I feel that there might have been a way to do it without baking it so hard into the runtime (?). Maybe like coroutines in Python, which run on a single thread and are optional. M:N threading causes a lot of other complications too, like calling from C to Go, etc.
- d0mine 11y ago
  A signal interrupts a blocking syscall. You can call PyErr_CheckSignals() that may execute Python signal handler before you restart the syscal on EINTR in C code. Look at how os.waitpid() is implemented: https://github.com/python/cpython/blob/master/Modules/posixmodule.c https://github.com/python/cpython/blob/master/Modules/posixm...

deleted 11y ago

[deleted]

noselasd 11y ago
Option 1) is not feasible though, as you'll block your one thread. Though I wonder if you can work around all this by using signalfd (http://man7.org/linux/man-pages/man2/signalfd.2.html http://man7.org/linux/man-pages/man2/signalfd.2.html) and have SIGCHLD delivered through normal file descriptors - I don't know the low levels of Go enough to know if you can integrate a raw file descriptor with its runtime, or whether messing with SIGCHLD causes other issues, but it might be worth pursuing.
- nteon 11y ago
  You could easily do a syscall to get a signalfd file descriptor, then call os.NewFile to turn that numerical FD into a Go *os.File (https://golang.org/pkg/os/#NewFile https://golang.org/pkg/os/#NewFile). It seems like that should work fine!
chubot 11y ago
Yes, but if you see my edit, even with GOMAXPROCS=1, you are still constrained to the syscall.ForkExec API, which severely limits the usefulness of an init program. Its entire purpose is to start processes in different states (i.e. with different UID, etc.) Not sure what you mean by 2). If you're suggesting polling, then that sucks, because one important feature of an init system is to restart processes immediately. All in all my point is that Go is a very poor language for this... worse than Python (which I also tried and ended up using). I think it's great that the author is experimenting though -- I think that's the only way to make these issues clear.
ominous_prime 11y ago
#1 is not equivalent to being single threaded (every go program is multithreaded), it just means there only 1 thread for user goroutines. Every call to wait will block and spawn a new thread. If you force it to stay on the same thread, you block the runtime, and your entire program. An init system has to call wait, that's one of its primary tasks, especially on orphaned processes it inherits.

sebcat 11y ago

I don't understand. How is using ForkExec, Wait4 forcing you to use threads? What do you mean with "You basically can't do anything with file descriptors or containers at all with this model."? It is possible to pass a ProcAddr containing a SysProcAddr to ForkExec with fds, chroot, namespaces, &c. Attached simple demo code of what I mean with ForkExec, Wait4. EDIT: to clarify, the demo code is not intended to work like init, just to spawn a couple of processes and wait for them to terminate. package main import ( "log" "syscall" ) func main() { cmds := [][]string{ {"/usr/bin/sleep", "4"}, {"/usr/bin/sleep", "5"}, {"/usr/bin/sleep", "1"}, {"/usr/bin/sleep", "2"}, } for _, cmd := range cmds { if len(cmd) > 0 { if _, err := syscall.ForkExec(cmd[0], cmd, nil); err != nil { log.Fatal(err) } } } for nwaited := 0; nwaited < len(cmds); nwaited++ { var w syscall.WaitStatus pid, err := syscall.Wait4(-1, &w, 0, nil) if err != nil { log.Println(err) } else { log.Println("pid", pid, "exited", w.Exited(), "exit status", w.ExitStatus()) } } }

chubot 11y ago
OK, I see that there is a lot of stuff like chroot and UIDs in SysProcAttr [1]. I think they have added a bunch since I last tried this. But the point still stands... you get a subset of the OS API. Everything has to be "canned" into that struct and you can't execute arbitrary code between fork and exec. Especially with containers, this is a never-ending target... They apparently have user namespaces, but what about network namespaces or IPC namespaces? [1] https://golang.org/src/syscall/exec_linux.go https://golang.org/src/syscall/exec_linux.go Now, it might be good enough for your application, and it probably IS good enough for everything except a full-fledged production quality init system. For that case I would be worried about running into limitations.
- sebcat 11y ago
  > Especially with containers, this is a never-ending target... They apparently have user namespaces, but what about network namespaces or IPC namespaces? There's no reason calls to e.g. unshare(2) and setns(2) can't be added to forkAndExecInChild in exec_linux.go like they have been added for the other syscalls needed to set up a child. I guess if anyone is reading this thread and wants that they can follow the guidelines for contributions and look at https://github.com/golang/go/issues/5968 https://github.com/golang/go/issues/5968 EDIT: It's also possible to supply CloneFlags though the SysProcAttr struct, e.g., CLONE_NEWIPC, CLONE_NEWNET.

falcolas 11y ago

So, don't do process management inside init. Spawn a child and check on it after every "wait" to handle the process management. Less opportunity for the kernel panic, less for init to handle (fewer bugs), greater flexibility.

vetrom 11y ago

wait() doesn't need to block at all actually, you can use WNOHANG.

chubot 11y ago
Yeah but then you're polling, which isn't good as mentioned elsewhere).