22 ms·

The Go runtime scheduler's way of dealing with system calls

255 points by todotask 7y ago

duelingjello 7y ago
1. Does OS thread M get pinned to run only on a particular processor P? (It seems like "yes" when default.) 2. If M blocks in a syscall too long in the optimistic case: 2.a. is M unpinned from P but continues to block until the syscall returns? 2.b. is another thread from the pool used or new thread created, and pinned to P so that P can be used for other work? (I think this depends on configuration if there are fewer, same or more threads than processors.) 2.c. is there an upper limit on outstanding blocked syscall worker threads or will it simply be the last task any extra created threads beyond the normal limit would ever process?
- siebenmann 7y ago
  An OS thread M can run on any available P. While there are some caches associated with each P, Ps are fundamentally there to insure that only so many CPUs worth of Go user code is ever running at once, so the important thing is that an M that wants to run user Go code has some P, not a particular P. Ms claim and release Ps as they go in and out of running Go user code, but I believe they don't release and then re-acquire a P as they switch between goroutines. (I believe the actual implementation treats Ms as a sort of secondary thing. For instance, I think that the local list of runnable goroutines is attached to the P, not to the M. At one level, the M is just a context for running things on Ps.) In the optimistic case when the system call blocks for too long, the M is unpinned from the P it was using and continues to sit in the system call (the Go runtime doesn't attempt to interrupt the system call itself). If there is another runnable goroutine and there are no free M's, the Go scheduler will create another M to run the goroutine on the now-free P. I think that the runtime directly allocates the free P to the newly created M rather than letting the new M try to contend with other things for the P, but I'm not sure. I don't think there's any limit on the number of Ms (OS threads) that the Go runtime will create, but I haven't checked the code carefully. Idle Ms are reclaimed under some circumstances. (I'm the author of the linked-to article.)
pythux 7y ago
Nice read! I am not very familiar with this field of research but, could runtimes of other languages (say, Node.js or Python) benefit from such optimizations? What about libraries like libuv, I guess they must be fairly fine-tuned already? Or is this something that is specific to Go and would be hard in other contexts?
- kevingadd 7y ago
  Go is one of the only languages that does syscalls itself (mostly because it's extremely high-risk and low-payoff), so some of its syscall-related techniques are not easily adapted to other runtimes.
  - derefr 7y ago
    Do JITing runtimes like the JVM or LuaJIT, just generate libc-syscall-wrapper calls in the emitted object code, then?
    - blattimwind 7y ago
      Something like LuaJIT doesn't have the concept of a syscall, only calling into external C code. The same is probably(?¹) true for the JVM, since Java uses native methods to talk to the OS as well. So the JIT'ed code would call into the language runtime library, which in turn would call a syscall wrapper provided by the libc. ¹ It's possible the JVM, being the highly optimized workhorse VM it is, has specialized optimizations for I/O and does indeed skip over JNI and libc in these cases.
      - justincormack 7y ago
        ljsyscall for example calls into the libc syscall() wrapper. Most ffi type APIs can’t generate assembly for calling directly.
        fsfod 7y ago
        In theory you could do it directly with the intrinsic system I built for LuaJIT[1]. It would dynamically generated the assembly for a user declared intrinsic\arbitrary machine opcode when there first called in the interpreter and the opcode is directly emitted when the code is JIT'ed. I think defining an intrinsic for a system call would just be a matter of setting the correct input and output registers. [1] https://github.com/LuaJIT/LuaJIT/pull/116 https://github.com/LuaJIT/LuaJIT/pull/116
sara1240 7y ago
These are cute cats must see them https://worldtopfive5.blogspot.com/2019/12/worlds-top-5-most-cutest-cats.html https://worldtopfive5.blogspot.com/2019/12/worlds-top-5-most...
protomyth 7y ago
https://marc.info/?l=openbsd-cvs&m=157500930922882&w=2 https://marc.info/?l=openbsd-cvs&m=157500930922882&w=2 For dynamic binaries, we continue to to permit the main program exec segment because "go" (and potentially a few other applications) have embedded system calls in the main program. Hopefully at least go gets fixed soon. We declare the concept of embedded syscalls a bad idea for numerous reasons, as we notice the ecosystem has many of static-syscall-in-base-binary which are dynamically linked against libraries which in turn use libc, which contains another set of syscall stubs. We've been concerned about adding even one additional syscall entry point... but go's approach tends to double the entry-point attack surface. https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 [edit for convenience of readers - read the above linked thread - I just grabbed the go part] Unfortunately our current go build model hasn't followed solaris/macos approach yet of calling libc stubs, and uses the inappropriate "embed system calls directly" method, so for now we'll need to authorize the main program text as well. A comment in exec_elf.c explains this. If go is adapted to call library-based system call stubs on OpenBSD as well, this problem will go away. There may be other environments creating raw system calls. I guess we'll need to find them as time goes by, and hope in time we can repair those also. [/edit]
- lonelappde 7y ago
  Does this mean that they want to ban syscalls from everything but approved "fat client" libraries like libc? (And perhaps ban versions of libc that have bugs?) How is that implemented? I guess it's by only allowing syscalls if the calling code is in a special part of memory, and the OS can gatekeep access to that memory?
  - Reelin 7y ago
    My understanding is that system calls can currently only be made from -w+x regions; attempting otherwise results in the process being killed. The idea is to extend this protection to only allow system calls from expected address ranges, so that a successful exploit can't simply make raw calls but instead has to track down an existing authorized one (and thus contend with ASLR). To that end, the new call-once syscall msyscall(2) is added. The linker uses it to register libc.so with the kernel after randomly mapping it into the current process.
tyingq 7y ago
Makes me somewhat curious how go deals with a hung NFS mount ("hard mount"). I suspect everything would stop, where a normal OS thread wouldn't hang if it weren't interacting with NFS.
- siebenmann 7y ago
  This should work fine. The goroutine making the system call that touches the NFS mount will consume an OS thread (an 'M' in Go terminology), but it will release its hold on other resources. Go uses as many OS threads as necessary to cope with running user code and doing OS system calls and so on (and starts new ones on demand). If you had lots of goroutines do lots of things that stalled on hung NFS mounts, you would build up a lot of OS threads (all sitting in system calls) and might run into limits there. But that's inevitable in any synchronous system call that can stall. (I'm the author of the linked-to article.)
jnwatson 7y ago
This scheduler is probably the most salient feature of Go, but is only indirectly described in the language specification. Perhaps it is just me, but it seems all this user space rigamarole to map bits of execution onto cores points to an overall architecture “smell”. This should be performed and enabled by the OS. You can see the seams between the OS and the go runtime tear a little whenever a library acquires an ownership lock where the thread id is recorded. In Go, computation moves freely between threads, so that lock doesn’t work (at least without special instructions to the runtime to lock that goroutine to a thread). The whole POSIX threading model seems broken in this context.
- gok 7y ago
  POSIX threading is not broken, the Go scheduler just does a bunch of goofy things that aren't really supported. Moving stacks between threads breaks all kinds of things. A more idiomatic approach would be for the compiler to emit properly resumable functions, like most async/await implementations do.
  - mitchty 7y ago
    The go runtime moves stacks between threads? Oof that’s horrible, any pointers to the logic behind it? I’m curious the rationale.
    - duelingjello 7y ago
      I think to keep Go code directly callable from C, they have to follow the platform's C calling conventions which means the same stack layout. So for cooperative concurrency on a single thread to work, each Goroutine needs its very own stack. On Intel, that means saving stack pointers RSP and RBP (16 bytes) for each. Also, each will need memory allocated for its stack for the stack pointers to point to... another 8-16 bytes (pointer and length).
      - echlebek 7y ago
        The gc compiler, used by the vast majority of Go developers, does not use the C calling convention. https://golang.org/doc/faq#Do_Go_programs_link_with_Cpp_programs https://golang.org/doc/faq#Do_Go_programs_link_with_Cpp_prog...
amluto 7y ago
A side effect of this scheme is that a long sequence of slow-but-not-that-slow syscalls becomes extremely slow because the Go scheduler gets invoked each time.
derefr 7y ago
I would love to see a compare-and-contrast between the Golang scheduler and the Erlang scheduler, in the way they handle network-IO-heavy workloads. Maybe throw in the JVM scheduler, too (though its JIT would likely complicate things.)
jancsika 7y ago
> For example, on modern systems the 'system call' to get the current time may not even enter the kernel (see vdso(7) on Linux). Is there a way to check from the running process whether that is the case or not?
- wyldfire 7y ago
  Check whether it happened to do that for a given call or a portable way to predict whether it will? dump_vdso [1] will write the vdso to stdout, you can use binutils like objdump or nm to list the symbols present. [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/luto/misc-tests/+/5655bd41ffedc002af69e3a8d1b0a168c22f2549/dump-vdso.c https://kernel.googlesource.com/pub/scm/linux/kernel/git/lut...
  - monocasa 7y ago
    And if you wanted the same information at runtime, the base address of the VDSO is passed in as an auxv, and then passing that address into libelf would get you everything.

255 points by todotask 7y ago

1. Does OS thread M get pinned to run only on a particular processor P? (It seems like "yes" when default.) 2. If M blocks in a syscall too long in the optimistic case: 2.a. is M unpinned from P but continues to block until the syscall returns? 2.b. is another thread from the pool used or new thread created, and pinned to P so that P can be used for other work? (I think this depends on configuration if there are fewer, same or more threads than processors.) 2.c. is there an upper limit on outstanding blocked syscall worker threads or will it simply be the last task any extra created threads beyond the normal limit would ever process?

siebenmann 7y ago
An OS thread M can run on any available P. While there are some caches associated with each P, Ps are fundamentally there to insure that only so many CPUs worth of Go user code is ever running at once, so the important thing is that an M that wants to run user Go code has some P, not a particular P. Ms claim and release Ps as they go in and out of running Go user code, but I believe they don't release and then re-acquire a P as they switch between goroutines. (I believe the actual implementation treats Ms as a sort of secondary thing. For instance, I think that the local list of runnable goroutines is attached to the P, not to the M. At one level, the M is just a context for running things on Ps.) In the optimistic case when the system call blocks for too long, the M is unpinned from the P it was using and continues to sit in the system call (the Go runtime doesn't attempt to interrupt the system call itself). If there is another runnable goroutine and there are no free M's, the Go scheduler will create another M to run the goroutine on the now-free P. I think that the runtime directly allocates the free P to the newly created M rather than letting the new M try to contend with other things for the P, but I'm not sure. I don't think there's any limit on the number of Ms (OS threads) that the Go runtime will create, but I haven't checked the code carefully. Idle Ms are reclaimed under some circumstances. (I'm the author of the linked-to article.)

pythux 7y ago

Nice read! I am not very familiar with this field of research but, could runtimes of other languages (say, Node.js or Python) benefit from such optimizations? What about libraries like libuv, I guess they must be fairly fine-tuned already? Or is this something that is specific to Go and would be hard in other contexts?

kevingadd 7y ago
Go is one of the only languages that does syscalls itself (mostly because it's extremely high-risk and low-payoff), so some of its syscall-related techniques are not easily adapted to other runtimes.
- derefr 7y ago
  Do JITing runtimes like the JVM or LuaJIT, just generate libc-syscall-wrapper calls in the emitted object code, then?
  - blattimwind 7y ago
    Something like LuaJIT doesn't have the concept of a syscall, only calling into external C code. The same is probably(?¹) true for the JVM, since Java uses native methods to talk to the OS as well. So the JIT'ed code would call into the language runtime library, which in turn would call a syscall wrapper provided by the libc. ¹ It's possible the JVM, being the highly optimized workhorse VM it is, has specialized optimizations for I/O and does indeed skip over JNI and libc in these cases.
    - justincormack 7y ago
      ljsyscall for example calls into the libc syscall() wrapper. Most ffi type APIs can’t generate assembly for calling directly.
      - fsfod 7y ago
        In theory you could do it directly with the intrinsic system I built for LuaJIT[1]. It would dynamically generated the assembly for a user declared intrinsic\arbitrary machine opcode when there first called in the interpreter and the opcode is directly emitted when the code is JIT'ed. I think defining an intrinsic for a system call would just be a matter of setting the correct input and output registers. [1] https://github.com/LuaJIT/LuaJIT/pull/116 https://github.com/LuaJIT/LuaJIT/pull/116

sara1240 7y ago

These are cute cats must see them https://worldtopfive5.blogspot.com/2019/12/worlds-top-5-most-cutest-cats.html https://worldtopfive5.blogspot.com/2019/12/worlds-top-5-most...

protomyth 7y ago

https://marc.info/?l=openbsd-cvs&m=157500930922882&w=2 https://marc.info/?l=openbsd-cvs&m=157500930922882&w=2 For dynamic binaries, we continue to to permit the main program exec segment because "go" (and potentially a few other applications) have embedded system calls in the main program. Hopefully at least go gets fixed soon. We declare the concept of embedded syscalls a bad idea for numerous reasons, as we notice the ecosystem has many of static-syscall-in-base-binary which are dynamically linked against libraries which in turn use libc, which contains another set of syscall stubs. We've been concerned about adding even one additional syscall entry point... but go's approach tends to double the entry-point attack surface. https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 [edit for convenience of readers - read the above linked thread - I just grabbed the go part] Unfortunately our current go build model hasn't followed solaris/macos approach yet of calling libc stubs, and uses the inappropriate "embed system calls directly" method, so for now we'll need to authorize the main program text as well. A comment in exec_elf.c explains this. If go is adapted to call library-based system call stubs on OpenBSD as well, this problem will go away. There may be other environments creating raw system calls. I guess we'll need to find them as time goes by, and hope in time we can repair those also. [/edit]

lonelappde 7y ago
Does this mean that they want to ban syscalls from everything but approved "fat client" libraries like libc? (And perhaps ban versions of libc that have bugs?) How is that implemented? I guess it's by only allowing syscalls if the calling code is in a special part of memory, and the OS can gatekeep access to that memory?
- Reelin 7y ago
  My understanding is that system calls can currently only be made from -w+x regions; attempting otherwise results in the process being killed. The idea is to extend this protection to only allow system calls from expected address ranges, so that a successful exploit can't simply make raw calls but instead has to track down an existing authorized one (and thus contend with ASLR). To that end, the new call-once syscall msyscall(2) is added. The linker uses it to register libc.so with the kernel after randomly mapping it into the current process.

tyingq 7y ago

Makes me somewhat curious how go deals with a hung NFS mount ("hard mount"). I suspect everything would stop, where a normal OS thread wouldn't hang if it weren't interacting with NFS.

siebenmann 7y ago
This should work fine. The goroutine making the system call that touches the NFS mount will consume an OS thread (an 'M' in Go terminology), but it will release its hold on other resources. Go uses as many OS threads as necessary to cope with running user code and doing OS system calls and so on (and starts new ones on demand). If you had lots of goroutines do lots of things that stalled on hung NFS mounts, you would build up a lot of OS threads (all sitting in system calls) and might run into limits there. But that's inevitable in any synchronous system call that can stall. (I'm the author of the linked-to article.)

jnwatson 7y ago

This scheduler is probably the most salient feature of Go, but is only indirectly described in the language specification. Perhaps it is just me, but it seems all this user space rigamarole to map bits of execution onto cores points to an overall architecture “smell”. This should be performed and enabled by the OS. You can see the seams between the OS and the go runtime tear a little whenever a library acquires an ownership lock where the thread id is recorded. In Go, computation moves freely between threads, so that lock doesn’t work (at least without special instructions to the runtime to lock that goroutine to a thread). The whole POSIX threading model seems broken in this context.

gok 7y ago
POSIX threading is not broken, the Go scheduler just does a bunch of goofy things that aren't really supported. Moving stacks between threads breaks all kinds of things. A more idiomatic approach would be for the compiler to emit properly resumable functions, like most async/await implementations do.
- mitchty 7y ago
  The go runtime moves stacks between threads? Oof that’s horrible, any pointers to the logic behind it? I’m curious the rationale.
  - duelingjello 7y ago
    I think to keep Go code directly callable from C, they have to follow the platform's C calling conventions which means the same stack layout. So for cooperative concurrency on a single thread to work, each Goroutine needs its very own stack. On Intel, that means saving stack pointers RSP and RBP (16 bytes) for each. Also, each will need memory allocated for its stack for the stack pointers to point to... another 8-16 bytes (pointer and length).
    - echlebek 7y ago
      The gc compiler, used by the vast majority of Go developers, does not use the C calling convention. https://golang.org/doc/faq#Do_Go_programs_link_with_Cpp_programs https://golang.org/doc/faq#Do_Go_programs_link_with_Cpp_prog...

amluto 7y ago

A side effect of this scheme is that a long sequence of slow-but-not-that-slow syscalls becomes extremely slow because the Go scheduler gets invoked each time.

derefr 7y ago

I would love to see a compare-and-contrast between the Golang scheduler and the Erlang scheduler, in the way they handle network-IO-heavy workloads. Maybe throw in the JVM scheduler, too (though its JIT would likely complicate things.)

jancsika 7y ago

> For example, on modern systems the 'system call' to get the current time may not even enter the kernel (see vdso(7) on Linux). Is there a way to check from the running process whether that is the case or not?

wyldfire 7y ago
Check whether it happened to do that for a given call or a portable way to predict whether it will? dump_vdso [1] will write the vdso to stdout, you can use binutils like objdump or nm to list the symbols present. [1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/luto/misc-tests/+/5655bd41ffedc002af69e3a8d1b0a168c22f2549/dump-vdso.c https://kernel.googlesource.com/pub/scm/linux/kernel/git/lut...
- monocasa 7y ago
  And if you wanted the same information at runtime, the base address of the VDSO is passed in as an auxv, and then passing that address into libelf would get you everything.