4 ms·

The callback address is loaded and invoked here: https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src/lib/libc/amd64/sys/door.S#L121 https://github.com/

by wahern 2y ago

The callback address is loaded and invoked here: https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src/lib/libc/amd64/sys/door.S#L121 https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src... It's why door_server_dispatch copies out door_results to the server thread stack. The aha moment was when I realized that the door_return(2) syscall is how threads yield and wait to service the next request. In retrospect it make sense, but I didn't see it until I tried to figure out how a user space thread polled for requests--a thread first calls door_bind, which associates it to the private pool, and then calls door_return with empty arguments to wait for the initial call. (See example door_bind at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L535 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src... and door_return at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L574 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src...) One of the confusing aspects for me was that there's both a global pool of threads and "private" pool of threads. By default threads are pulled from the global pool to service requests, but if you specify DOOR_PRIVATE to door_create, it uses the private pool bound to the door. AFAICT, this is conceptually a typical worker thread pooling implementation, with condition variables, etc for waking and signaling. (See door_get_server at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L1823 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) Context switching does seem to be optimized so door_call doesn't need to bounce through the system scheduler. (See shuttle_resume at https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src/uts/common/disp/shuttle.c#L78 https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src...) And door_call/door_return data is copied across caller and callee address spaces like you'd expect, except the door_return magic permits the kernel to copy the data to the stack, adjusting the stack pointer before resuming the thread and resetting it when it returns. (See door.S above and door_args at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L2438 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) This works just as one might expect: no real magic except for the direct thread-thread context switching. But that's a similar capability provided by user space scheduler activations in NetBSD, or the switchto proposal for Linux. The nomenclature is just different. It's a slightly different story for in-kernel doors, but that's not surprising, either, and there's nothing surprsing there, AFAICT (but I didn't trace it as much). Thanks for finding those source code links. I cloned the repo and started from there, grep'ing the whole tree to find the user space wrappers, etc.

kragen 2y ago
aha! i should have looked in the assembly. in opensolaris it's opensolaris/usr/src/lib/libc/amd64/sys/door.s line 121; as far as i can see the file hasn't changed at all except for being renamed to door.S. and of course the assembly doesn't call the struct field di_proc, it defines a struct field address macro so door_return (or, in opensolaris, __door_return) is the elusive door_recv syscall, eh? (i'll follow the rest of your links later to read the code) i wonder if spring (which was never released afaik) used a different implementation, maybe a more efficient one this is strongly reminiscent of ipc in liedtke's l4 (which doesn't implicitly create threads) or in keykos (which only has one thread in a domain) thank you so much for figuring this out!
- wahern 2y ago
  The one thing I wasn't sure about was how the per-process global thread pool for non-private doors was populated. I missed it earlier (wasn't looking closely enough, and the hacks to handle forkall headaches make it look more complicated than it is), but the user space door_create wrapper code--specifically door_create_cmn--invokes door_create_server (through the global door_server_func pointer) on the first non-private door_create call. door_create_server creates a single thread which then calls door_return. On wakeup, before calling the application's handler function, the door_return assembly wrapper conditionally invokes the global door_depletion_cb (defined back in door_calls.c) which can spin up additional threads. The more I read, the more Doors seems pretty clever. Yes, the secret sauce is very much like scheduler activations or Google's switchto, but you can pass data at the same time, making it useful across processes, not just threads sharing an address space. And you can't get the performance benefit merely by using AF_UNIX sockets because the door_call and door_return have an association for the pendency of the call. The tricky part from an API perspective isn't switching the time slice to the server thread, it's switching it back to the client thread on the return. Microkernels can do this, of course, but figuring out a set of primitives and an API that fit into the Unix model without introducing anything too foreign, like Mach Ports in macOS, does take some consideration. Now that Linux has process file descriptors, maybe they could be used as an ancillary control message token to get the round-trip time slice and data delivery over AF_UNIX sockets. And memfd objects could be used for messages larger than the receiver's buffer, which seems to be a niche case, anyhow. Or maybe that's over complicating things.

by wahern 2y ago

kragen 2y ago

aha! i should have looked in the assembly. in opensolaris it's opensolaris/usr/src/lib/libc/amd64/sys/door.s line 121; as far as i can see the file hasn't changed at all except for being renamed to door.S. and of course the assembly doesn't call the struct field di_proc, it defines a struct field address macro so door_return (or, in opensolaris, __door_return) is the elusive door_recv syscall, eh? (i'll follow the rest of your links later to read the code) i wonder if spring (which was never released afaik) used a different implementation, maybe a more efficient one this is strongly reminiscent of ipc in liedtke's l4 (which doesn't implicitly create threads) or in keykos (which only has one thread in a domain) thank you so much for figuring this out!

wahern 2y ago
The one thing I wasn't sure about was how the per-process global thread pool for non-private doors was populated. I missed it earlier (wasn't looking closely enough, and the hacks to handle forkall headaches make it look more complicated than it is), but the user space door_create wrapper code--specifically door_create_cmn--invokes door_create_server (through the global door_server_func pointer) on the first non-private door_create call. door_create_server creates a single thread which then calls door_return. On wakeup, before calling the application's handler function, the door_return assembly wrapper conditionally invokes the global door_depletion_cb (defined back in door_calls.c) which can spin up additional threads. The more I read, the more Doors seems pretty clever. Yes, the secret sauce is very much like scheduler activations or Google's switchto, but you can pass data at the same time, making it useful across processes, not just threads sharing an address space. And you can't get the performance benefit merely by using AF_UNIX sockets because the door_call and door_return have an association for the pendency of the call. The tricky part from an API perspective isn't switching the time slice to the server thread, it's switching it back to the client thread on the return. Microkernels can do this, of course, but figuring out a set of primitives and an API that fit into the Unix model without introducing anything too foreign, like Mach Ports in macOS, does take some consideration. Now that Linux has process file descriptors, maybe they could be used as an ancillary control message token to get the round-trip time slice and data delivery over AF_UNIX sockets. And memfd objects could be used for messages larger than the receiver's buffer, which seems to be a niche case, anyhow. Or maybe that's over complicating things.