4 ms·
this is very confusing and now i want to see truss output man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://do
by kragen 2y ago
this is very confusing and now i want to see truss output
man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented?
but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls
> The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations.
<https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...>
apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl
<https://www.unix.com/man-page/opensolaris/3c/door_create/ https://www.unix.com/man-page/opensolaris/3c/door_create/>
/me git clone https://github.com/kofemann/opensolaris https://github.com/kofemann/opensolaris
jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h
okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L193 https://github.com/illumos/illumos-gate/blob/master/usr/src/...
unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used
__door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist)
it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback
i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked
aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L1284 https://github.com/kofemann/opensolaris/blob/master/usr/src/...
this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/door.h#L142 https://github.com/illumos/illumos-gate/blob/master/usr/src/...
...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called
probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?
- wahern 2y agoThe callback address is loaded and invoked here: https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src/lib/libc/amd64/sys/door.S#L121 https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src... It's why door_server_dispatch copies out door_results to the server thread stack. The aha moment was when I realized that the door_return(2) syscall is how threads yield and wait to service the next request. In retrospect it make sense, but I didn't see it until I tried to figure out how a user space thread polled for requests--a thread first calls door_bind, which associates it to the private pool, and then calls door_return with empty arguments to wait for the initial call. (See example door_bind at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L535 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src... and door_return at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L574 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src...) One of the confusing aspects for me was that there's both a global pool of threads and "private" pool of threads. By default threads are pulled from the global pool to service requests, but if you specify DOOR_PRIVATE to door_create, it uses the private pool bound to the door. AFAICT, this is conceptually a typical worker thread pooling implementation, with condition variables, etc for waking and signaling. (See door_get_server at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L1823 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) Context switching does seem to be optimized so door_call doesn't need to bounce through the system scheduler. (See shuttle_resume at https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src/uts/common/disp/shuttle.c#L78 https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src...) And door_call/door_return data is copied across caller and callee address spaces like you'd expect, except the door_return magic permits the kernel to copy the data to the stack, adjusting the stack pointer before resuming the thread and resetting it when it returns. (See door.S above and door_args at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L2438 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) This works just as one might expect: no real magic except for the direct thread-thread context switching. But that's a similar capability provided by user space scheduler activations in NetBSD, or the switchto proposal for Linux. The nomenclature is just different. It's a slightly different story for in-kernel doors, but that's not surprising, either, and there's nothing surprsing there, AFAICT (but I didn't trace it as much). Thanks for finding those source code links. I cloned the repo and started from there, grep'ing the whole tree to find the user space wrappers, etc.