3 ms·

Maybe you know more about it I don't want to say you're wrong because the only thing I know of it is reading the page linked to. But that page semes to disagree

Maybe you know more about it I don't want to say you're wrong because the only thing I know of it is reading the page linked to. But that page semes to disagree with you (EDIT: sorry I don't know how to best format quoting the article vs quoting your reply): Doors Implementation [...] the default server threads that are created in response to incoming door requests on the server. A new synchronization object, called a shuttle [...] allowing a direct scheduler hand-off operation between two threads. [...] a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread. Server Threads Server threads are normally created on demand by the doors library [...] The first server thread is created automatically when the server issues a door create call. [...] Once created, a server thread will place itself in a pool of available threads and wait for a door invocation. There would be no need for these server threads if the client thread transferred directly to the server process address space. > There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests It says the thread pool on the server is created by the doors library when the server creates a door. So the process of receiving and processing requests would be carried out internally within the doors library, so there would be no need for the server application to have an API for the server application to accept requests, it's handled by the library. At least that's what is described in the link, AFAIKS. It's only a conceptual door, underneath it is implemented in some message passing style maybe with some extra sugar for performance and process control nicety with this "shuddle" thing for making the requests.

kragen 2y ago
this is very confusing and now i want to see truss output man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented? but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls > The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations. <https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...> apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl <https://www.unix.com/man-page/opensolaris/3c/door_create/ https://www.unix.com/man-page/opensolaris/3c/door_create/> /me git clone https://github.com/kofemann/opensolaris https://github.com/kofemann/opensolaris jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L193 https://github.com/illumos/illumos-gate/blob/master/usr/src/... unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used __door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist) it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L1284 https://github.com/kofemann/opensolaris/blob/master/usr/src/... this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/door.h#L142 https://github.com/illumos/illumos-gate/blob/master/usr/src/... ...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?
- wahern 2y ago
  The callback address is loaded and invoked here: https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src/lib/libc/amd64/sys/door.S#L121 https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src... It's why door_server_dispatch copies out door_results to the server thread stack. The aha moment was when I realized that the door_return(2) syscall is how threads yield and wait to service the next request. In retrospect it make sense, but I didn't see it until I tried to figure out how a user space thread polled for requests--a thread first calls door_bind, which associates it to the private pool, and then calls door_return with empty arguments to wait for the initial call. (See example door_bind at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L535 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src... and door_return at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L574 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src...) One of the confusing aspects for me was that there's both a global pool of threads and "private" pool of threads. By default threads are pulled from the global pool to service requests, but if you specify DOOR_PRIVATE to door_create, it uses the private pool bound to the door. AFAICT, this is conceptually a typical worker thread pooling implementation, with condition variables, etc for waking and signaling. (See door_get_server at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L1823 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) Context switching does seem to be optimized so door_call doesn't need to bounce through the system scheduler. (See shuttle_resume at https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src/uts/common/disp/shuttle.c#L78 https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src...) And door_call/door_return data is copied across caller and callee address spaces like you'd expect, except the door_return magic permits the kernel to copy the data to the stack, adjusting the stack pointer before resuming the thread and resetting it when it returns. (See door.S above and door_args at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L2438 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) This works just as one might expect: no real magic except for the direct thread-thread context switching. But that's a similar capability provided by user space scheduler activations in NetBSD, or the switchto proposal for Linux. The nomenclature is just different. It's a slightly different story for in-kernel doors, but that's not surprising, either, and there's nothing surprsing there, AFAICT (but I didn't trace it as much). Thanks for finding those source code links. I cloned the repo and started from there, grep'ing the whole tree to find the user space wrappers, etc.
fanf2 2y ago
> There would be no need for these server threads if the client thread transferred directly to the server process address space. The client’s stack isn’t present in the server’s address space: the server needs a pool of threads so that a door call does not need to allocate memory for the server stack.
- starspangled 2y ago
  It wouldn't need a pool of threads for a stack, it just needs some memory in the server's address space for the stack. Together with the other points about marking the current thread sleeping and passing control to the server thread, and about server threads waiting for an invocation, I think what is described is pretty clearly the server threads are being used to execute code on the server on behalf of the client request.
  - kragen 2y ago
    can you figure out how this mechanism works? my notes in https://news.ycombinator.com/item?id=41056051 https://news.ycombinator.com/item?id=41056051 reflect me rooting around and giving up; it's nonobvious
    - kragen 2y ago
      wahern did figure it out
dan353hehe 2y ago
I did some debugging about 8 years ago in SmartOS. There were some processing that were running for way too long without using the cpu. I ended up using kdbg to trace where the processes were hung. I had to jump between different processes to trace the door calls. There was no unified stack in one process that spanned multiple address spaces, and the calling threads were paused. So, yes. The documentation in right, the calling thread is paused while a different one is used one the other side of the door.
netbsdusers 2y ago
Kernel and user threads are distinct on Solaris. The implementation of doors is not by message passing. There needs to be a userland thread context in the server process with a thread control block and all the other things that the userland threading system depends on to provide things like thread-local variables (which include such essentials as errno). I do not know whether there is a full kernel thread (or LWP as they call it in Solaris), but a thread is primarily a schedulable entity, and if the transfer of control is such that the activated server thread is treated in the same way in terms of scheduling as the requesting client thread, then effectively it is still what it says it is.

kragen 2y ago

this is very confusing and now i want to see truss output man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented? but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls > The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations. <https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...> apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl <https://www.unix.com/man-page/opensolaris/3c/door_create/ https://www.unix.com/man-page/opensolaris/3c/door_create/> /me git clone https://github.com/kofemann/opensolaris https://github.com/kofemann/opensolaris jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L193 https://github.com/illumos/illumos-gate/blob/master/usr/src/... unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used __door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist) it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L1284 https://github.com/kofemann/opensolaris/blob/master/usr/src/... this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/door.h#L142 https://github.com/illumos/illumos-gate/blob/master/usr/src/... ...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?

wahern 2y ago
The callback address is loaded and invoked here: https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src/lib/libc/amd64/sys/door.S#L121 https://github.com/illumos/illumos-gate/blob/5d9d909/usr/src... It's why door_server_dispatch copies out door_results to the server thread stack. The aha moment was when I realized that the door_return(2) syscall is how threads yield and wait to service the next request. In retrospect it make sense, but I didn't see it until I tried to figure out how a user space thread polled for requests--a thread first calls door_bind, which associates it to the private pool, and then calls door_return with empty arguments to wait for the initial call. (See example door_bind at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L535 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src... and door_return at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src/lib/libc/port/threads/door_calls.c#L574 https://github.com/illumos/illumos-gate/blob/4a38094/usr/src...) One of the confusing aspects for me was that there's both a global pool of threads and "private" pool of threads. By default threads are pulled from the global pool to service requests, but if you specify DOOR_PRIVATE to door_create, it uses the private pool bound to the door. AFAICT, this is conceptually a typical worker thread pooling implementation, with condition variables, etc for waking and signaling. (See door_get_server at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L1823 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) Context switching does seem to be optimized so door_call doesn't need to bounce through the system scheduler. (See shuttle_resume at https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src/uts/common/disp/shuttle.c#L78 https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src...) And door_call/door_return data is copied across caller and callee address spaces like you'd expect, except the door_return magic permits the kernel to copy the data to the stack, adjusting the stack pointer before resuming the thread and resetting it when it returns. (See door.S above and door_args at https://github.com/illumos/illumos-gate/blob/915894e/usr/src/uts/common/fs/doorfs/door_sys.c#L2438 https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) This works just as one might expect: no real magic except for the direct thread-thread context switching. But that's a similar capability provided by user space scheduler activations in NetBSD, or the switchto proposal for Linux. The nomenclature is just different. It's a slightly different story for in-kernel doors, but that's not surprising, either, and there's nothing surprsing there, AFAICT (but I didn't trace it as much). Thanks for finding those source code links. I cloned the repo and started from there, grep'ing the whole tree to find the user space wrappers, etc.

fanf2 2y ago

> There would be no need for these server threads if the client thread transferred directly to the server process address space. The client’s stack isn’t present in the server’s address space: the server needs a pool of threads so that a door call does not need to allocate memory for the server stack.

starspangled 2y ago
It wouldn't need a pool of threads for a stack, it just needs some memory in the server's address space for the stack. Together with the other points about marking the current thread sleeping and passing control to the server thread, and about server threads waiting for an invocation, I think what is described is pretty clearly the server threads are being used to execute code on the server on behalf of the client request.
- kragen 2y ago
  can you figure out how this mechanism works? my notes in https://news.ycombinator.com/item?id=41056051 https://news.ycombinator.com/item?id=41056051 reflect me rooting around and giving up; it's nonobvious
  - kragen 2y ago
    wahern did figure it out

dan353hehe 2y ago

I did some debugging about 8 years ago in SmartOS. There were some processing that were running for way too long without using the cpu. I ended up using kdbg to trace where the processes were hung. I had to jump between different processes to trace the door calls. There was no unified stack in one process that spanned multiple address spaces, and the calling threads were paused. So, yes. The documentation in right, the calling thread is paused while a different one is used one the other side of the door.

netbsdusers 2y ago

Kernel and user threads are distinct on Solaris. The implementation of doors is not by message passing. There needs to be a userland thread context in the server process with a thread control block and all the other things that the userland threading system depends on to provide things like thread-local variables (which include such essentials as errno). I do not know whether there is a full kernel thread (or LWP as they call it in Solaris), but a thread is primarily a schedulable entity, and if the transfer of control is such that the activated server thread is treated in the same way in terms of scheduling as the requesting client thread, then effectively it is still what it says it is.