4 ms·

Writing a top level comment here to hopefully address some of the misconceptions present across this comment section. Doors at the end of the day aren't messag

Writing a top level comment here to hopefully address some of the misconceptions present across this comment section. Doors at the end of the day aren't message passing based RPC. There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests. Doors at the end of the day are a control transfer primitive. In a very real sense the calling thread is simply transferred to the callee's address space and continues execution there until a door_return(2) syscall transfers it back into the caller address_space. It truly is a 'door' into another address space. This is most similar to some of the CPU control transfer primitives. It's most like task gate style constructs like seen on 286/i432/mill CPUs. Arguably it's kind of like the system call instruction itself too, transferring execution directly to another address space/context.

kragen 2y ago
what does the stack look like in the callee's address space? is it a stack that previously existed (you seem to be saying it isn't) or a new one created on demand? maybe https://docs.oracle.com/cd/E88353_01/html/E37843/door-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-create... is where i should start reading...
starspangled 2y ago
Maybe you know more about it I don't want to say you're wrong because the only thing I know of it is reading the page linked to. But that page semes to disagree with you (EDIT: sorry I don't know how to best format quoting the article vs quoting your reply): Doors Implementation [...] the default server threads that are created in response to incoming door requests on the server. A new synchronization object, called a shuttle [...] allowing a direct scheduler hand-off operation between two threads. [...] a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread. Server Threads Server threads are normally created on demand by the doors library [...] The first server thread is created automatically when the server issues a door create call. [...] Once created, a server thread will place itself in a pool of available threads and wait for a door invocation. There would be no need for these server threads if the client thread transferred directly to the server process address space. > There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests It says the thread pool on the server is created by the doors library when the server creates a door. So the process of receiving and processing requests would be carried out internally within the doors library, so there would be no need for the server application to have an API for the server application to accept requests, it's handled by the library. At least that's what is described in the link, AFAIKS. It's only a conceptual door, underneath it is implemented in some message passing style maybe with some extra sugar for performance and process control nicety with this "shuddle" thing for making the requests.
- kragen 2y ago
  this is very confusing and now i want to see truss output man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented? but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls > The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations. <https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...> apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl <https://www.unix.com/man-page/opensolaris/3c/door_create/ https://www.unix.com/man-page/opensolaris/3c/door_create/> /me git clone https://github.com/kofemann/opensolaris https://github.com/kofemann/opensolaris jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L193 https://github.com/illumos/illumos-gate/blob/master/usr/src/... unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used __door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist) it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L1284 https://github.com/kofemann/opensolaris/blob/master/usr/src/... this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/door.h#L142 https://github.com/illumos/illumos-gate/blob/master/usr/src/... ...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?
wang_li 2y ago
>Doors at the end of the day are a control transfer primitive. In a very real sense the calling thread is simply transferred to the callee's address space and continues execution there until a door_return(2) syscall transfers it back into the caller address_space. Your phrasing is misleading. A door is a scheduler operation. What would it even mean for a thread to go from a caller to a callee? They are different processes with completely different contexts on the system. Different owners, different address spaces, etc. What the door is doing is passing time on a CPU core from the calling thread to the receiving thread and back.
ajross 2y ago
> Doors at the end of the day are a control transfer primitive. This is true, but isomorphic to RPC, so I really don't think I understand the distinction you're trying to draw. A "procedure call" halts the "caller", passes "arguments" into the call, and "returns a value" when it completes. And that's what a door call does. Nothing about "Remote Procedure Calls" requires thread pools or a recv() system call, or an idea of "waiting for a request". Those are all just implementation details of existing RPC mechanisms implemented without a send+recv call like doors. Nor FWIW are doors alone in this space. Android Binder is effectively the same metaphor, just with a more opinionated idea of how the kernel should manage the discovery and dispatch layer (Solaris just gives you a descriptor you can call and lets userspace figure it out).
- klodolph 2y ago
  I thought the parent comment was drawing a very clear distinction—that doors do not involve message passing, but a different mechanism. If it were a message passing system, you would be able to save the message to disk and inspect it later, or transmit the message over the network, etc.
  - ajross 2y ago
    Is this a pedantic argument about a specific definition for "message passing"? If not, then it's wrong: door_call() allows passing an arbitrary/opaque buffer of data into the call, and returning arbitrary data as part of the return. (Maybe the confusion stems from the fact that the sample code in the linked article skipped the second argument to door_call(), but you can see it documented in the man page). If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing". That, again, is just an implementation detail of other RPC mechanisms that don't implement the call as directly as doors (or binder) do.
    - klodolph 2y ago
      > Is this a pedantic argument about a specific definition for "message passing"? No. This is just colloquial usage of the term “message passing”. Yes, you can use doors to pass messages. That is not the only thing you can do. > If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing". Yeah. Everyone else here agrees with that. The original comment you replied to said, “Doors at the end of the day aren't message passing based RPC.” This absolutely indicates that the poster agrees with you on that point—that not all forms of RPC are based on a message passing system. There are other forms of RPC, and this is one of them. Ultimately, I think you can conceive of “message passing” broadly enough that, well, everything is message passing. What is a CPU register but something that receives a message and later re-transmits it? I would rather think of the colloquial use of “message passing”. Likewise, you can probably dig into doors and find some message system somewhere in the implementation. Or at least, you’ll find something you can call a message.
cryptonector 2y ago
A much better description is that doors is RPC-like but with priority and CPU quantum transfer to the server thread. This works great as long as the server thread isn't itself doing slow I/O. The moment the server thread does slow I/O you want to get back to async I/O, and the whole point of doors is that it's synchronous because only by being synchronous do you get to transfer priority and CPU quantum from the client to the server. For async RPC-like schemes you cannot transfer priority and CPU quantum to the server because the client will continue executing with that same CPU quantum (that being implied by "async"). Also, this really only works locally. For remote services the server could get priority metadata from the client's authentication credentials, but that's it.

kragen 2y ago

what does the stack look like in the callee's address space? is it a stack that previously existed (you seem to be saying it isn't) or a new one created on demand? maybe https://docs.oracle.com/cd/E88353_01/html/E37843/door-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-create... is where i should start reading...

starspangled 2y ago

Maybe you know more about it I don't want to say you're wrong because the only thing I know of it is reading the page linked to. But that page semes to disagree with you (EDIT: sorry I don't know how to best format quoting the article vs quoting your reply): Doors Implementation [...] the default server threads that are created in response to incoming door requests on the server. A new synchronization object, called a shuttle [...] allowing a direct scheduler hand-off operation between two threads. [...] a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread. Server Threads Server threads are normally created on demand by the doors library [...] The first server thread is created automatically when the server issues a door create call. [...] Once created, a server thread will place itself in a pool of available threads and wait for a door invocation. There would be no need for these server threads if the client thread transferred directly to the server process address space. > There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests It says the thread pool on the server is created by the doors library when the server creates a door. So the process of receiving and processing requests would be carried out internally within the doors library, so there would be no need for the server application to have an API for the server application to accept requests, it's handled by the library. At least that's what is described in the link, AFAIKS. It's only a conceptual door, underneath it is implemented in some message passing style maybe with some extra sugar for performance and process control nicety with this "shuddle" thing for making the requests.

kragen 2y ago
this is very confusing and now i want to see truss output man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented? but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls > The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations. <https://docs.oracle.com/cd/E88353_01/html/E37843/door-server-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...> apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl <https://www.unix.com/man-page/opensolaris/3c/door_create/ https://www.unix.com/man-page/opensolaris/3c/door_create/> /me git clone https://github.com/kofemann/opensolaris https://github.com/kofemann/opensolaris jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L193 https://github.com/illumos/illumos-gate/blob/master/usr/src/... unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used __door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist) it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/uts/common/fs/doorfs/door_sys.c#L1284 https://github.com/kofemann/opensolaris/blob/master/usr/src/... this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/sys/door.h#L142 https://github.com/illumos/illumos-gate/blob/master/usr/src/... ...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?

wang_li 2y ago

>Doors at the end of the day are a control transfer primitive. In a very real sense the calling thread is simply transferred to the callee's address space and continues execution there until a door_return(2) syscall transfers it back into the caller address_space. Your phrasing is misleading. A door is a scheduler operation. What would it even mean for a thread to go from a caller to a callee? They are different processes with completely different contexts on the system. Different owners, different address spaces, etc. What the door is doing is passing time on a CPU core from the calling thread to the receiving thread and back.

ajross 2y ago

> Doors at the end of the day are a control transfer primitive. This is true, but isomorphic to RPC, so I really don't think I understand the distinction you're trying to draw. A "procedure call" halts the "caller", passes "arguments" into the call, and "returns a value" when it completes. And that's what a door call does. Nothing about "Remote Procedure Calls" requires thread pools or a recv() system call, or an idea of "waiting for a request". Those are all just implementation details of existing RPC mechanisms implemented without a send+recv call like doors. Nor FWIW are doors alone in this space. Android Binder is effectively the same metaphor, just with a more opinionated idea of how the kernel should manage the discovery and dispatch layer (Solaris just gives you a descriptor you can call and lets userspace figure it out).

klodolph 2y ago
I thought the parent comment was drawing a very clear distinction—that doors do not involve message passing, but a different mechanism. If it were a message passing system, you would be able to save the message to disk and inspect it later, or transmit the message over the network, etc.
- ajross 2y ago
  Is this a pedantic argument about a specific definition for "message passing"? If not, then it's wrong: door_call() allows passing an arbitrary/opaque buffer of data into the call, and returning arbitrary data as part of the return. (Maybe the confusion stems from the fact that the sample code in the linked article skipped the second argument to door_call(), but you can see it documented in the man page). If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing". That, again, is just an implementation detail of other RPC mechanisms that don't implement the call as directly as doors (or binder) do.
  - klodolph 2y ago
    > Is this a pedantic argument about a specific definition for "message passing"? No. This is just colloquial usage of the term “message passing”. Yes, you can use doors to pass messages. That is not the only thing you can do. > If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing". Yeah. Everyone else here agrees with that. The original comment you replied to said, “Doors at the end of the day aren't message passing based RPC.” This absolutely indicates that the poster agrees with you on that point—that not all forms of RPC are based on a message passing system. There are other forms of RPC, and this is one of them. Ultimately, I think you can conceive of “message passing” broadly enough that, well, everything is message passing. What is a CPU register but something that receives a message and later re-transmits it? I would rather think of the colloquial use of “message passing”. Likewise, you can probably dig into doors and find some message system somewhere in the implementation. Or at least, you’ll find something you can call a message.

cryptonector 2y ago

A much better description is that doors is RPC-like but with priority and CPU quantum transfer to the server thread. This works great as long as the server thread isn't itself doing slow I/O. The moment the server thread does slow I/O you want to get back to async I/O, and the whole point of doors is that it's synchronous because only by being synchronous do you get to transfer priority and CPU quantum from the client to the server. For async RPC-like schemes you cannot transfer priority and CPU quantum to the server because the client will continue executing with that same CPU quantum (that being implied by "async"). Also, this really only works locally. For remote services the server could get priority metadata from the client's authentication credentials, but that's it.