6 ms·
"Doors" in Solaris: Lightweight RPC Using File Descriptors (1996)
- heinrichhartman 2y ago> Conceptually, during a door invocation the client thread that issues the door procedure call migrates to the server process associated with the door, and starts executing the procedure while in the address space of the server. When the service procedure is finished, a door return operation is performed and the thread migrates back to the client's address space with the results, if any, from the procedure call. Note that Server/Client refer to threads on the same machine. While I can see performance benefits of this approach, over traditional IPC (sockets, shared memory), this "opens the door" for potentially worse concurrency headaches you have with threads you spawn and control yourself. Has anyone here hands-on experience with these and can comment on how well this worked in practice?
- actionfromafar 2y agoReminds me vaguely of how Linux processes are (or were? I haven't looked at this in ages) elevated to kernel mode during a syscall. With a "door", a client is elevated to the "server" mode.
- deleted 2y ago[deleted]
- creshal 2y agoSounds like Android's binder was heavily inspired by this. Works "well" in practice in that I can't recall ever having concurrency problems, but I would not bother trying to benchmark the efficiency of Android's mess of abstraction layers piled over `/dev/binder`. It's hard to tell how much of the overhead is required to use this IPC style safely, and how much of the overhead is just Android being Android.
- p_l 2y agoNot sure which one came first, but Binder is direct descendant (down to sometimes still matching symbol names and calls) of BeOS IPC system. All the low level components (Binder, Looper, serialization model even) come from there.
- kragen 2y agowhat's the best introduction to how beos ipc worked?
- p_l 2y agoBe Book, Haiku source code, and yes Android low level internals docs. A quick look through BeOS and Android Binder-related APIs will quickly show how Android side is derived from it (through OpenBinder, which was for a time going to be used in next Palm system based on Linux, at least one of them)
- kragen 2y agothank you very much!
- stuaxo 2y agoThat's really interesting, I wonder if a Haiku on Linux could use it. Will binder ever make it into mainline Linux?
- creshal 2y agoAs far as I understand, it is already mainlined, it's just not built by "desktop" distributions since nobody really cares - all the cool kids want dbusFactorySingletonFactoryPatternSingletons to undo 20 years of hardware performance increases instead.
- p_l 2y agoBunch of desktop distros include it to run anbox/waydroid
- rerdavies 2y agoConceptually. What are they actually doing, and why is it faster than other RPC techniques?
- fch42 2y agoThink of it in terms of REST. A door is an endpoint/path provided by a service. The client can make a request to it (call it). The server can/will respond. The "endpoint" is set up via door_create(); the client connects by opening it (or receiving the open fd in other ways), and make the request by door_call(). The service sends its response by door_return(). Except that the "handover" between client and service is inline and synchronous, "nothing ever sleeps" in the process. The service needn't listen for and accept connections. The operating system "transfers" execution directly - context switches to the service, runs the door function, context switches to the client on return. The "normal" scheduling (where the server/client sleeps, becomes runnable from pending I/O and is eventually selected by the scheduler) is bypassed here and latency is lower. Purely functionality-wise, there's nothing you can do with doors that you couldn't do with a (private) protocol across pipes, sockets, HTTP connections. You "simply" use a faster/lower-latency mechanism. (I actually like the "task gate" comparison another poster made, though doors do not require a hardware-assisted context switch)
- p_l 2y agoWell, Doors' speed was derived from hardware-assisted context switching, at least on SPARC. Combination of ASIDs (which allowed task switching with reduced TLB flushing) and WIM register (which marked which register windows are valid for access by userspace) meant that IPC speed could be greatly increased - in fact that was basis for "fast path" IPC in Spring OS from which Doors were ported into Solaris.
- fch42 2y agoI was (more) of a Solaris/x86 kernel guy on that particular level and know the x86 kernel did not use task gates for doors (or any context switching other than the double fault handler). Linux did taskswitch via task gates on x86 till 2.0, IIRC. But then, hw assist or no, x86 task gates "aren't that fast". The SPARC context switch code, to me, always was very complex. The hardware had so much "sharing" (the register window set could split to multiple owners, so would the TSB/TLB, and the "MMU" was elaborate software in sparcv9 amyway). SPARC's achilles heel always were the "spills" - needless register window (and other cpu state) to/from memory. I'm kinda still curious from a "historical" point of view - thanks!
- akira2501 2y ago> this "opens the door" for potentially worse concurrency headaches you have with threads you spawn and control yourself. What makes doors "potentially worse" than regular threads?
- wahern 2y agoIIUC, what they mean by "migrate" is the client thread is paused and the server thread given the remainder of the time slice, similar to how pipe(2) originally worked in Unix and even, I think, early Linux. It's the flow of control that "conceptually" shifts synchronously. This can provide surprising performance benefits in alot of RPC scenarios, though less now as TLB, etc, flushing as part of a context switch has become more costly. There are no VM shenanigans except for some page mapping optimizations for passing large chunks of data, which apparently wasn't even implemented in the original Solaris implementation. The kernel can spin up a thread on the server side, but this works just like common thread pool libraries, and I'm not sure the kernel has any special role here except to optimize context switching when there's no spare thread to service an incoming request and a new thread needs to be created. With a purely userspace implementation there may be some context switch bouncing unless an optimized primitive (e.g. some special futex mode, perhaps?) is available. Other than maybe the file namespace attaching API (not sure of the exact semantics), and presuming I understand properly, I believe Doors, both functionally and the literal API, could be implemented entirely in userspace using Unix domain sockets, SCM_RIGHTS, and mmap. It just wouldn't have the context switching optimization without new kernel work. (See the switchto proposal for Linux from Google, though that was for threads in the same process.) I'm basing all of this on the description of Doors at https://web.archive.org/web/20121022135943/https://blogs.oracle.com/meem/entry/head_title_dsvclockd_1m_using https://web.archive.org/web/20121022135943/https://blogs.ora... and http://www.rampant.org/doors/linux-doors.pdf http://www.rampant.org/doors/linux-doors.pdf
- gpderetta 2y agoI think what doors do is rendezvous synchronization: the caller is atomically blocked as the callee is unblocked (and vice versa on return). I don't think there is an efficient way to do that with just plain POSIX primitives or even with Linux specific syscalls (Binder and io_uring possibly might).
- xoranth 2y agoSounds a bit like Google's proposal for a `switchto_switch` syscall [1] that would allow for cooperative multithreading bypassing the scheduler. (the descendants of that proposal is `sched_ext`, so maybe it is possible to implement doors in eBPF + sched_ext?) [1]: https://youtu.be/KXuZi9aeGTw?t=900 https://youtu.be/KXuZi9aeGTw?t=900
- blacklion 2y agoConceptually is key word here. Later in this article authors says tat Server manage its own pool (optionally bounded) of threads to serve requests.
- p_l 2y agoThe thread in this context refers to kernel scheduler thread[1], essentially the entity used to schedule user processes. By migrating the thread, the calling process is "suspended", it's associated kernel thread (and thus scheduled time quanta, run queue position, etc.) saves the state into Door "shuttle", picks up the server process, continues execution of the server procedure, and when the server process returns from the handler, the kernel thread picks up the Door "shuttle", restores the right client process state from it, and lets it continue - with the result of the IPC call. This means that when you do a Door IPC call, the service routine is called immediately, not at some indefinite point in time in the future when the server process gets picked by scheduler to run and finds out an event waiting for it on select/poll kind of call. If the service handler returns fast enough, it might return even before client process' scheduler timeslice ends. The rapid changing of TLB etc. are mitigated by hardware features in CPU that permit faster switches, something that Sun had already experience with at the time from the Spring Operating System project - from which the Doors IPC in fact came to be. Spring IPC calls were often faster than normal x86 syscalls at the time (timings just on the round trip: 20us on 486DX2 for typical syscall, 11us for sparcstation Spring IPC, >100us for Mach syscall/IPC) EDIT: [1] Some might remember references to 1:1 and M:N threading in the past, especially in discussions about threading support in various unices, etc. The "1:1" originally referred to relationship between "kernel" thread and userspace thread, where kernel thread didn't mean "posix like thread in kernel" and more "the scheduler entity/concept", whether it was called process, thread, or "lightweight process"
- grishka 2y agoWhy would you care who spawned the thread? If your code is thread-safe, it shouldn't make a difference. One potential problem with regular IPC I see is that it's nondeterministic in terms of performance/throughput because you can't be sure when the scheduler will decide to run the other side of whatever IPC mechanism you're using. With these "doors", you bypass scheduling altogether, you call straight "into" the server process thread. This may make a big difference for systems under load.
- fuhh 2y ago[flagged]
- robinhouston 2y agoBack in 1998 or so, a colleague and I were tasked with building a system to target adverts to particular users on a website. It obviously needed to run very efficiently, because it would be invoked on every page load and the server hardware of the time was very underpowered by today’s standards. The Linux server revolution was still a few years away (at least it was for us – perhaps we were behind the curve), so our webservers were all Sun Enterprise machines running Solaris. We decided to use a server process that had an in-memory representation of the active users, which we would query using doors from a custom Apache module. (I had read about doors and thought they sounded cool, but neither of us had used them before.) It worked brilliantly and was extremely fast, though in the end it was never used in production because of changing business priorities.
- codetrotter 2y agoSolaris was ahead of its time. They had Zones, which was a container technology, years ago. Likewise, FreeBSD way ahead with Jails. Only later did Docker on Linux come into existence and then containerization exploded in popularity.
- layer8 2y agoIn the early 2000s I rented a managed VPS based on FreeBSD jails. This was nice because you were root in your jail while the distribution was managed for you (security updates in particular), and you could still override any system files you wanted via the union file system. It was like the best of both worlds between a managed Unix account and a VPS where you can do anything you want as root.
- bionsystem 2y agoAND you are running bare metal, without the virtualization layer overhead, which is quite nice especially in the early 2000s :) My first project as a junior was to deploy Solaris Zones to replace a bunch of old SPARC machines. It's such a great tech and was a fun project.
- hnlmorg 2y agoLinux had a few different containerisation technologies before Docker blew up. But even they felt like a step backwards from Jails and Zones. Personally I think its more fair to say that Linux was behind the times because containerisation and virtualisation were common concepts in UNIXes long before they became Linux staples.
- stormking 2y agoWell, Docker did a few things differently then these older approaches. First, the one-process-per-container mantra, second the layered file system and build process and third, the "wireing together" of multiple containers via exposed ports. Zones and Jails were more like LXC and were mostly used as a "lightweight virtual machine".
- panick21_ 2y ago
- danwills 2y agoSomehow I'm only just noticing the name-conflict with SideFX Houdini's new(ish) USD context, which is also called 'Solaris'! .. Guess I don't search for the old SunSoft kind of Solaris much these days eh!
- rbanffy 2y agoYou probably noticed that before Oracle's legal dept.
- lukeh 2y ago1998 paper on Linux implementation: http://www.rampant.org/doors/linux-doors.pdf http://www.rampant.org/doors/linux-doors.pdf
- oecumena 2y agoI did another one back in 2001: https://ldoor.sourceforge.net/ https://ldoor.sourceforge.net/ .
- monocasa 2y agoWriting a top level comment here to hopefully address some of the misconceptions present across this comment section. Doors at the end of the day aren't message passing based RPC. There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests. Doors at the end of the day are a control transfer primitive. In a very real sense the calling thread is simply transferred to the callee's address space and continues execution there until a door_return(2) syscall transfers it back into the caller address_space. It truly is a 'door' into another address space. This is most similar to some of the CPU control transfer primitives. It's most like task gate style constructs like seen on 286/i432/mill CPUs. Arguably it's kind of like the system call instruction itself too, transferring execution directly to another address space/context.
- kragen 2y agowhat does the stack look like in the callee's address space? is it a stack that previously existed (you seem to be saying it isn't) or a new one created on demand? maybe https://docs.oracle.com/cd/E88353_01/html/E37843/door-create-3c.html https://docs.oracle.com/cd/E88353_01/html/E37843/door-create... is where i should start reading...
- starspangled 2y agoMaybe you know more about it I don't want to say you're wrong because the only thing I know of it is reading the page linked to. But that page semes to disagree with you (EDIT: sorry I don't know how to best format quoting the article vs quoting your reply): Doors Implementation [...] the default server threads that are created in response to incoming door requests on the server. A new synchronization object, called a shuttle [...] allowing a direct scheduler hand-off operation between two threads. [...] a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread. Server Threads Server threads are normally created on demand by the doors library [...] The first server thread is created automatically when the server issues a door create call. [...] Once created, a server thread will place itself in a pool of available threads and wait for a door invocation. There would be no need for these server threads if the client thread transferred directly to the server process address space. > There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests It says the thread pool on the server is created by the doors library when the server creates a door. So the process of receiving and processing requests would be carried out internally within the doors library, so there would be no need for the server application to have an API for the server application to accept requests, it's handled by the library. At least that's what is described in the link, AFAIKS. It's only a conceptual door, underneath it is implemented in some message passing style maybe with some extra sugar for performance and process control nicety with this "shuddle" thing for making the requests.
- bux93 2y agoI wonder if the name is related to BBS doors? https://en.wikipedia.org/wiki/Door_(bulletin_board_system) https://en.wikipedia.org/wiki/Door_(bulletin_board_system)
- kragen 2y agoi don't think so; i think people independently chose the name 'door' for a way to get out of one program and into another in both cases. bbs people and sun labs kernel researchers didn't talk to each other that much unfortunately, and the mechanisms aren't really that similar
- rbanffy 2y agoI wonder if anyone ended up saving a copy of the Spring operating system.
- quotemstr 2y agoNobody has ever done IPC better than Microsoft did with COM/RPC/AIPC. Nobody else even came close. I will die on this hill. The open source world has done itself a tremendous disservice eschewing object capability systems with in-process bypasses.
- debo_ 2y agoI didn't realize "staff engineer" was a common software title as far back as 1996.
- crest 2y agoI wish FreeBSD and Linux had Solaris style doors extended to support kqueue()/epoll() in addition to threads.
- cryptonector 2y agoDoors are designed to be synchronous. Therefore doors are not really compatible with kqueue/epoll/event ports. That is, the point of doors is that they are for clients to temporarily and synchronously transfer thread priority and CPU quantum to a server thread that will respond quickly because it does no blocking I/O. This transfer of CPU quantum thing cannot happen for async versions of doors, which would defeat the point of doors, thus there is no point to async doors. One can implement async doors by having the client transfer the FD to a door where the client is the service, then have the server do what it can synchronously, schedule async work, return immediately, and later call back the client on its door when the slow work is done. But if you'd do this then you might as well just use more traditional (and portable) IPC techniques.
- hksgdk 2y ago[dead]