5 ms·

It's there really some reason not to generalize IPC to RPC "between processes"? To be more specific, is there really a justifiable specific of either of these

by vletal 5y ago

It's there really some reason not to generalize IPC to RPC "between processes"? To be more specific, is there really a justifiable specific of either of these applications which prevent us from having a single tool which would cover both?

surajrmal 5y ago
Sure. In IPC, you can have shared resources such as memory. In RPC, you need to assume all parties may be on separate systems. In IPC, you may be able to make guarantees about message ordering that are more challenging with RPC. For instance, packets may be dropped over a network link and packets may arrive out of order. In IPC, you can express backpressure without the need for packet dropping, and therefore can avoid overhead of things like TCP. With IPC, links are trusted, so you can avoid encrypting messages, but that is not something you can safely assume with RPC. Because of these sorts of things, the cost of sending a message is relatively low (measured in us) compared to RPC (measured in hundreds of us), and so the cost of serialization may be of greater concern as it may start to dominate cost. As a result, you may optimize your serialization format differently. Beyond that, an IPC system may allow for advanced usage of the shared kernel which is not possible across different systems each with a separate kernel.
- spc476 5y ago
  QNX makes no distinction between IPC and RPC. In fact, a program won't be able to tell if the other end of a message port was on the same CPU, same computer, or network (well, maybe by timing results). You didn't even need to recompile code to get that either. It just works right out of the box.
  - surajrmal 5y ago
    When you do this, you have to live by the combined constraints of both transports. It's not necessarily wrong, but having different mechanisms which are optimized for the specific constraints of each is also valid.
    - ithkuil 5y ago
      I wonder how many of those constraints actually apply to the IDL and surrounding programming model as opposed to the mere (de)serialization. Could we have a single IDL that generates two sets of serialization code one optimized for IPC and the other for network transport?
      - pjmlp 5y ago
        COM, .NET Remote and RMI do it.
      - surajrmal 5y ago
        I'll give you a more concrete experience. We have protocols designed to take advantage of shared memory to allow dma directly from the hardware to the pages backing the file you end up reading from. There are many layers/processes between my program and the driver which ultimately talks to the hardware, and being able to bypass copies for the data is useful. Now the protocol encodes this expectation of shared memory usage within it. We later built a mechanism to take existing protocols and use them across network boundaries, but were unable to utilize any protocols which relied on shared memory. It may be possible to emulate shared memory across the network boundary, but it's hard to do so performantly in all cases. So rather than modify the existing protocol to avoid shared memory, negatively affecting existing use cases, we opted to create a second protocol which was optimized for network use cases. There are more of these sorts of examples I could enumerate if you find it worthwhile.
    - Kinrany 5y ago
      It should be possible to encode those constraints as part of the language. For example, the distinction between sync and async calls can be represented by having a Future type and wrapping the return type in it.
      - znwu 5y ago
        Sync and async are all about cooperatively yielding control flow. However, in many cases, you may want to yield control on IPC, or to hold onto control on RPC. Yielding control depends on the sender logic. IPC/RPC depends on receiver characteristics. They are very orthogonal concepts.
  - sriram_sun 5y ago
    QNX is able to do this because QNX doesn't understand shared memory. Everything is a message/channel.
- guiriduro 5y ago
  I'm struggling to see why you'd want a specialised explicit IDL for IPC-only, as against it being an implementation detail using same address-space optimisation if services are co-located. E.g. cf OmniORB (CORBA) [1] While you can have tightly-coupled, highly interactive IPC that would be pathological over RPC, where DSL/IDL and protocol semantics rightly focus on more loosly-coupled service interfaces with network error recovery, I question whether it is desirable even in the IPC case. Intra-service IPC within a single runtime can be as chatty as needed without becoming an Inter-service call (and potentially RPC in other architectures), so does Fuschia answer to an actual need - is there actually a valid use case for tight-coupled highly interactive local inter-service IPC that isn't better architected assuming RPC? Is the benefit of assumed IPC between co-located client/servers an actual problem that 'colocation optimised' RPC doesn't already solve well enough? [1] https://www.omniorb-support.com/pipermail/omniorb-list/2006-December/028275.html https://www.omniorb-support.com/pipermail/omniorb-list/2006-...
  - kllrnohj 5y ago
    RPCs aren't just slower for IPC, they are also missing very useful features. Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC. It's how something like an "open file dialog" can work without needing the app process to have full access to all the user's files.
    - guiriduro 5y ago
      Well, thinking around why you'd pass around a local FD among some processes via IPC, what are the typical use cases? As a source/sink in a chain of piped processes? the connections are the pipes, not the file except at the ends. Perhaps several services cooperating to differentially process parts of a file? you'd probably architect that as a FileService that would wrap the FD and trigger events or stream to individual processing services, and here there's no fundamental difference if it runs RPC or IPC (although IPC throughput can be optimised of course, but done transparently by the ORB.) Whereas an FD-passing IPC solution is locked to local processing anyway (and only vertical scalability), and if you're already paying the price to write IDL for that, isn't the RPC/IPC style solution above both more flexible (allows hz scale), not significantly more complex or locking you into yet another IDL ?
kjeetgill 5y ago
It's a surprisingly readable doc, check it out [0]. Compared to protobufs, it's much more concerned with being more fixed width, word aligned, efficient copy-free in-memory accessable. Proto3 spends a lot of effort compressing ints on one hand but remaining flexible and allowing fields to be optional on the other. On top of all that, there's a good deal that's specific to it's kernel, Zircon, it's permission/capabilities model, and how these messages interact with their syscalls. I'd imagine these kinda details are going to matter a lot for a microkernel. I don't know it as deeply but I'd say it's much closer to cap'n proto. Not sure who uses that though! [0]: https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire-format https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire...

by vletal 5y ago

surajrmal 5y ago

Sure. In IPC, you can have shared resources such as memory. In RPC, you need to assume all parties may be on separate systems. In IPC, you may be able to make guarantees about message ordering that are more challenging with RPC. For instance, packets may be dropped over a network link and packets may arrive out of order. In IPC, you can express backpressure without the need for packet dropping, and therefore can avoid overhead of things like TCP. With IPC, links are trusted, so you can avoid encrypting messages, but that is not something you can safely assume with RPC. Because of these sorts of things, the cost of sending a message is relatively low (measured in us) compared to RPC (measured in hundreds of us), and so the cost of serialization may be of greater concern as it may start to dominate cost. As a result, you may optimize your serialization format differently. Beyond that, an IPC system may allow for advanced usage of the shared kernel which is not possible across different systems each with a separate kernel.

spc476 5y ago
QNX makes no distinction between IPC and RPC. In fact, a program won't be able to tell if the other end of a message port was on the same CPU, same computer, or network (well, maybe by timing results). You didn't even need to recompile code to get that either. It just works right out of the box.
- surajrmal 5y ago
  When you do this, you have to live by the combined constraints of both transports. It's not necessarily wrong, but having different mechanisms which are optimized for the specific constraints of each is also valid.
  - ithkuil 5y ago
    I wonder how many of those constraints actually apply to the IDL and surrounding programming model as opposed to the mere (de)serialization. Could we have a single IDL that generates two sets of serialization code one optimized for IPC and the other for network transport?
    - pjmlp 5y ago
      COM, .NET Remote and RMI do it.
    - surajrmal 5y ago
      I'll give you a more concrete experience. We have protocols designed to take advantage of shared memory to allow dma directly from the hardware to the pages backing the file you end up reading from. There are many layers/processes between my program and the driver which ultimately talks to the hardware, and being able to bypass copies for the data is useful. Now the protocol encodes this expectation of shared memory usage within it. We later built a mechanism to take existing protocols and use them across network boundaries, but were unable to utilize any protocols which relied on shared memory. It may be possible to emulate shared memory across the network boundary, but it's hard to do so performantly in all cases. So rather than modify the existing protocol to avoid shared memory, negatively affecting existing use cases, we opted to create a second protocol which was optimized for network use cases. There are more of these sorts of examples I could enumerate if you find it worthwhile.
  - Kinrany 5y ago
    It should be possible to encode those constraints as part of the language. For example, the distinction between sync and async calls can be represented by having a Future type and wrapping the return type in it.
    - znwu 5y ago
      Sync and async are all about cooperatively yielding control flow. However, in many cases, you may want to yield control on IPC, or to hold onto control on RPC. Yielding control depends on the sender logic. IPC/RPC depends on receiver characteristics. They are very orthogonal concepts.
- sriram_sun 5y ago
  QNX is able to do this because QNX doesn't understand shared memory. Everything is a message/channel.
guiriduro 5y ago
I'm struggling to see why you'd want a specialised explicit IDL for IPC-only, as against it being an implementation detail using same address-space optimisation if services are co-located. E.g. cf OmniORB (CORBA) [1] While you can have tightly-coupled, highly interactive IPC that would be pathological over RPC, where DSL/IDL and protocol semantics rightly focus on more loosly-coupled service interfaces with network error recovery, I question whether it is desirable even in the IPC case. Intra-service IPC within a single runtime can be as chatty as needed without becoming an Inter-service call (and potentially RPC in other architectures), so does Fuschia answer to an actual need - is there actually a valid use case for tight-coupled highly interactive local inter-service IPC that isn't better architected assuming RPC? Is the benefit of assumed IPC between co-located client/servers an actual problem that 'colocation optimised' RPC doesn't already solve well enough? [1] https://www.omniorb-support.com/pipermail/omniorb-list/2006-December/028275.html https://www.omniorb-support.com/pipermail/omniorb-list/2006-...
- kllrnohj 5y ago
  RPCs aren't just slower for IPC, they are also missing very useful features. Like it's not possible to send an open file descriptor across RPC, but it's very possible (and very useful) to do the same over IPC. It's how something like an "open file dialog" can work without needing the app process to have full access to all the user's files.
  - guiriduro 5y ago
    Well, thinking around why you'd pass around a local FD among some processes via IPC, what are the typical use cases? As a source/sink in a chain of piped processes? the connections are the pipes, not the file except at the ends. Perhaps several services cooperating to differentially process parts of a file? you'd probably architect that as a FileService that would wrap the FD and trigger events or stream to individual processing services, and here there's no fundamental difference if it runs RPC or IPC (although IPC throughput can be optimised of course, but done transparently by the ORB.) Whereas an FD-passing IPC solution is locked to local processing anyway (and only vertical scalability), and if you're already paying the price to write IDL for that, isn't the RPC/IPC style solution above both more flexible (allows hz scale), not significantly more complex or locking you into yet another IDL ?

kjeetgill 5y ago

It's a surprisingly readable doc, check it out [0]. Compared to protobufs, it's much more concerned with being more fixed width, word aligned, efficient copy-free in-memory accessable. Proto3 spends a lot of effort compressing ints on one hand but remaining flexible and allowing fields to be optional on the other. On top of all that, there's a good deal that's specific to it's kernel, Zircon, it's permission/capabilities model, and how these messages interact with their syscalls. I'd imagine these kinda details are going to matter a lot for a microkernel. I don't know it as deeply but I'd say it's much closer to cap'n proto. Not sure who uses that though! [0]: https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire-format https://fuchsia.dev/fuchsia-src/reference/fidl/language/wire...