4 ms·

We're in this situation because we're using a model of dynamic linking that's decades out of date. Why aren't we using process-isolated sandboxed components tal

We're in this situation because we're using a model of dynamic linking that's decades out of date. Why aren't we using process-isolated sandboxed components talking over io_uring-based low-latency IPC to express most software dependencies? The vast majority of these dependencies absolutely do not need to be co-located with their users. Consider liblzma: would liblzma-as-a-service really be that bad, especially if the service client and service could share memory pages for zero-copy data transfer, just as we already do for, e.g. video decode? Or consider React Native: RN works by having an application thread send a GUI scene to a renderer thread, which then adjusts a native widget tree to match what the GUI thread wants. Why do these threads have to be in the same process? You're doing a thread switch anyway to jump from the GUI thread to the renderer thread: is switching address spaces at the same time going to kill you? Especially if the two threads live on different cores and nothing has to "switch"? Both dynamic linking and static linking should be rare in modern software ecosystems. We need to instead reinvigorate the idea of agent-based component systems with strongly isolated components.

rwmj 3y ago
Essentially path dependency. It would be almost impossible to change how it works now.
- vlovich123 3y ago
  I don't think this stuff is as hard as you make it out to be. Consider that companies like Apple regularly change how things are done successfully. It just requires a good plan, time, & budget. Wayland is one example of what that looks like & it's not a good story. Pulse audio followed by pipewire is another example of migrations happening. I suspect this would probably be slightly worse than Wayland unless some kind of transparent shim could be written for each boundary so that it can be slotted in transparently.
  - Analemma_ 3y ago
    Apple can regularly change how things are done because they have absolute control over their platform and use an "our way or the highway" approach to breaking changes, where developers have to go along or lose access to a lucrative market. This approach really really would not work on Linux: consider that the rollout of systemd was one-tenth as dictatorial as is SOP for changes from Apple, and it caused legions of Linux users to scream for Poettering's head on a stick.
    - vlovich123 3y ago
      That's not a path dependency though. That's just a critique of the bazaar development model. And honestly I think if the big distros got together and agreed this would be a significant security improvement, they could drag the community kicking & screaming just like they did with systemd (people hated systemd so much at first that they tried to get other init systems to not suck but over time persistent effort wins out).
dosshell 3y ago
This is very interesting! Are there any movements to move towards this? Wouldn't it open up for a new attack vector where process could read each other data?
ajross 3y ago
> Why aren't we using process-isolated sandboxed components talking over io_uring-based low-latency IPC to express most software dependencies? To some extent we are, if what you do is work on backend RPC or web app frameworks. But the better answer is because sometimes what you actually want is the ability to put a C function in a separate file that can be versioned and updated on its own, which is what a shared library captures. Trying to replace a function call of 2-3 instructions with your io_uring monstrosity is... suboptimal for a lot of applications. And in any case, the protocol parsing you'd need to provide to enable all that RPC is going to need to live somewhere, right? What is that going to be, other than a shared library or equivalent?
otabdeveloper4 3y ago
> is switching address spaces at the same time going to kill you? The answer is "yes". I won't stop you, if you want to make React even slower, be my guest. I want off this ride.
- quotemstr 3y ago
  > The answer is "yes". And if you have one per core anyway so nothing "switches"? Computers aren't single-core 80486es anymore. We have highly parallel machines nowadays and old intuition about what's expensive and what's cheap decays by the year.
  - JackSlateur 3y ago
    You know that process A (the caller) is currently running. You assume that process B (the callee) is running in another core on the same NUMA nodes.
  - otabdeveloper4 3y ago
    Sorry, not dedicating a whole CPU core for your shitty React app.
  - josephg 3y ago
    I only have 16 cores. Linux, windows and macOS already load about 50+ processes at startup. If we moved shared libraries into their own processes, we’d be talking hundreds or thousands of processes running all the time. They don’t get a core each. But, if you’re interested in this architecture, smalltalk did something similar. Fire up a smalltalk vm and play around!
jcelerier 3y ago
currently involved professionally in a software architecture based on pretty much raw shared memory IPC, it's still too slow compared to in-process. See also VST hosts that allow grouping plug-ins together in one process or separating them in distinct processes, like Bitwig: for just a few dozen plug-ins you can very easily get 10+% of CPU impact (and CPU is an extremely dire commodity when making pro audio, it's pretty much a constant fight against high CPU usage in larger music making sessions)
- quotemstr 3y ago
  > it's still too slow compared to in-process Why? Relative to the in-process case, properly done multi-process data flow pipelines don't necessarily incur extra copies. Sure, switching to a different process is somewhat more expensive than switching to a different thread due to page table changes, but if you're doing bulk data processing, you amortize any process-separation-driven costs across lots of compute anyway --- and in a many-core world, you can run different parts of your system on different cores anyway and get away with not paying context-switch costs at all. Also, 10% is actually a pretty modest price to pay for increased software robustness and modularity. We're paying more than that for speculative execution vulnerability anyway. Do you run your fancy low-level audio processing pipeline with "mitigations=off" in /proc/cmdline?
  - jcelerier 3y ago
    > Also, 10% is actually a pretty modest price to pay for increased software robustness and modularity. We're paying more than that for speculative execution vulnerability anyway. it's a completely crazy price to pay in a field where people routinely spend thousands of $$$ for <5% improvement > Do you run your fancy low-level audio processing pipeline with "mitigations=off" in /proc/cmdline? obviously yes! along with power saving CPU C-states or anything throttling-related disabled, specific real-time IRQ and threading configuration (e.g. making sure that the sound card interrupts aren't going to happen on a core handling network interrupts) and two dozen other optimizations (which do make a difference, I regularly set-up new machines from scratch for shows, art installations, etc. and always do this setup step-by-step to see if things are finally "good enough" and they always make a difference, in really a make-or-break sense).
bno1 3y ago
A big issue with IPC is thread scheduling. Thread B needs to get scheduled to see the request from thread A and thread A needs to get scheduled to see the response from thread B. I think there are WIP solutions to deal with this [1] this but I'm not up to date. [1] https://www.phoronix.com/news/Google-User-Thread-Futex-Swap https://www.phoronix.com/news/Google-User-Thread-Futex-Swap
rini17 3y ago
Someone got a microservice hammer, so everything looks like a nail eh? What is really needed, is sane memory model where you can easily call any function with buffers (pointer + size) and it is allowed to access only these buffers and nothing else(note). Not this mess coming from C where this is difficult by design. (note)since HN likes to split hairs: except for its private storage and other well thought exceptions
- ptx 3y ago
  This would be what's known as software-based fault isolation, right? Here's a paper from 1993: https://dl.acm.org/doi/abs/10.1145/168619.168635 https://dl.acm.org/doi/abs/10.1145/168619.168635 I don't understand why this idea keeps failing to take hold even though it's constantly reintroduced in various forms. Surely now, 30 years after that paper was published, we can bear the "slightly increased execution time for distrusted modules" in return for (as the paper suggests) faster communication between isolated modules?
- o11c 3y ago
  So basically like that one Rust hardware project that gets posted periodically? You could probably do it pretty decently in C via `pkey_mprotect` (probably with `dlmopen`). https://www.man7.org/linux/man-pages/man7/pkeys.7.html https://www.man7.org/linux/man-pages/man7/pkeys.7.html
  - quotemstr 3y ago
    It'd be nice if dlmopen weren't broken and made a linker namespace so separate that it couldn't even share pthreads with another.
JackSlateur 3y ago
You would replace function calls with syscalls ? Yeah well, if you omit performance and complexity, why not .. io_uring is nice. Yet my application have to call that lzma function and get the result now : you now add cross-process synchronization (via the kernel => syscall) as well as insert the scheduler in the mix
pjmlp 3y ago
Because hardware resources, 20 years ago doing something like VSCode with tons of external processes per plugin, would drag your computer to a crawl. Emacs wasn't Eight Megabytes and Constantly Swapping only due to Elisp.
marcosdumay 3y ago
That's because IPC is not low-latency. No modern processor architecture has a proper message passing mechanism. All of them expect you to use interruptions; with it's inherent problems of losing cache, disrupting pipelines, and well, interrupting your process flow. All the modern architectures are also so close to have a proper message passing mechanism that it's unsettling. You actually need this to have uniform memory in a multi-core CPU. They have all the mechanisms for zero copy sharing of memory, enforcing coherence, atomicity, etc. AFAIK, they just lack a userspace mechanism to signal other processes.
- oasisaimlessly 3y ago
  Futexes allow low-latency signalling to other processes, and FUTEX_SWAP [1, 2] promises to decrease the cost even further. [1]: https://lore.kernel.org/lkml/20200722234538.166697-1-posk@posk.io/ https://lore.kernel.org/lkml/20200722234538.166697-1-posk@po... [2]: https://www.phoronix.com/news/Google-User-Thread-Futex-Swap https://www.phoronix.com/news/Google-User-Thread-Futex-Swap
  - marcosdumay 3y ago
    This doesn't seem to touch on any of the points on my comment. It does take a lot of unnecessary stuff out of the way. But isn't enough to change the picture at all.
    - quotemstr 3y ago
      > This doesn't seem to touch on any of the points on my comment. What points did your comment make? You didn't define "signalling" specifically enough to discuss. Can you elaborate on precisely what kind of "signaling" primitive processors or operating systems should provide?
zbentley 3y ago
This was roughly the dream of DBus. However, outside of desktop-shaped niches it proved to be extremely difficult to secure, standardize, and debug. Process-level/address-space-level dependency sharing remains both easier to think about and simpler to implement (and capabilities are taking bites out of the security risks entailed by this model as time goes on).
josephg 3y ago
If the goal is to put a security boundary within the process between libraries, there might be better ways to do it than process boundaries. One approach is to wasm sandbox library code. Firefox apparently does this - compiling some libraries to wasm, then compiling the wasm back to C and linking it. They get all the benefits of wasm but without any need to JIT compile the code. Another approach would be to leverage a language like rust. I’d love it if rust provided a way to deny any sensitive access to part of my dependency tree. I want to pull a library but deny it the ability to run unsafe code or make any syscall (or maybe, make syscalls but I’ll whitelist what it’s allowed to call). Restrictions should be transitive to all of that library’s dependencies (optionally with further restrictions). Both of these approaches would stop the library from doing untoward things. Way more so than you’d get running the library in a separate process.
magicalhippo 3y ago
Think you would have loved Signularity OS[1]. It featured software-isolated processes that communicated via contract-based message passing, which allowed for zero-copy exchange of data. As a research OS it never became a fully-fledged OS[2], but an interesting attempt IMHO. [1]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/osr2007_rethinkingsoftwarestack.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... [2]: https://en.wikipedia.org/wiki/Singularity_%28operating_system%29 https://en.wikipedia.org/wiki/Singularity_%28operating_syste...
zokier 3y ago
Dbus is a thing. Bus1 was its still-born spiritual kernel-space successor.

rwmj 3y ago

Essentially path dependency. It would be almost impossible to change how it works now.

vlovich123 3y ago
I don't think this stuff is as hard as you make it out to be. Consider that companies like Apple regularly change how things are done successfully. It just requires a good plan, time, & budget. Wayland is one example of what that looks like & it's not a good story. Pulse audio followed by pipewire is another example of migrations happening. I suspect this would probably be slightly worse than Wayland unless some kind of transparent shim could be written for each boundary so that it can be slotted in transparently.
- Analemma_ 3y ago
  Apple can regularly change how things are done because they have absolute control over their platform and use an "our way or the highway" approach to breaking changes, where developers have to go along or lose access to a lucrative market. This approach really really would not work on Linux: consider that the rollout of systemd was one-tenth as dictatorial as is SOP for changes from Apple, and it caused legions of Linux users to scream for Poettering's head on a stick.
  - vlovich123 3y ago
    That's not a path dependency though. That's just a critique of the bazaar development model. And honestly I think if the big distros got together and agreed this would be a significant security improvement, they could drag the community kicking & screaming just like they did with systemd (people hated systemd so much at first that they tried to get other init systems to not suck but over time persistent effort wins out).

dosshell 3y ago

This is very interesting! Are there any movements to move towards this? Wouldn't it open up for a new attack vector where process could read each other data?

ajross 3y ago

> Why aren't we using process-isolated sandboxed components talking over io_uring-based low-latency IPC to express most software dependencies? To some extent we are, if what you do is work on backend RPC or web app frameworks. But the better answer is because sometimes what you actually want is the ability to put a C function in a separate file that can be versioned and updated on its own, which is what a shared library captures. Trying to replace a function call of 2-3 instructions with your io_uring monstrosity is... suboptimal for a lot of applications. And in any case, the protocol parsing you'd need to provide to enable all that RPC is going to need to live somewhere, right? What is that going to be, other than a shared library or equivalent?

otabdeveloper4 3y ago

> is switching address spaces at the same time going to kill you? The answer is "yes". I won't stop you, if you want to make React even slower, be my guest. I want off this ride.

quotemstr 3y ago
> The answer is "yes". And if you have one per core anyway so nothing "switches"? Computers aren't single-core 80486es anymore. We have highly parallel machines nowadays and old intuition about what's expensive and what's cheap decays by the year.
- JackSlateur 3y ago
  You know that process A (the caller) is currently running. You assume that process B (the callee) is running in another core on the same NUMA nodes.
- otabdeveloper4 3y ago
  Sorry, not dedicating a whole CPU core for your shitty React app.
- josephg 3y ago
  I only have 16 cores. Linux, windows and macOS already load about 50+ processes at startup. If we moved shared libraries into their own processes, we’d be talking hundreds or thousands of processes running all the time. They don’t get a core each. But, if you’re interested in this architecture, smalltalk did something similar. Fire up a smalltalk vm and play around!

jcelerier 3y ago

currently involved professionally in a software architecture based on pretty much raw shared memory IPC, it's still too slow compared to in-process. See also VST hosts that allow grouping plug-ins together in one process or separating them in distinct processes, like Bitwig: for just a few dozen plug-ins you can very easily get 10+% of CPU impact (and CPU is an extremely dire commodity when making pro audio, it's pretty much a constant fight against high CPU usage in larger music making sessions)

quotemstr 3y ago
> it's still too slow compared to in-process Why? Relative to the in-process case, properly done multi-process data flow pipelines don't necessarily incur extra copies. Sure, switching to a different process is somewhat more expensive than switching to a different thread due to page table changes, but if you're doing bulk data processing, you amortize any process-separation-driven costs across lots of compute anyway --- and in a many-core world, you can run different parts of your system on different cores anyway and get away with not paying context-switch costs at all. Also, 10% is actually a pretty modest price to pay for increased software robustness and modularity. We're paying more than that for speculative execution vulnerability anyway. Do you run your fancy low-level audio processing pipeline with "mitigations=off" in /proc/cmdline?
- jcelerier 3y ago
  > Also, 10% is actually a pretty modest price to pay for increased software robustness and modularity. We're paying more than that for speculative execution vulnerability anyway. it's a completely crazy price to pay in a field where people routinely spend thousands of $$$ for <5% improvement > Do you run your fancy low-level audio processing pipeline with "mitigations=off" in /proc/cmdline? obviously yes! along with power saving CPU C-states or anything throttling-related disabled, specific real-time IRQ and threading configuration (e.g. making sure that the sound card interrupts aren't going to happen on a core handling network interrupts) and two dozen other optimizations (which do make a difference, I regularly set-up new machines from scratch for shows, art installations, etc. and always do this setup step-by-step to see if things are finally "good enough" and they always make a difference, in really a make-or-break sense).

bno1 3y ago

A big issue with IPC is thread scheduling. Thread B needs to get scheduled to see the request from thread A and thread A needs to get scheduled to see the response from thread B. I think there are WIP solutions to deal with this [1] this but I'm not up to date. [1] https://www.phoronix.com/news/Google-User-Thread-Futex-Swap https://www.phoronix.com/news/Google-User-Thread-Futex-Swap

rini17 3y ago

Someone got a microservice hammer, so everything looks like a nail eh? What is really needed, is sane memory model where you can easily call any function with buffers (pointer + size) and it is allowed to access only these buffers and nothing else(note). Not this mess coming from C where this is difficult by design. (note)since HN likes to split hairs: except for its private storage and other well thought exceptions

ptx 3y ago
This would be what's known as software-based fault isolation, right? Here's a paper from 1993: https://dl.acm.org/doi/abs/10.1145/168619.168635 https://dl.acm.org/doi/abs/10.1145/168619.168635 I don't understand why this idea keeps failing to take hold even though it's constantly reintroduced in various forms. Surely now, 30 years after that paper was published, we can bear the "slightly increased execution time for distrusted modules" in return for (as the paper suggests) faster communication between isolated modules?
o11c 3y ago
So basically like that one Rust hardware project that gets posted periodically? You could probably do it pretty decently in C via `pkey_mprotect` (probably with `dlmopen`). https://www.man7.org/linux/man-pages/man7/pkeys.7.html https://www.man7.org/linux/man-pages/man7/pkeys.7.html
- quotemstr 3y ago
  It'd be nice if dlmopen weren't broken and made a linker namespace so separate that it couldn't even share pthreads with another.

JackSlateur 3y ago

You would replace function calls with syscalls ? Yeah well, if you omit performance and complexity, why not .. io_uring is nice. Yet my application have to call that lzma function and get the result now : you now add cross-process synchronization (via the kernel => syscall) as well as insert the scheduler in the mix

pjmlp 3y ago

Because hardware resources, 20 years ago doing something like VSCode with tons of external processes per plugin, would drag your computer to a crawl. Emacs wasn't Eight Megabytes and Constantly Swapping only due to Elisp.

marcosdumay 3y ago

That's because IPC is not low-latency. No modern processor architecture has a proper message passing mechanism. All of them expect you to use interruptions; with it's inherent problems of losing cache, disrupting pipelines, and well, interrupting your process flow. All the modern architectures are also so close to have a proper message passing mechanism that it's unsettling. You actually need this to have uniform memory in a multi-core CPU. They have all the mechanisms for zero copy sharing of memory, enforcing coherence, atomicity, etc. AFAIK, they just lack a userspace mechanism to signal other processes.

oasisaimlessly 3y ago
Futexes allow low-latency signalling to other processes, and FUTEX_SWAP [1, 2] promises to decrease the cost even further. [1]: https://lore.kernel.org/lkml/20200722234538.166697-1-posk@posk.io/ https://lore.kernel.org/lkml/20200722234538.166697-1-posk@po... [2]: https://www.phoronix.com/news/Google-User-Thread-Futex-Swap https://www.phoronix.com/news/Google-User-Thread-Futex-Swap
- marcosdumay 3y ago
  This doesn't seem to touch on any of the points on my comment. It does take a lot of unnecessary stuff out of the way. But isn't enough to change the picture at all.
  - quotemstr 3y ago
    > This doesn't seem to touch on any of the points on my comment. What points did your comment make? You didn't define "signalling" specifically enough to discuss. Can you elaborate on precisely what kind of "signaling" primitive processors or operating systems should provide?

zbentley 3y ago

This was roughly the dream of DBus. However, outside of desktop-shaped niches it proved to be extremely difficult to secure, standardize, and debug. Process-level/address-space-level dependency sharing remains both easier to think about and simpler to implement (and capabilities are taking bites out of the security risks entailed by this model as time goes on).

josephg 3y ago

If the goal is to put a security boundary within the process between libraries, there might be better ways to do it than process boundaries. One approach is to wasm sandbox library code. Firefox apparently does this - compiling some libraries to wasm, then compiling the wasm back to C and linking it. They get all the benefits of wasm but without any need to JIT compile the code. Another approach would be to leverage a language like rust. I’d love it if rust provided a way to deny any sensitive access to part of my dependency tree. I want to pull a library but deny it the ability to run unsafe code or make any syscall (or maybe, make syscalls but I’ll whitelist what it’s allowed to call). Restrictions should be transitive to all of that library’s dependencies (optionally with further restrictions). Both of these approaches would stop the library from doing untoward things. Way more so than you’d get running the library in a separate process.

magicalhippo 3y ago

Think you would have loved Signularity OS[1]. It featured software-isolated processes that communicated via contract-based message passing, which allowed for zero-copy exchange of data. As a research OS it never became a fully-fledged OS[2], but an interesting attempt IMHO. [1]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/osr2007_rethinkingsoftwarestack.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... [2]: https://en.wikipedia.org/wiki/Singularity_%28operating_system%29 https://en.wikipedia.org/wiki/Singularity_%28operating_syste...

zokier 3y ago

Dbus is a thing. Bus1 was its still-born spiritual kernel-space successor.