7 ms·

io_uring_spawn: Launching New Processes with io_uring

109 points by goranmoomin 4y ago

staticassertion 4y ago
So, to my knowledge io_uring is a complete seccomp bypass (for all functionality it implements). Docker allows io_uring because "so far it isn't that big of a deal" (paraphrasing from Github issue 2 years ago). Given that io_uring now provides file IO and process execution... isn't it about time that someone starts thinking about the security implications? Are we going to just have a universal seccomp bypass for anyone who uses io_uring? How're the audit hooks coming along? Or LSM hooks? edit: K I've reached my HN rate limit again, thanks Dang. Anyway, here's a source for the Docker seccomp filter since someone replied inquiring. https://github.com/thaJeztah/docker/blob/master/profiles/seccomp/default.json https://github.com/thaJeztah/docker/blob/master/profiles/sec...
- rotifer 4y ago
  I don't follow it particularly closely, but I recall that there was a recent LWN article, "Security requirements for new kernel features" [1], which discussed the issue. [1] https://lwn.net/Articles/902466/ https://lwn.net/Articles/902466/
  - staticassertion 4y ago
    Yeah, but the discussion hasn't gone anywhere lol the conclusion of the article is basically "yeah I guess they don't care"
- the8472 4y ago
  seccomp filters generally are whitelists. If they haven't whitelisted io_uring_setup it's a non-issue. They only suffer the performance consequences of userspace having to use the synchronous syscalls. also, iouring doesn't allow arbitrary syscalls. so it's not really a complete seccomp bypass even when allowed
  - ori_b 4y ago
    > They only suffer the performance consequences of userspace having to use the synchronous syscalls. And the need to implement all the functionality twice. Who is going to do that?
    - the8472 4y ago
      Everyone who doesn't tightly control the systems they deploy to. io_uring is a new and developing interface so to support extant systems running libraries need to have fallback codepaths. Or more likely: they already have existing codepaths for the previous interfaces and add io_uring as a faster alternative.
ori_b 4y ago
Does this matter when you have shared libraries? ld.so will be doing hundreds of syscalls at start regardless.
- plorkyeran 4y ago
  If you're doing something where process launch speed is extremely important you're hopefully already not using shared libraries.
- JoshTriplett 4y ago
  Working on that too. One of the wonderful things about Plumbers is that so many different people working on infrastructure are there. I had a great conversation with Carlos O'Donell (glibc maintainer) about using io_uring in glibc, including in ld.so for that exact reason...
wmf 4y ago
If I'm reading this right it should also allow memory overcommit to be disabled.
- scottlamb 4y ago
  Every large process using vfork/posix_spawn/io_uring_spawn instead of fork would eliminate one reason to keep overcommit enabled. I think there are at least some niche cases in which folks would still want it on. For example, some processes may rely on anonymous mappings being cheap until first write via Linux's "empty_zero_page". IIRC, they once tried to remove this concept but put it back in. Some context here: https://lwn.net/Articles/340370/ https://lwn.net/Articles/340370/
deleted 4y ago
[deleted]
boffinAudio 4y ago
This is really exciting work - it has the potential to make Linux software services way, way more efficient. I look forward to seeing the impact that this change has, for example, on future builds. A significant amount of time is spent at the beginning of a process spawn, setting things up - being able to batch things up without the process startup load, is going to make things much, much faster. Process pools? Yes please! :) I will definitely be a customer of posix_spawn() when its ready .. One has to wonder, though, given Josh' work with Rust, if any of this work is going to be done in Rust? One can dream ..
- JoshTriplett 4y ago
  > One has to wonder, though, given Josh' work with Rust, if any of this work is going to be done in Rust? One can dream .. Not yet, since the Rust-in-Linux work hasn't been merged yet. And even once it is, it won't be usable in the core kernel yet. But I'm hopeful that io_uring will be able to make use of Rust in the future. And the userspace bits will definitely be possible to do in Rust. As soon as this ships in a stable kernel, I plan to add a code path in the Rust standard library to launch processes this way. :)
egberts1 4y ago
Reminds me of a single pit stop where a queue of race cars await the “go” words. Still need resources behind the pit stop, lots and lots of resources.
the_duke 4y ago
What kind of workloads spawn so many processes that saving microseconds becomes relevant? I guess shells (eg with xargs) would be one example. Edit: the talk mentions build systems as a motivating use case.
- turminal 4y ago
  If you're writing anything with performance considerations using shell scripts, you're almost certainly doing something wrong. If it includes spawning thousands of processes, you're definitely doing something wrong.
- staticassertion 4y ago
  IDEs can go insane with process executions. VisualStudio shells out like crazy when doing builds. Also, processes are an awesome abstraction - they give security and reliability. Reducing their cost is nice.
- xroche 4y ago
  > What kind of workloads spawn so many processes that saving microseconds becomes relevant? With a huge process, you have a timeframe between the child is spawned and it executes exec*() where you typically "do stuff" (such as closing a lot of fd) During this timeframe the parent process has its universe COW'ed, and each write will trigger a page fault. The performance impact can be concerning in the _parent_ process.
- the8472 4y ago
  For example rust doctests consist of compiling each little doc snippet and executing it ensure that the asserts are true. That's two processes (rustc + the compiled piece of code) per test. This is an order of magnitude slower than multi-threaded in-process tests. Process startup and teardown is a significant fraction of that overhead. Now fork+exec isn't the biggest chunk of that. But it fingers the memory subsystem. On many-core systems this leads to lock contention. So the cheaper starting a new process gets in terms of virtual memory operations the less lock-contention you get overall which also speeds up other parts.
- sc68cal 4y ago
nos4A2 4y ago
The comparison does not seem to be apples to apples. The perf gain seems to come from bundling the fork and exec call allowing the kernel to do vfork like optimizations. I think a fair comparison would be against a new sys call that does fork_exec vs io_uring_spawn, and not fork + exec vs io_uring_spawn.
- rst 4y ago
  It looked to me like providing a cleaner interface which still effectively supports everything vfork does, with all the performance gains (and fewer context switches) was in fact the point of the exercise.
- JoshTriplett 4y ago
  That's absolutely where much of the performance currently comes from, and I did in fact benchmark exactly that and it provides comparable performance (for now). I also noted in my talk that you can get much of the way there by doing clone with CLONE_VM and doing exec from there. But also, the tricks to make it faster without using io_uring are much less safe than using io_uring. vfork is dangerous. io_uring is the standard kernel mechanism for doing multiple operations in one syscall, so it seems like the most logical fit for this. Jens Axboe (io_uring maintainer) and I have many more plans to make this faster. Leaving that aside, there are many additional capabilities this unlocks. For instance, we can maintain a pool of processes set up and ready to exec.
- scottlamb 4y ago
  > I think a fair comparison would be against a new sys call that does fork_exec vs io_uring_spawn, and not fork + exec vs io_uring_spawn. I agree that's a useful comparison, although the status quo is also worth benchmarking. A fork_exec syscall sounds like a posix_spawn syscall. IIRC, glibc's current posix_spawn is a library function that does a vfork followed by setup operations that portable code isn't allowed to call after vfork. I think there are two dimensions in which io_uring can be used to amortize/eliminate syscalls: * sequence of operations: in this case, (v)fork, then setup calls, then exec. This sequence is unusually expensive because doing them separately requires extra page table work (particularly for the fork case, but even for vfork vs having a blank process). A posix_spawn syscall could achieve similar efficiency gains as this io_uring_spawn. * batches of operations: in this case, lots of processes to spawn. It sounds like he hasn't posted numbers on that and intends to: "He's looking forward to ... supporting a pre-spawned process pool". If that turns out well, then the io_uring approach seems worth doing, and also can be used to implement a new in-user-space posix_spawn, for less total syscall surface area than doing both io_uring and a synchronous posix_spawn syscall. Maybe the supported setup operations also can be in common with io_uring operations for other purposes. I'm curious about that but don't see a link to benchmark code / implementation code / manpage.
kyaghmour 4y ago
I attended Josh's talk earlier this week at LPC. As I told Josh yesterday in the hallway track, I've been attending Linux conferences for 20+ years and often times some topics keep coming back and you can see the specialization. But in this case, I was completely blown away. This is a totally new approach to an old problem. I mean, fork() was invented in the '70's and aside from threads and clone(), in Linux, there really hasn't been any innovation in this space over the years. I look forward to seeing what else gets io_uring'ed following this model.
- vlovich123 4y ago
  I hope I/O write queues at some point. Being able to inject I/O barriers relevant to the I/O stream instead of a system-wide barrier can let you write some higher efficiency I/O.
joeyh 4y ago
The performance of this might be useful for a few things. The correctness seems more generally useful. The first two things I looked at that currently use vfork both go on to do things that are unsafe (according to the information in this talk anyway). I'll bet a substantial number of programs that use fork don't limit themselves to signal-safety(7) functions either.
- JoshTriplett 4y ago
  Exactly. I started out aiming for performance, and quickly realized that the benefits to correctness were even higher.

109 points by goranmoomin 4y ago

So, to my knowledge io_uring is a complete seccomp bypass (for all functionality it implements). Docker allows io_uring because "so far it isn't that big of a deal" (paraphrasing from Github issue 2 years ago). Given that io_uring now provides file IO and process execution... isn't it about time that someone starts thinking about the security implications? Are we going to just have a universal seccomp bypass for anyone who uses io_uring? How're the audit hooks coming along? Or LSM hooks? edit: K I've reached my HN rate limit again, thanks Dang. Anyway, here's a source for the Docker seccomp filter since someone replied inquiring. https://github.com/thaJeztah/docker/blob/master/profiles/seccomp/default.json https://github.com/thaJeztah/docker/blob/master/profiles/sec...

rotifer 4y ago
I don't follow it particularly closely, but I recall that there was a recent LWN article, "Security requirements for new kernel features" [1], which discussed the issue. [1] https://lwn.net/Articles/902466/ https://lwn.net/Articles/902466/
- staticassertion 4y ago
  Yeah, but the discussion hasn't gone anywhere lol the conclusion of the article is basically "yeah I guess they don't care"
the8472 4y ago
seccomp filters generally are whitelists. If they haven't whitelisted io_uring_setup it's a non-issue. They only suffer the performance consequences of userspace having to use the synchronous syscalls. also, iouring doesn't allow arbitrary syscalls. so it's not really a complete seccomp bypass even when allowed
- ori_b 4y ago
  > They only suffer the performance consequences of userspace having to use the synchronous syscalls. And the need to implement all the functionality twice. Who is going to do that?
  - the8472 4y ago
    Everyone who doesn't tightly control the systems they deploy to. io_uring is a new and developing interface so to support extant systems running libraries need to have fallback codepaths. Or more likely: they already have existing codepaths for the previous interfaces and add io_uring as a faster alternative.

ori_b 4y ago

Does this matter when you have shared libraries? ld.so will be doing hundreds of syscalls at start regardless.

plorkyeran 4y ago
If you're doing something where process launch speed is extremely important you're hopefully already not using shared libraries.
JoshTriplett 4y ago
Working on that too. One of the wonderful things about Plumbers is that so many different people working on infrastructure are there. I had a great conversation with Carlos O'Donell (glibc maintainer) about using io_uring in glibc, including in ld.so for that exact reason...

wmf 4y ago

If I'm reading this right it should also allow memory overcommit to be disabled.

scottlamb 4y ago
Every large process using vfork/posix_spawn/io_uring_spawn instead of fork would eliminate one reason to keep overcommit enabled. I think there are at least some niche cases in which folks would still want it on. For example, some processes may rely on anonymous mappings being cheap until first write via Linux's "empty_zero_page". IIRC, they once tried to remove this concept but put it back in. Some context here: https://lwn.net/Articles/340370/ https://lwn.net/Articles/340370/

deleted 4y ago

[deleted]

boffinAudio 4y ago

This is really exciting work - it has the potential to make Linux software services way, way more efficient. I look forward to seeing the impact that this change has, for example, on future builds. A significant amount of time is spent at the beginning of a process spawn, setting things up - being able to batch things up without the process startup load, is going to make things much, much faster. Process pools? Yes please! :) I will definitely be a customer of posix_spawn() when its ready .. One has to wonder, though, given Josh' work with Rust, if any of this work is going to be done in Rust? One can dream ..

JoshTriplett 4y ago
> One has to wonder, though, given Josh' work with Rust, if any of this work is going to be done in Rust? One can dream .. Not yet, since the Rust-in-Linux work hasn't been merged yet. And even once it is, it won't be usable in the core kernel yet. But I'm hopeful that io_uring will be able to make use of Rust in the future. And the userspace bits will definitely be possible to do in Rust. As soon as this ships in a stable kernel, I plan to add a code path in the Rust standard library to launch processes this way. :)

egberts1 4y ago

Reminds me of a single pit stop where a queue of race cars await the “go” words. Still need resources behind the pit stop, lots and lots of resources.

the_duke 4y ago

What kind of workloads spawn so many processes that saving microseconds becomes relevant? I guess shells (eg with xargs) would be one example. Edit: the talk mentions build systems as a motivating use case.

turminal 4y ago
If you're writing anything with performance considerations using shell scripts, you're almost certainly doing something wrong. If it includes spawning thousands of processes, you're definitely doing something wrong.
staticassertion 4y ago
IDEs can go insane with process executions. VisualStudio shells out like crazy when doing builds. Also, processes are an awesome abstraction - they give security and reliability. Reducing their cost is nice.
xroche 4y ago
> What kind of workloads spawn so many processes that saving microseconds becomes relevant? With a huge process, you have a timeframe between the child is spawned and it executes exec*() where you typically "do stuff" (such as closing a lot of fd) During this timeframe the parent process has its universe COW'ed, and each write will trigger a page fault. The performance impact can be concerning in the _parent_ process.
the8472 4y ago
For example rust doctests consist of compiling each little doc snippet and executing it ensure that the asserts are true. That's two processes (rustc + the compiled piece of code) per test. This is an order of magnitude slower than multi-threaded in-process tests. Process startup and teardown is a significant fraction of that overhead. Now fork+exec isn't the biggest chunk of that. But it fingers the memory subsystem. On many-core systems this leads to lock contention. So the cheaper starting a new process gets in terms of virtual memory operations the less lock-contention you get overall which also speeds up other parts.
sc68cal 4y ago

nos4A2 4y ago

The comparison does not seem to be apples to apples. The perf gain seems to come from bundling the fork and exec call allowing the kernel to do vfork like optimizations. I think a fair comparison would be against a new sys call that does fork_exec vs io_uring_spawn, and not fork + exec vs io_uring_spawn.

rst 4y ago
It looked to me like providing a cleaner interface which still effectively supports everything vfork does, with all the performance gains (and fewer context switches) was in fact the point of the exercise.
JoshTriplett 4y ago
That's absolutely where much of the performance currently comes from, and I did in fact benchmark exactly that and it provides comparable performance (for now). I also noted in my talk that you can get much of the way there by doing clone with CLONE_VM and doing exec from there. But also, the tricks to make it faster without using io_uring are much less safe than using io_uring. vfork is dangerous. io_uring is the standard kernel mechanism for doing multiple operations in one syscall, so it seems like the most logical fit for this. Jens Axboe (io_uring maintainer) and I have many more plans to make this faster. Leaving that aside, there are many additional capabilities this unlocks. For instance, we can maintain a pool of processes set up and ready to exec.
scottlamb 4y ago
> I think a fair comparison would be against a new sys call that does fork_exec vs io_uring_spawn, and not fork + exec vs io_uring_spawn. I agree that's a useful comparison, although the status quo is also worth benchmarking. A fork_exec syscall sounds like a posix_spawn syscall. IIRC, glibc's current posix_spawn is a library function that does a vfork followed by setup operations that portable code isn't allowed to call after vfork. I think there are two dimensions in which io_uring can be used to amortize/eliminate syscalls: * sequence of operations: in this case, (v)fork, then setup calls, then exec. This sequence is unusually expensive because doing them separately requires extra page table work (particularly for the fork case, but even for vfork vs having a blank process). A posix_spawn syscall could achieve similar efficiency gains as this io_uring_spawn. * batches of operations: in this case, lots of processes to spawn. It sounds like he hasn't posted numbers on that and intends to: "He's looking forward to ... supporting a pre-spawned process pool". If that turns out well, then the io_uring approach seems worth doing, and also can be used to implement a new in-user-space posix_spawn, for less total syscall surface area than doing both io_uring and a synchronous posix_spawn syscall. Maybe the supported setup operations also can be in common with io_uring operations for other purposes. I'm curious about that but don't see a link to benchmark code / implementation code / manpage.

kyaghmour 4y ago

I attended Josh's talk earlier this week at LPC. As I told Josh yesterday in the hallway track, I've been attending Linux conferences for 20+ years and often times some topics keep coming back and you can see the specialization. But in this case, I was completely blown away. This is a totally new approach to an old problem. I mean, fork() was invented in the '70's and aside from threads and clone(), in Linux, there really hasn't been any innovation in this space over the years. I look forward to seeing what else gets io_uring'ed following this model.

vlovich123 4y ago
I hope I/O write queues at some point. Being able to inject I/O barriers relevant to the I/O stream instead of a system-wide barrier can let you write some higher efficiency I/O.

joeyh 4y ago

The performance of this might be useful for a few things. The correctness seems more generally useful. The first two things I looked at that currently use vfork both go on to do things that are unsafe (according to the information in this talk anyway). I'll bet a substantial number of programs that use fork don't limit themselves to signal-safety(7) functions either.

JoshTriplett 4y ago
Exactly. I started out aiming for performance, and quickly realized that the benefits to correctness were even higher.