10 ms·

On the Costs of Syscalls (2021)

105 points by K0nserv 3y ago

Dwedit 3y ago
I tested the performance of system calls on Windows 10 by repeatedly calling NtClose on an invalid handle. A system call took around 1200 CPU cycles.
- ComputerGuru 3y ago
  Which timing mechanism did you use?
  - Dwedit 3y ago
    This was a while ago and I don't remember the exact details... I had two loops with a large number of operations. One called a trivial Win32 function (GetACP), the other made a System Call (NtClose) as well as the same trivial Win32 function. I used QueryPerformanceCounter to time the loops. From there, I estimated the number of machine cycles per iteration (using my clock speed of the time), and subtracted the "trivial" version which made no system calls.
    - ComputerGuru 3y ago
      Clever approach!
      - Dwedit 3y ago
        Peeking at "ntoskrnl.exe" in Ghidra, it appears that "NtVdmControl" is a true "do-nothing" system call in Windows 10 x64, and Windows 11 removed the function entirely.
vvern 3y ago
Any good resources/deeper dives into the details of what actually happens on modern computers when making these syscalls on, say, linux? I might have a reasonable idea about what goes on when performing a mode switch or context switch, but I’d love to have a reference or a nice walkthrough.
- deleted 3y ago
  [deleted]
- mindwok 3y ago
  My introduction to this was the 'Direct Execution' chapter of the book 'Operating Systems: Three Easy Pieces'. It's a fantastic write up on not just how system calls work, but the motivation behind why operating systems even implement them in the first place. It's not specifically about Linux, but the book is clearly heavily inspired by early *nix designs and it's still applicable. You can read it for free here: https://pages.cs.wisc.edu/~remzi/OSTEP/cpu-mechanisms.pdf https://pages.cs.wisc.edu/~remzi/OSTEP/cpu-mechanisms.pdf
- cyberax 3y ago
  This one is pretty good: https://blog.packagecloud.io/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/the-definitive-guide-to-linux-s...
  - matheusmoreira 3y ago
    > It’s not a great idea to call system calls by writing your own assembly code. > One big reason for this is that some system calls have additional code that runs in glibc before or after the system call runs. That's not really the fault of the system calls. The problem is the C library itself. It's too stateful, it's full of global data and action at a distance. If you use certain system calls, you invalidate its countless assumptions and invariants, essentially pulling the rug from under it. I've found programming in freestanding C with Linux system calls and no libc to be a very rewarding experience. Getting rid of libc and its legacy vastly improves C as a language. The Linux system call interface is quite clean. Don't even have to deal with the nonsense that is the errno variable.
    - PaulDavisThe1st 3y ago
      Sounds roughly equivalent to discussing how much more rewarding it is to walk 1000km than to drive it. Yes, everybody knows you can get from A to B on foot. But for multiple reasons, and over and extended period of time, we developed other methods of doing so. That doesn't invalidate the experience of doing it on foot, but it does make the walk into a very conscious choice that probably isn't the one most people are going to make most of the time.
snvzz 3y ago
There are also RTOS-capable microkernels such as seL4[0], with few but extremely fast syscalls[1]. Note times are in cycles, not ns. 0. https://sel4.systems/ https://sel4.systems/ 1. https://sel4.systems/About/Performance/ https://sel4.systems/About/Performance/
- monocasa 3y ago
  AFAIK sel4 is similar now as most of the weight comes from spectre/meltdown mitigations from flushing the TLB on privilege changes.
billpg 3y ago
A website from French Antarctica? Cool.
vlovich123 3y ago
Anyone know why getpid/getuid still aren’t implemented as vdso?
- tptacek 3y ago
  When are you ever calling getuid in a tight loop?
  - bitcharmer 3y ago
    GP's question is perfectly valid. It doesn't have to be called within a tight loop at all. It could be a latency sensitive routine for example. I can see at least a few real life applications where it would make sense to have the syscall vDSO'd.
    - tptacek 3y ago
      What's one of them?
      - vlovich123 3y ago
        https://bugzilla.redhat.com/show_bug.cgi?id=1469670 https://bugzilla.redhat.com/show_bug.cgi?id=1469670 I would appreciate if someone explained why it’s not a vdso rather than just repeating it’s not necessary like that’s a sufficient explanation.
        masklinn 3y ago
        It’s not a VDSO because it was never added as one. “Not a VDSO” is the default, a VDSO is additional complexity and maintenance so you need to justify its addition / existence, not its non-existence. Also given https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47ir1bZGAK/comp/linux/getpid_caching.html https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i... while Linus may have changed tack in the decades since I would not expect much support from stating getpid is performance critical to you.
        3y ago
flakes 3y ago
The best part of this thread was the link to the discussion on the removal of pid caching in `getpid` causing regressions in systemd https://bugzilla.redhat.com/show_bug.cgi?id=1469670 https://bugzilla.redhat.com/show_bug.cgi?id=1469670 I've been on a code-deleting rampage lately, killing off many minor features that add a lot of complexity for seemingly little gain. Glad I don't have anyone interrogating me like this!
- ape4 3y ago
  That was interesting - "Umm, this change causes major slowdowns in various bits of systemd code, which assumes that getpid() is fast. libsystemd uses getpid() to detect when processes fork". A regular userspace program doesn't need to detect when a fork() has occurred since its whats doing the fork().
  - deleted 3y ago
    [deleted]
- senkora 3y ago
  The snark here is impressive: > in order to quickly fix this pressing matter, I've attached code that I believe will fix your issue. It uses a custom caching mechanism and fits well into the systemd ecosystem. pid_t systemd_getpid(void) { return 0; }
  - fullstop 3y ago
    It's kind of a dick answer, imo.
    - marcosdumay 3y ago
      I'm really not sure if it's perfectly context sensitive or too much even for systemd. But dick answers is the specialty of the systemd development team. I wouldn't want to get into a contest against them.
  - flakes 3y ago
pjmlp 3y ago
While interesting to read, the title should be "On the Costs of Linux Syscalls"
- alkonaut 3y ago
  Yeah this paragraph stands out as a bit strange "I ran the benchmark on a heterogeneous set of hosts, i.e. different kernels, operating systems and configurations" In the fine print: the "different OS:es" were two Linux Distributions which are (if I understand correctly) not even that far apart tech wise? (RHEL and Fedora).
- drpixie 3y ago
  And even more specifically "On the Costs of Linux Syscalls on Intel processors". I would like to see this extended to other processor families.
exDM69 3y ago
This micro benchmark measures the direct cost of system calls. But flushing the TLB and trashing the caches carries an indirect cost too. According to some benchmarks I've seen (sorry no link), a system call in the middle of a memory heavy inner loop can impact performance for 100 us before performance is back to steady state. This is 100x longer than the system call alone. Of course there is work being done, but at a reduced throughput. This is in line with my practical experience from working with performance sensitive code. These things are difficult to benchmark reliably and draw actionable conclusions from the results. This is not a criticism to the author of this article, the article clearly describes the methodology of the benchmarks and does not suggest any wrong conclusions from the data.
- sapiogram 3y ago
  > According to some benchmarks I've seen (sorry no link), a system call in the middle of a memory heavy inner loop can impact performance for 100 us before performance is back to steady state. This is 100x longer than the system call alone. Do you some intuition of what causes this? I don't have how much experience with this kind of work, but 100µs is enough time to do hundreds of random memory accesses. How can a single syscall do so much damage to the cache?
  - Karellen 3y ago
    As the GP mentioned, it's TLB cache invalidation that can be the problem. Reloading the mappings from virtual (per-process) addresses to physical addresses after returning from the kernel can cause delays (on some architectures), even if most of the physical memory is still in the cache and valid. https://en.wikipedia.org/wiki/Translation_lookaside_buffer https://en.wikipedia.org/wiki/Translation_lookaside_buffer (Also, worth pointing out that they didn't claim a delay of 100µs, just that some (presumably, much smaller, on the order of ns?) delays can show up up to 100µs later before "steady state" is fully restored.)
    - gpderetta 3y ago
      As syscalls run more code and access more data they can increase TLB pressure in the same way the increase general cache pressure, but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right (there were exceptions like the brief period when people where running with full 4GB user address space on 32bits cpu).
secondcoming 3y ago
Now that's interesting. I remember years ago profiling the gettimeofday() function on various platforms. It was fine on all except our AWS VMs. Turns out it wasn't implemented as a vDSO and the performance was all over the place. That's been fixed since. We use getpid() in logging and maybe it's unnecessary.
cb321 3y ago
Using https://github.com/c-blake/batch/blob/master/examples/closes.c https://github.com/c-blake/batch/blob/master/examples/closes... from the examples/ directory of https://github.com/c-blake/batch https://github.com/c-blake/batch, I get: L2:bat/examples$ BATCH_EMUL=1 ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 3.815 *$2 + 695.6826 *$3 bootstrap-stderr-corr matrix 0.8872 -0.7856 0.04174 L2:bat/examples$ BATCH_EMUL=1 ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 4.528 *$2 + 695.3265 *$3 bootstrap-stderr-corr matrix 1.174 -0.7640 0.05302 L2:bat/examples$ a (695.3265 +- 0.05302)-(695.6826 +- 0.04174) -0.356 +- 0.067 # i.e. run-to-run consistent L2:bat/examples$ ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 779.67 *$2 + 48.861 *$3 bootstrap-stderr-corr matrix 1.550 -0.7495 0.2023 L2:bat/examples$ ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 787.77 *$2 + 48.745 *$3 bootstrap-stderr-corr matrix 1.350 -0.6207 0.1635 L2:bat/examples$ a (48.861 +- 0.2023)-(48.745 +- 0.1635) 0.12 +- 0.26 # i.e. run-to-run consistent L2:bat/examples$ a (695.3265 +- 0.05302)/(48.745 +- 0.1635) 14.265 +- 0.048 # i.e. batching makes it 14X faster! One can go further to get a mean +- std.err of ~695.52 +- ~0.04 nanosec hot-cache time per syscall overhead or even do point-weighting there, but it's probably more important to check fit residuals for serial autocorrelation (very do-able with https://github.com/c-blake/fitl https://github.com/c-blake/fitl, EDIT: oh, and if you care `a` is basically shown in https://github.com/SciNim/Measuremancer/pull/12 https://github.com/SciNim/Measuremancer/pull/12 which uses standard error propagation). The bigger point would not be ever more careful measurement of the cost if you do not, but rather the ease of "batchifying" if you do of any highly regular call interface similar to syscalls with high latency and also the "follow on" API design, granularity-wise. The inner code of a mini-assembly language letting you implement, say, open/fstat/mmap/close all in one syscall crossing is only about 30 lines of C. { Yes, yes..It could use multi-CPU-arch and syscall auditing integration.. and sure, ebpf & io_uring alter the Linux landscape these days }. Also, @exDM69 is very correct that pure hot cache-hot loop numbers are only one part of the cost story (https://news.ycombinator.com/item?id=39188551 https://news.ycombinator.com/item?id=39188551).
gpderetta 3y ago
The measured mitigation cost overhead is very interesting. Those XG machines have very fast syscalls (<100ns), but are more than twice as slow with mitigation on, even though they are fairly recent and most meltdown/spectre bugs should have been patched out.
alberth 3y ago
Is the cost of syscalls expensive for all OS's (e.g. BSD, Redox, etc)? Does OpenBSD limiting how you can call a syscall help at on performance (or hurt)?

105 points by K0nserv 3y ago

I tested the performance of system calls on Windows 10 by repeatedly calling NtClose on an invalid handle. A system call took around 1200 CPU cycles.

ComputerGuru 3y ago
Which timing mechanism did you use?
- Dwedit 3y ago
  This was a while ago and I don't remember the exact details... I had two loops with a large number of operations. One called a trivial Win32 function (GetACP), the other made a System Call (NtClose) as well as the same trivial Win32 function. I used QueryPerformanceCounter to time the loops. From there, I estimated the number of machine cycles per iteration (using my clock speed of the time), and subtracted the "trivial" version which made no system calls.
  - ComputerGuru 3y ago
    Clever approach!
    - Dwedit 3y ago
      Peeking at "ntoskrnl.exe" in Ghidra, it appears that "NtVdmControl" is a true "do-nothing" system call in Windows 10 x64, and Windows 11 removed the function entirely.

vvern 3y ago

Any good resources/deeper dives into the details of what actually happens on modern computers when making these syscalls on, say, linux? I might have a reasonable idea about what goes on when performing a mode switch or context switch, but I’d love to have a reference or a nice walkthrough.

deleted 3y ago
[deleted]
mindwok 3y ago
My introduction to this was the 'Direct Execution' chapter of the book 'Operating Systems: Three Easy Pieces'. It's a fantastic write up on not just how system calls work, but the motivation behind why operating systems even implement them in the first place. It's not specifically about Linux, but the book is clearly heavily inspired by early *nix designs and it's still applicable. You can read it for free here: https://pages.cs.wisc.edu/~remzi/OSTEP/cpu-mechanisms.pdf https://pages.cs.wisc.edu/~remzi/OSTEP/cpu-mechanisms.pdf
cyberax 3y ago
This one is pretty good: https://blog.packagecloud.io/the-definitive-guide-to-linux-system-calls/ https://blog.packagecloud.io/the-definitive-guide-to-linux-s...
- matheusmoreira 3y ago
  > It’s not a great idea to call system calls by writing your own assembly code. > One big reason for this is that some system calls have additional code that runs in glibc before or after the system call runs. That's not really the fault of the system calls. The problem is the C library itself. It's too stateful, it's full of global data and action at a distance. If you use certain system calls, you invalidate its countless assumptions and invariants, essentially pulling the rug from under it. I've found programming in freestanding C with Linux system calls and no libc to be a very rewarding experience. Getting rid of libc and its legacy vastly improves C as a language. The Linux system call interface is quite clean. Don't even have to deal with the nonsense that is the errno variable.
  - PaulDavisThe1st 3y ago
    Sounds roughly equivalent to discussing how much more rewarding it is to walk 1000km than to drive it. Yes, everybody knows you can get from A to B on foot. But for multiple reasons, and over and extended period of time, we developed other methods of doing so. That doesn't invalidate the experience of doing it on foot, but it does make the walk into a very conscious choice that probably isn't the one most people are going to make most of the time.

snvzz 3y ago

There are also RTOS-capable microkernels such as seL4[0], with few but extremely fast syscalls[1]. Note times are in cycles, not ns. 0. https://sel4.systems/ https://sel4.systems/ 1. https://sel4.systems/About/Performance/ https://sel4.systems/About/Performance/

monocasa 3y ago
AFAIK sel4 is similar now as most of the weight comes from spectre/meltdown mitigations from flushing the TLB on privilege changes.

billpg 3y ago

A website from French Antarctica? Cool.

vlovich123 3y ago

Anyone know why getpid/getuid still aren’t implemented as vdso?

tptacek 3y ago
When are you ever calling getuid in a tight loop?
- bitcharmer 3y ago
  GP's question is perfectly valid. It doesn't have to be called within a tight loop at all. It could be a latency sensitive routine for example. I can see at least a few real life applications where it would make sense to have the syscall vDSO'd.
  - tptacek 3y ago
    What's one of them?
    - vlovich123 3y ago
      https://bugzilla.redhat.com/show_bug.cgi?id=1469670 https://bugzilla.redhat.com/show_bug.cgi?id=1469670 I would appreciate if someone explained why it’s not a vdso rather than just repeating it’s not necessary like that’s a sufficient explanation.
      - masklinn 3y ago
        It’s not a VDSO because it was never added as one. “Not a VDSO” is the default, a VDSO is additional complexity and maintenance so you need to justify its addition / existence, not its non-existence. Also given https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47ir1bZGAK/comp/linux/getpid_caching.html https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i... while Linus may have changed tack in the decades since I would not expect much support from stating getpid is performance critical to you.
      - 3y ago

flakes 3y ago

The best part of this thread was the link to the discussion on the removal of pid caching in `getpid` causing regressions in systemd https://bugzilla.redhat.com/show_bug.cgi?id=1469670 https://bugzilla.redhat.com/show_bug.cgi?id=1469670 I've been on a code-deleting rampage lately, killing off many minor features that add a lot of complexity for seemingly little gain. Glad I don't have anyone interrogating me like this!

ape4 3y ago
That was interesting - "Umm, this change causes major slowdowns in various bits of systemd code, which assumes that getpid() is fast. libsystemd uses getpid() to detect when processes fork". A regular userspace program doesn't need to detect when a fork() has occurred since its whats doing the fork().
- deleted 3y ago
  [deleted]
senkora 3y ago
The snark here is impressive: > in order to quickly fix this pressing matter, I've attached code that I believe will fix your issue. It uses a custom caching mechanism and fits well into the systemd ecosystem. pid_t systemd_getpid(void) { return 0; }
- fullstop 3y ago
  It's kind of a dick answer, imo.
  - marcosdumay 3y ago
    I'm really not sure if it's perfectly context sensitive or too much even for systemd. But dick answers is the specialty of the systemd development team. I wouldn't want to get into a contest against them.
- flakes 3y ago

pjmlp 3y ago

While interesting to read, the title should be "On the Costs of Linux Syscalls"

alkonaut 3y ago
Yeah this paragraph stands out as a bit strange "I ran the benchmark on a heterogeneous set of hosts, i.e. different kernels, operating systems and configurations" In the fine print: the "different OS:es" were two Linux Distributions which are (if I understand correctly) not even that far apart tech wise? (RHEL and Fedora).
drpixie 3y ago
And even more specifically "On the Costs of Linux Syscalls on Intel processors". I would like to see this extended to other processor families.

exDM69 3y ago

This micro benchmark measures the direct cost of system calls. But flushing the TLB and trashing the caches carries an indirect cost too. According to some benchmarks I've seen (sorry no link), a system call in the middle of a memory heavy inner loop can impact performance for 100 us before performance is back to steady state. This is 100x longer than the system call alone. Of course there is work being done, but at a reduced throughput. This is in line with my practical experience from working with performance sensitive code. These things are difficult to benchmark reliably and draw actionable conclusions from the results. This is not a criticism to the author of this article, the article clearly describes the methodology of the benchmarks and does not suggest any wrong conclusions from the data.

sapiogram 3y ago
> According to some benchmarks I've seen (sorry no link), a system call in the middle of a memory heavy inner loop can impact performance for 100 us before performance is back to steady state. This is 100x longer than the system call alone. Do you some intuition of what causes this? I don't have how much experience with this kind of work, but 100µs is enough time to do hundreds of random memory accesses. How can a single syscall do so much damage to the cache?
- Karellen 3y ago
  As the GP mentioned, it's TLB cache invalidation that can be the problem. Reloading the mappings from virtual (per-process) addresses to physical addresses after returning from the kernel can cause delays (on some architectures), even if most of the physical memory is still in the cache and valid. https://en.wikipedia.org/wiki/Translation_lookaside_buffer https://en.wikipedia.org/wiki/Translation_lookaside_buffer (Also, worth pointing out that they didn't claim a delay of 100µs, just that some (presumably, much smaller, on the order of ns?) delays can show up up to 100µs later before "steady state" is fully restored.)
  - gpderetta 3y ago
    As syscalls run more code and access more data they can increase TLB pressure in the same way the increase general cache pressure, but syscalls per-se (outside of things like munmap) don't typically invalidate the TLB as the mapping is not changed on an user-space kernel-space transiation, only the access right (there were exceptions like the brief period when people where running with full 4GB user address space on 32bits cpu).

secondcoming 3y ago

Now that's interesting. I remember years ago profiling the gettimeofday() function on various platforms. It was fine on all except our AWS VMs. Turns out it wasn't implemented as a vDSO and the performance was all over the place. That's been fixed since. We use getpid() in logging and maybe it's unnecessary.

cb321 3y ago

Using https://github.com/c-blake/batch/blob/master/examples/closes.c https://github.com/c-blake/batch/blob/master/examples/closes... from the examples/ directory of https://github.com/c-blake/batch https://github.com/c-blake/batch, I get: L2:bat/examples$ BATCH_EMUL=1 ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 3.815 *$2 + 695.6826 *$3 bootstrap-stderr-corr matrix 0.8872 -0.7856 0.04174 L2:bat/examples$ BATCH_EMUL=1 ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 4.528 *$2 + 695.3265 *$3 bootstrap-stderr-corr matrix 1.174 -0.7640 0.05302 L2:bat/examples$ a (695.3265 +- 0.05302)-(695.6826 +- 0.04174) -0.356 +- 0.067 # i.e. run-to-run consistent L2:bat/examples$ ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 779.67 *$2 + 48.861 *$3 bootstrap-stderr-corr matrix 1.550 -0.7495 0.2023 L2:bat/examples$ ./closes|awk '{print $3,$2,$1*$2}'|fitl -s0 -c,=,n,b -b100 1 2 3 $1= 787.77 *$2 + 48.745 *$3 bootstrap-stderr-corr matrix 1.350 -0.6207 0.1635 L2:bat/examples$ a (48.861 +- 0.2023)-(48.745 +- 0.1635) 0.12 +- 0.26 # i.e. run-to-run consistent L2:bat/examples$ a (695.3265 +- 0.05302)/(48.745 +- 0.1635) 14.265 +- 0.048 # i.e. batching makes it 14X faster! One can go further to get a mean +- std.err of ~695.52 +- ~0.04 nanosec hot-cache time per syscall overhead or even do point-weighting there, but it's probably more important to check fit residuals for serial autocorrelation (very do-able with https://github.com/c-blake/fitl https://github.com/c-blake/fitl, EDIT: oh, and if you care `a` is basically shown in https://github.com/SciNim/Measuremancer/pull/12 https://github.com/SciNim/Measuremancer/pull/12 which uses standard error propagation). The bigger point would not be ever more careful measurement of the cost if you do not, but rather the ease of "batchifying" if you do of any highly regular call interface similar to syscalls with high latency and also the "follow on" API design, granularity-wise. The inner code of a mini-assembly language letting you implement, say, open/fstat/mmap/close all in one syscall crossing is only about 30 lines of C. { Yes, yes..It could use multi-CPU-arch and syscall auditing integration.. and sure, ebpf & io_uring alter the Linux landscape these days }. Also, @exDM69 is very correct that pure hot cache-hot loop numbers are only one part of the cost story (https://news.ycombinator.com/item?id=39188551 https://news.ycombinator.com/item?id=39188551).

gpderetta 3y ago

The measured mitigation cost overhead is very interesting. Those XG machines have very fast syscalls (<100ns), but are more than twice as slow with mitigation on, even though they are fairly recent and most meltdown/spectre bugs should have been patched out.

alberth 3y ago

Is the cost of syscalls expensive for all OS's (e.g. BSD, Redox, etc)? Does OpenBSD limiting how you can call a syscall help at on performance (or hurt)?