4 ms·
Before you look at these benchmarks and concluded system calls are cheap, please be aware the real cost is not 100 cycles, it's 100 cycles plus the cost of clob
by eloff 7y ago
Before you look at these benchmarks and concluded system calls are cheap, please be aware the real cost is not 100 cycles, it's 100 cycles plus the cost of clobbering your L1 and often L2 cache, instruction cache, TLB, etc. This is made worse on many current systems due to mitigations for Intel's buggy processors. The real cost is often more than 10,000-30,000 cycles. Which is why for anything very syscall heavy, like a network server handling millions of very small packets per second you get more than an order of magnitude better performance by bypassing the kernel and avoiding syscalls altogether. It's also why threads are "faster" than processes even although they're implemented the same way, no need to flush the TLB on context switch.
- BubRoss 7y agoIs it possible to use larger chunks of data when working with packets from the kernel?
- tenebrisalietum 7y agohttps://en.wikipedia.org/wiki/New_API https://en.wikipedia.org/wiki/New_API
- PaulDavisThe1st 7y agoAny idea on how this would apply to something like clock_gettime(CLOCK_MONOTONIC) ? This seems to have extremely low overhead ...
- wtallis 7y agoThat's one of the functions provided by the vDSO, so userspace code can call it without a context switch: http://man7.org/linux/man-pages/man7/vdso.7.html http://man7.org/linux/man-pages/man7/vdso.7.html
- cesarb 7y agoThe current trend seems to be to use ring buffers between the kernel and userspace (see: perf, io_uring, this proposal https://lwn.net/Articles/789603/ https://lwn.net/Articles/789603/), which reduce this overhead.
- eloff 7y agoAt the lowest level this is how it works, so using ring buffers as the abstraction makes tremendous sense. Plus it can be done safely without requiring a context switch on the fast path, with a little care. Its also easy to naturally batch things. When the load is low the batch size is 1, and then it naturally grows as more work gets queued during processing of the prior work. I love ring buffers and queues.
- Veserv 7y agoAlthough you are technically correct in that the cost of system calls on most systems is high, your post also implies to a casual reader that the cost is fundamental. This is not true. As demonstrated in the benchmark, the hardware cost of systems calls/trapping is quite low. Even accounting for hardware workarounds (which were not even known about when the common consensus decided system calls were slow), excepting the case of hardware virtualization which has much more expensive mitigations, the fundamental cost is a fraction of the actual cost. The majority of the cost (~90+%) is actually due to software. In the system call case this is the implementation of the system call. In the context switch case this is usually the cost of the scheduling decision. These operations take material amounts of time and access memory which evicts user caches. It is in speeding up the software component that real gains can be made. This is not to say that system calls could ever be made "free" since function calls/loads/stores will still be ~100x faster, but the cost would probably be ~10% of the current cost if they software component could magically be made instant. As an example of what I am talking about, there is this talk by Google from 2013 [1] which describes a "fast context switch" system call (benchmark at 18:40). This "context switch" knows what it is switching to which lets it bypass the scheduling decision in normal context switches. As a result it takes ~170 ns (which is pretty close to the hardware cost mentioned in the OP) vs 2000 ns of a normal context switch. I could go into more detail as to some of the causes of the underlying software costs, but that is a discussion for another time, so I will just leave this off with summarizing my point. tl;dr: System calls are slow. It is due to the software implementing them, not the hardware. [1]: https://www.youtube.com/watch?v=KXuZi9aeGTw https://www.youtube.com/watch?v=KXuZi9aeGTw
- jeffnappi 7y agoWould BPF count as "bypassing the kernel" or are you referring to hardware acceleration? e.g. https://blog.cloudflare.com/cloudflare-architecture-and-how-bpf-eats-the-world/ https://blog.cloudflare.com/cloudflare-architecture-and-how-...
- eloff 7y agoBPF is more like moving your compute into the kernel, but it achieves the same effect - you avoid context switch overhead associated with syscalls. Kernel bypass is typically using ring buffers and a poll-mode network driver (e.g. DPDK, snabbswitch) and skipping the whole kernel network stack.