9 ms·
Linux Pipes Are Slow
- stabbles 2y agoA bold claim for a blog that takes about 20 seconds to load.
- yas_hmaheshwari 2y agoThis post has gone to the top of hacker news, so I think we should give him some slack Looks like an amazing article, and so much to learn on what happens under the hood
- ben-schaaf 2y agoHN generates ~20k page views over the course of a day with a peak of 2k/h: https://harrisonbroadbent.com/blog/hacker-news-traffic-spike-anatomy/ https://harrisonbroadbent.com/blog/hacker-news-traffic-spike.... At ~1MB per page load - not sure how accurate this is, I don't think it fully loaded - this static blogpost requires 0.55MB/s to meet demand. An original raspberry pi B (10mpbs ethernet) on the average french mobile internet connection (8mbps) provides double that. I don't mean this as a slight to anyone, I just want to point out the HN "hug of death" can be trivially handled by a single cheap VPS without even breaking a sweat.
- qsantos 2y agoTotally agree, my server should definitely be able to handle the load. But this is a WordPress install, which is definitely doing too much work for what it is when just serving the pages. I plan to improve on this!
- wvh 2y agoI believe that when it's a .fr, they call it nonchalance...
- deleted 2y ago[deleted]
- jheriko 2y agojust never use pipes. they are some weird archaism that need to die :P the only time ive used them is external constraints. they are just not useful.
- henearkr 2y agoPipes are extremely useful. But I guess it just depends on your use case. I do a lot of scripting. If you dislike their (relative) slowness, it's open source, you can participate in making them faster. And I'm sure that after this HN post we'll see some patches and merge requests.
- noloblo 2y ago+1 yes pipes are what shell scripting quite useful and allow for easy composition of the different unix shell utilities
- hnlmorg 2y agoThe very thing that makes pipes useful is what also makes them slow. I don't think there is much we can do to fix that without breaking POSIX compatibility entirely. Personally I think there's much worse ugliness in POSIX than pipes. For example, I've just spent the last couple of days debugging a number of bugs in a shell's job control code (`fg`, `bg`, `jobs`, etc). But despite its warts, I'm still grateful we have something like POSIX to build against.
- nitwit005 2y agoJust about every form of IPC is "slow". You have decided to pay a performance cost for safety.
- marcosdumay 2y agoYou shouldn't have to pay that much. Pipes give you almost nothing, so they should cost almost nothing. Specifically, there aren't many reasons for your fastest IPC to be slower than a long function call.
- nitwit005 2y agoIf you don't think pipes offer much, don't use them. Saying "long function call" doesn't mean much since a function can take infinitely long.
- marcosdumay 2y agoA long distance function call, that invalidates everything on your cache.
- saagarjha 2y ago…which is quite expensive.
- marcosdumay 2y agoYes, it is. But it's much cheaper than interacting by pipe. Linux is optimizing sockets with a similar goal. And it's quite far on that direction. But there's still some margin to gain.
- brigade 2y agoPipes don’t exist for safety, they exist as an optimization to pass data between existing programs.
- djaouen 2y agoSo is Python, but I'm still gonna use it lol
- RevEng 2y agoI didn't quite grasp why the original splice has to be so slow. They pointed out what made it slower than vmsplice - in particular allocating buffers and using scalar instructions - but why is this necessary? Why couldn't splice just be reimplemented as vmsplice? I'm sure there is a good reason, but I've missed it.
- Izkata 2y ago> Why couldn't splice just be reimplemented as vmsplice? A possible answer that's currently just below your comment: https://news.ycombinator.com/item?id=41351870 https://news.ycombinator.com/item?id=41351870 > vmslice doesn't work with every type of file descriptor.
- koverstreet 2y agoOne of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)
- wakawaka28 2y agoBuffering is there for a reason and this approach will lead to weird failure modes and fragility in scripts. The core issue is that any stream producer might go slower than any given consumer. Even a momentary hiccup will totally mess up the pipe unless there is adequate buffering, and the amount needed is system-dependent.
- foota 2y agoMaybe I misunderstand, but if the ring buffer is full isn't it ok for the sender to just block?
- mort96 2y agoYeah, and if the ring buffer is empty it's okay for the receiver to just block... exactly as happens today with pipes
- Spivak 2y agoWhat makes this any different than other buffer implementations that have a max size? Buffer fills, writes block. What failure mode are you worried about that can't occur with pipes which are also bounded?
- hackernudes 2y agoI think the OP's proposal has buffering. It is different from a pipe - instead of using read/write to copy data from/to a kernel buffer, it gives user space a mapped buffer object and they need to take care to use it properly (using atomic operations on the head/tail and such). If you own the code for the reader and writer, it's like using shared memory for a buffer. The proposal is about standardizing an interface.
- 0xbadcafebee 2y agoCalling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
- AkBKukU 2y agoI have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.
- whartung 2y agoWhat about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.
- AkBKukU 2y agoIt looks like FFmpeg does support reading from sockets natively[1], I didn't know that. That might be a better solution in this case, I'll have to look into some C code for writing my output to a socket to try that some time. [1] https://ffmpeg.org/ffmpeg-protocols.html#unix https://ffmpeg.org/ffmpeg-protocols.html#unix
- ptx 2y agoWhy should sockets be faster?
- uncanneyvalley 2y agoSockets remap pages without moving any data while pipes have to copy the data between fds.
- 2y ago
- JoshTriplett 2y agoThis is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Move String", which makes "rep movsb" faster for shorter moves too).
- jeffbee 2y agoAlso worth noting that Linux has changed the way it uses ERMS and FSRM in x86 copy multiple times since kernel 6.1 used in the article. As a data-dote, my machine that has FSRM and ERMS — surprisingly, the latter is not implied by the former — hits 17GB/s using plain old pipes and a 32KiB buffer on Linux 6.8
- Lockal 2y agoThis is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster. All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S.html#19 https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch... And they are not final, i. e. Noah Goldstein still updates them every year.
- jeffbee 2y agoWhich is these is "faster" depends greatly on whether you have the very rare memcpy-only workload, or if your program actually does something useful. Many people believe, often with good evidence, that the most important thing is for memcpy to occupy as few instruction cache lines as is practical, instead of being something that branches all over kilobytes of machine code. For comparison, see the x86 implementations in LLVM libc. https://github.com/llvm/llvm-project/blob/main/libc/src/string/memory_utils/x86_64/inline_memcpy.h#L158 https://github.com/llvm/llvm-project/blob/main/libc/src/stri...
- adrian_b 2y agoIt depends on the CPU. There is no good reason for "rep movsb" to be slower at any big enough data size. On a Zen 3 CPU, "rep movsb" becomes faster than or the same as anything else above a length slightly greater than 2 kB. However there is a range of multi-megabyte lengths, which correspond roughly with sizes below the L3 cache but exceeding the L2 cache, where for some weird reason "rep movsb" becomes slower than SIMD non-temporal stores. At lengths exceeding the L3 size, "rep movsb" becomes again the fastest copy method. The Intel CPUs have different behaviors.
- nyanpasu64 2y agoHow do you gather profiling information for kernel function calls from a user program?
- qsantos 2y agoI'll write an article on the flamegraphs specifically, but to get the data, just follow Julia's article! https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/ https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/
- ismaildonmez 2y agoCould you clarify how are you testing the speed of the first example where you are not writing anything to stdout? Thanks.
- qsantos 2y agoFor the first Rust program, where I just write to memory, I just use the time utility when running the program from zsh. Then, I divide the number of bytes written by the number of seconds elapsed. That's why it's not an infinite loop ;)
- ismaildonmez 2y agoThanks!
- cowsaymoo 2y agoWhat is the library used to profile the program?
- tzury 2y agopv https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv it is in the pipe command `... | pv > /dev/null`
- throw12390 2y ago`pv --discard` is faster by 8% (on my system). % pv </dev/zero >/dev/null 54.0GiB/s % pv </dev/zero --discard 58.7GiB/s
- IWeldMelons 2y agoWhich is suspiciously close to the speed of DDR4.
- qsantos 2y agoI am again getting the hug of death of Hacker News. The situation is better than the last time thanks to caching WordPress pages, but loading the page can still take a few seconds, so bear with me!
- Borg3 2y agoHaha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow! Anyway, nice article, its good to know whats going on under the hood.
- MaxBarraclough 2y agoI'd assumed Cygwin pipes are just Windows pipes, is that not the case?
- tyingq 2y agoNot a comprehensive list of problems, and not current but a good illustrative post of the kind of issues that people have run into in this post: https://cygwin.com/pipermail/cygwin-patches/2016q1/008301.html https://cygwin.com/pipermail/cygwin-patches/2016q1/008301.ht...
- Borg3 2y agoIts not that easy. Yeah, they are, but there is a lot of POSIX like glue inside so they work correctly with select() and other alarms. Code is very complicated. But still, kudos for Cygwin Developers for creating Cygwin :) Great work, even tho it have some issues.
- goodpoint 2y agoExcellent article even if, to be honest, the title is clickbait.
- chmaynard 2y agoAgreed. Titles that don't use quantifiers are almost always misleading at best.
- sixthDot 2y ago> I do not know why the JMP is not just a RET, however. The jump seems generated by the expansion of the `ASM_CLAC` macro, which is supposed to change the EFLAGS register ([1], [2]). However in this case the expansion looks like it does nothing (maybe because of the target ?). I 'd be interested to know more about that. Call to the wild. [1]: https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/smap.h#L17 https://github.com/torvalds/linux/blob/master/arch/x86/inclu... [2]: https://stackoverflow.com/a/60579385 https://stackoverflow.com/a/60579385
- fatcunt 2y ago> I do not know why the JMP is not just a RET, however. This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/linkage.h#L23 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/retpoline.S#L121 https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret... > The NOP instructions at the beginning and at the end of the function allow ftrace to insert tracing instructions when needed. These are from the ASM_CLAC and ASM_STAC macros, which make space for the CLAC and STAC instructions (both of them three bytes in length, same as the number of NOPs) to be filled in at runtime if X86_FEATURE_SMAP is detected. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/smap.h#L17-L26 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/include/asm/cpufeatures.h#L264 https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/kernel/alternative.c#L265 https://github.com/torvalds/linux/blob/v6.1/arch/x86/kernel/...
- qsantos 2y agoThanks a lot for the information! I was not quite sure what to look for in this case. I have added in note in the article.
- ndesaulniers 2y agoThere are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle
- michaelcampbell 2y agoI believe there are a lot more of your 2nd point than you might think.
- opello 2y agoAnd certainly those interested in the performance implications of the retpoline mitigation for the various speculative execution information leaks.
- arendtio 2y agoI know pipes primarily from shell scripts. Are they being used in other contexts as extensively, too? Like C or Rust programs?
- guenthert 2y agoMost shells are C programs, so it's clearly possible to use pipes there (and consequently any language with a C FFI, including Rust) and it's done. It's cumbersome though. That's why there are so called glue languages, including shells. Compiler and similar tools might establish pipes and will do so directly, rather than via a glue language. Perhaps a grep through github could tell how 'extensive' this truly is. For the data transfer rate it doesn't matter how (using which language) the pipe is established; C and Rust and the like will have a (small) edge up in the start-up time (latency) though.
- rwmj 2y agoBe interesting to see a version using io_uring, which I think would let you pre-share buffers with the kernel avoiding some copies, and avoid syscall overhead (though the latter seems negligible here).
- qsantos 2y agoThat sounds like a good idea!
- rwmj 2y agoI'm not claiming it'll be faster! Additionally io_uring has its own set of challenges, such as whether it's better to allocate one ring per core or one ring per application (shared by some or all cores). Pre-sharing buffers has trade-offs too, particularly in application complexity [alignment, you have to be careful not to reuse a buffer before it is consumed] versus the efficiency of zero copy.
- donaldihunter 2y agoSomething I didn't see mentioned in the article about AVX512, aside from the xsave/xrstor overhead, is that AVX512 is power hungry and causes CPU frequency scaling. See [1], [2] for details and as an example of how nuanced it can get. [1] https://www.intel.com/content/dam/www/central-libraries/us/en/documents/cryptography-processing-with-3rd-gen-intel-xeon-scalable-processors-19-may-2021.pdf https://www.intel.com/content/dam/www/central-libraries/us/e... [2] https://www.intel.com/content/www/us/en/developer/articles/technical/accelerating-x265-with-intel-advanced-vector-extensions-512-intel-avx-512.html https://www.intel.com/content/www/us/en/developer/articles/t...
- Narishma 2y agoThat is only the case in specific Intel CPU models.
- up2isomorphism 2y agoSomeone tasted a bread thinking it is not sweet enough, which is fine. But calling the bread bland is funny because it does not mean to taste sweet.
- mparnisari 2y agoI get PR_CONNECT_RESET_ERROR when trying to open the page
- qsantos 2y agoMy server struggles a bit with the load on the WordPress site. You should be fine just reloading. I will make sure to improve things for the next time!
- jvanderbot 2y ago> Although SSE2 is always available on x86-64, I also disabled the cpuid bit for SSE2 and SSE to see if it could nudge glibc into using scalar registers to copy data. I immediately got a kernel panic. Ah, well. I think you need to recompile your compiler, or disable those explicitly via link / cc flags. Compilers are fairly hard to get to coax / dissuade SIMD instructions, IMHO.
- faizshah 2y agoThis is a really cool post and that is a massive amount of throughput. In my experience in data engineering, it’s very unlikely you can exceed 500mb/s throughput of your business logic as most libraries you’re using are not optimized to that degree (SIMD etc.). That being said I think it’s a good technique to try out. I’m trying to think of other applications this could be useful for. Maybe video workflows?
- jeremyscanvic 2y agoGreat post! I didn't know about vmsplice(2). I'm glad to see a former ENSL student here as well!
- qsantos 2y agoHey!
- yencabulator 2y agoFUSE can be a bit trickier than a single queue of data chunks. Reads from /dev/fuse actually pick the right message to read based on priorities, and there's cases where the message queue is meddled with to e.g. cancel requests before they're even sent to userspace. If you naively switch it to eagerly putting messages into a userspace-visible ringbuffer, you might significantly change behavior in cases like interrupting slow operations. Imagine having to fulfill a ringbuf worth of requests to a misbehaving backend taking 5sec/op, just to see the cancellations at the very end.
- Marthcass 2y ago[flagged]