5 ms·
The Effect of Pipe Capacity on Unix Pipeline Performance
- steerablesafe 7y agoI expected some examples and benchmarks with some common unix tools.
- AdmiralAsshat 7y agoDitto. This article feels like it was published before the author had finished vetting out everything he/she wanted to demonstrate.
- dongyx 7y agoYou're right. I didn't provide enough examples. I found this theory in my real-world project. I've done a series of experiments and benchmarks. However, the project is commercial and I can't demonstrate it. I'll build some simple examples in the future if I have time.
- lotwxyz 7y agoBelieve it or not: I have to deal with issues just like this over in webdev-land! (See the Terminal app in https://dev.lotw.xyz/desk.os https://dev.lotw.xyz/desk.os or skip right to it, go to https://dev.lotw.xyz/shell.os https://dev.lotw.xyz/shell.os)
- anonsivalley652 7y agoAwhhh. :'( I wanted to see: - What happens when piping an 8 TiB file through awk to be stored on another physical volume (proly want to use xfs). - A graph of various block sizes, various file sizes, pipeline sizes and timing using a HPET. I don't know how you make an 2+-way pipe like it the diagrams except by process substitution, except that doesn't always work and you can't create cycles without creating FIFO (named pipes): cmd0 <(cmd1 here) <(cmd2 here) mkfifo -m600 ~joe/foo cmd0 <(cmd1 here) <(cmd2 here) | cmd3 > ~joe/foo & cmd4 <(cmd5 here) >(cmd6 here ~joe/foo) <>(...) # outer can r/w <(...) # outer can r >(...) # outer can w
- wnoise 7y agoMost shells don't have great support for making even non-linear graphs of (anonymous) pipe connections (though note that wilder, even dynamic graphs are possible with the "coproc" builtin of e.g. ksh, zsh, or modern bash). However, arbitrary graphs are possible with the system calls available -- just create all the pipes in the spawning process with pipe(), and after fork(), but before exec() move the right ends into the file descriptors 0, 1, and 2 with dup2().
- pushedx 7y agoThe diagrams are not diagrams of pipelines, they are diagrams of a single pipeline `a | b`, over time, with points in time shown when a is writing and when b is reading.
- thedance 7y agoI wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.
- jandrese 7y agoThe vast majority of shell script writers don't know or care what the pipe size is. The tools they are using treat them as just one solid stream of data. There's not much to break.
- thedance 7y agoI meant in the kernel. Like if I hooked up a fuzzer and just started swinging the pipe size around.
- jschwartzi 7y agoConsidering there's a semi-contiguous allocation in the kernel I bet you'd find some interesting bugs that way. I don't think very many people mess with the pipe sizes other than to tune to a multiple of the input/output block size for the application. And if you're trying to maximize throughput you're likely to reach your optimum block count long before you start blowing out buffers. Actually, come to think of it named pipes would be another good place to fuzz.
- cat199 7y agothe shell isn't the only user of pipe(2) - that said, since you can't count on fixed scheduling on either end, most things likely will handle things okay
- barrkel 7y agoUnless there's a dialog going on, it shouldn't matter a great deal. But it's easy to mess up a dialog (having a conversation with both stdin and stdout).
- duskwuff 7y agoVery little. The Linux default changed from 4 KB to 64 KB in 2.6.11, and the macOS XNU kernel will actually adjust pipe sizes dynamically based on the size of writes and system memory pressure.
- duskwuff 7y ago> If the two processes exchange data in this pattern [one-to-one exchange], a small pipe doesn’t cause unnecessary blockings. I don't think this is correct! Context switches are expensive -- a small pipe size will force both processes to make more system calls to move the same amount of data through the pipe. In this light, the UNIX specification's requirement of 4 KB is definitely too small; even the Linux 2.6+ default of 64 KB feels like it might be on the small side.
- dongyx 7y agoYou're right. The words I took are not precise. I don't know how to express the concept that the performance is not essentially affected. Is there a universally acknowledged concept to distinguish the difference? Like the concept of time complexity in the algorithm area.