4 ms·
Pipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my
by crazygringo 8y ago
Pipes are awesome and infuriating.
Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right.
But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bunzip would just inexplicably stop after about 1GB, but it was at a different exact spot every time (between 1GB and 1.5GB). No error message, no exit, my network connection is fine, just an infinite timeout. (While curl by itself worked flawlessly every time.)
And I've got another 10 stories like this (I do a lot of data processing). Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not. And even more frustrating, they'll often work on your local machine but not on your server, or vice-versa. And I'm just running basic commodity macOS locally and out-of-the-box Ubuntu on my servers.
I don't know why, but many times I've had to rewrite a piped command as streams in a Python script to get it to work reliably.
- oweiler 8y agoI've built a pipe which in very rare cases ran into a segfault. Never found out why.
- EE84M3i 8y agoPerhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?
- crazygringo 8y agoBecause it simply never occurred to me to check if ssh would have compression built-in. Because why would it? If the UNIX philosophy is to separate out tools and pipe them, then the UNIX philosophy should be to pipe through gzip and gunzip, not for ssh to provide its own redundant compression option, right?
- acdha 8y agoThis is a good example of where that simple rule breaks down: piping it would only work when you are running a command and feeding its output to a different location whereas having it in SSH helps with everything so e.g. typing `ls` in a big directory, `cat`-ing a file, using `scp` on text, etc. benefits.
- invsblduck 8y ago> Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not. While this may be your experience, the mechanism of FIFO pipes in Unix (which is filehandles and buffers, basically), is an old one that is both elegant and robust; it doesn't "randomly" fail due to unreliability of the core algorithm or components. In 20 years, I never had an init script or bash command fail due to the pipe(3) call itself being unreliable. If you misunderstand detailed behavior of the commands you are stitching together--or details of how you're transiting the network in case of an ssh remote command--then yes, things may go wrong. Especially if you are creating Hail Mary one-liners, which become unwieldy.
- laumars 8y agoIf got to agree. I can’t recall a pipe ever failing due to unreliability. One issue I did used to have (before I discovered ‘-o pipefail’[1]) was the annoyance that if an earlier command in a pipeline failed, all the other commands in the pipeline still ran albeit with no data or garbage data being piped to them. [1] https://stackoverflow.com/questions/1550933/catching-error-codes-in-a-shell-pipe#comment28578902_4959616 https://stackoverflow.com/questions/1550933/catching-error-c...