19 ms·
For the Love of Pipes
- yboris 8y agoI love pipes so much! In particular, I love pipes in Angular. I gave a talk about including a bit about pipes in Angular: https://youtu.be/Gv7IfU78vxw?t=578 https://youtu.be/Gv7IfU78vxw?t=578
- alias_neo 8y agoAnyone else disappointed that this wasn't some in-depth, Docker related pipe usage post? For those of us that know who Jess is, the post is a little lacking. I went there to read about some cool things she's doing with pipe, but was disappointed at a post that contained nothing.
- moonbug 8y agoThanks for the insight, internet guy.
- Insanity 8y agoI disagree - the pipe command is trivial at least to us whom have used linux (or unix) based systems for a long time. But to people who follow her and are rather new to this way of doing things, this could be a good read. Either way, it is nice to be reminded of the unix philosophy about chaining tools together for junior and senior alike. Edit: typo fixes
- alias_neo 8y agoFair enough. I'm a little surprised at the response. I didn't expect many people here aren't already intimately familiar with pipes, and for those that aren't *nix users, I'm not sure the content is of interest. I guess being honest around here is the wrong thing.
- zeebeeblebrox 8y ago>I guess being honest around here is the wrong thing. Imagine being this full of yourself.
- Insanity 8y agoFor what it's worth, I had no issue with your opinion and also didn't downvote either of your comments. And I also didn't upvote the article, because it is indeed quite basic. Though apperantly enough people did find it interesting enough to make it to the frontpage ^^
- depr 8y agono
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]
- nvarsj 8y agoI think you're being downvoted because of: "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." Having said that, it does seem like the author's posts seem to get upvoted no matter what by a contingent of readers. I also found TFA to be quite basic, and it couldn't have taken very much effort to write (I'm assuming the author is already well familiar with pipes based on her background). The article would have been more interesting with some more technical details or experimental results - or perhaps some novel information that most people aren't aware of.
- alias_neo 8y agoIt's unfortunate. I generally tend not to comment on things I want to criticise. I love Jess's work but saw zero value come from the time spent reading that. I think my point was more valid, in this case, given that this isn't some random person trying to contribute something, I'd have been less critical in that case and not commented, but this is Jess Frazelle. Those of us that know the name know she's an extremely skilled engineer, and _that_ is why I was disappointed at the post.
- darrenf 8y agoI twitched horribly at the final sentence, screaming inwardly "you don't pipe to /dev/null, you redirect to it". And now I feel like an arsehole.
- analpaper 8y agoredirect your feelings to /dev/null, because a pipe will just give us a Permission denied
- benj111 8y agoChmod +x /dev/null (havent tried above, not sure I recommend that you do)
- jgtrosh 8y agoWell you can't read either from /dev/null, and I don't think that's just a question of permissions. I'm pretty sure it's impossible to get /dev/null to behave like an executable.
- arghwhat 8y agoYou can read from /dev/null—it just behaves as a zero-length file, immediately returning EOF. This makes /dev/null a valid source file in many languages, C included.
- benj111 8y agoInteresting question. You could write a executable that accepts piped input and throws it away. When would it exit though? Would it exit successfully at the end of the input stream? That sounds sensible. That would be behaving like an executable wouldn't it?
- jacquesm 8y agoAs long as it does not exit you can still catch it through /proc/
- adem666 8y agoUnix's philosophy of “do one thing well” and “expect the output of every program to become the input to another” is living with "microservices" in nowadays.
- msravi 8y agoawk, grep, sort, and pipe. I'm always amazed at how well thought out, simple, functional, and fast the unix tools are. I still prefer to sift through and validate data using these tools rather than use excel or any full-fledged language. Edit: Also "column" to format your output into a table.
- gpderetta 8y agoAlthough I probably use it multiple times everyday, I hate column. At least the implementation I use has issues with empty fields and a fixed maximum line length. Edit: s/files/fields/
- icebraining 8y agoFor an alternative view, don't forget to read the section on Pipes of The Unix-Haters Handbook: http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~simsong/www/ugh.pdf (page 198)
- bagrow 8y ago> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)
- deleted 8y ago[deleted]
- pytester 8y agoThe core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the object. You can’t send pointers into another process’s address space. You can’t send file handles or tcp connections or permissions to access particular files or resources.
- majewsky 8y ago> You can’t send pointers into another process’s address space. Thank goodness.
- jimktrains2 8y agoTo be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.
- AnIdiotOnTheNet 8y ago
- aaaaaaaaaab 8y agoIf only there was some standard format to interchange structured data over pipes, other than plain text delimited with various (incompatible) combinations of whitespaces, using various (incompatible) escaping schemes.
- Kipters 8y agoThat's one of the design goals of Powershell, you don't pass streams of text between cmdlets, you pass objects, which have a type, properties and methods you can use in standard ways (they may also be strings)
- peterwwillis 8y agoA while back I made a PoC that used pipes to create bidirectional json-speaking connections between applications and then distribute the communication between distributed nodes. The idea being, don't just distribute TCP streams between distributed processes, but give 'em objects to make data exchange more expressive. I don't know where my PoC code went, but it wasn't very stable anyway. Just figured it was something we should have adopted by now (but that Plan9 probably natively supports)
- aquabeagle 8y agolibxo
- benj111 8y agoWhy isn't the pipe a construct that has caught on in 'proper' languages?
- arianvanp 8y agoIt is! It's the main way of programming in lazy functional programming languages like Haskell And many programming languages have libraries for something similar:. iterators in rust / c++, streams in java/c#, thinks like reactive
- benj111 8y agoHaskell was the only possible I found. Iterators don't really fully capture what a pipe is though? Theres no parallelism. And streams don't have the conceptual simplicity of a pipe?
- masklinn 8y ago> Iterators don't really fully capture what a pipe is though? Theres no parallelism. Pipes are concurrent, not necessarily parallel. Iterators are concurrent, and can be parallel (https://docs.rs/rayon/0.6.0/rayon/par_iter/index.html https://docs.rs/rayon/0.6.0/rayon/par_iter/index.html).
- lccarrasco 8y agoF# has a pipe operator
- Ndymium 8y agoElixir also has a pipe construct: f |> g(1) would be equivalent to g(f, 1)
- duckerude 8y agoMany functional languages have |> for piping, but chained method calls are also a lot like pipelines. Data goes from left to right. This javascript expression: [1, 2, 3].map(n => n + 1).join(',').length Is basically like this shell command: seq 3 | awk '{ print $1 + 1 }' | tr '\n' , | wc -c (the shell version gives 6 instead of 5 because of a trailing newline, but close enough)
- mpweiher 8y agoYes, pipes are awesome, and the concepts actually translate well to in-process usage with structured data. https://github.com/mpw/MPWFoundation/blob/master/Documentation/Streams.md https://github.com/mpw/MPWFoundation/blob/master/Documentati... One aspect is that the coordinating entity hooks up the pipeline and then gets out of the way, the pieces communicate amongst themselves, unlike FP simulations, which tend to have to come back to the coordinator. This is very useful in "scripted-components" settings where you use a flexible/dynamic/slow scripting language to orchestrate fixed/fast components, without the slowness of the scripting language getting in the way. See sh :-) Another aspect is error handling. Since results are actively passed on to the next filter, the error case is simply to not pass anything. Therefore the "happy path" simply doesn't have to deal with error cases at all, and you can deal with errors separately. In call/return architectures (so: mostly everything), you have to return something, even in the error case. So we have nil, Maybe, Either, tuples or exceptions to get us out of Dodge. None of these is particularly good. And of course | is such a perfect combinator because it is so sparse. It is obvious what each end does, all the components are forced to be uniform and at least syntactically composable/compatible. Yay pipes.
- jarpineh 8y agoI recently came across Ramda CLI's interactive mode [1] It essentially hijacks pipe's input and output into browser where you can play with the Ramda command. Then you just close browser tab and Ramda CLI applies your changed code in the pipe, resuming its operation. Now I'm thinking all kinds ways I use pipe that I could "tee" through a browser app. I can use browser for interactive JSON manipulation, visualization and all around playing. I'm now looking for ways to generalize Ramda CLI's approach. Pipes, Unix files and HTTP don't seem directly compatible, but the promise is there. Unix tee command doesn't "pause" the pipe, but probably one could just introduce pause/resume output passthrough command into the pipe after it. Then web server tool can send the tee'd file to browser and catch output from there. [1] https://github.com/raine/ramda-cli#interactive-mode https://github.com/raine/ramda-cli#interactive-mode
- avodonosov 8y agoYou can just store the first pipeline results in a file, edit it, then use the file as an input for the second pipeline.
- jarpineh 8y agoWell, yes, but this kind defeats transient nature of data moving through pipe. Testing and debugging and operating on pipe based processing benefits from close feedback cycle. I’d rather keep that as much as possible.
- MichaelMoser123 8y agoI think the Unix pipeline works because this it has this moldable and expressive text substance that is exchanged. Gstreamer also has pipelines, but the result in my opinion is quite awkward because events of many types are exchanged. Windows Powershell also has a pipeline where objects are exchanged, but it also somehow failed to become a huge success. i think the unix pipeline concept doesn't quite scale to other domains that try to exchange a different unit of information between the pipeline elements.
- kingosticks 8y agoWhile I do agree they can sometimes be tricky to use, Gstreamer pipelines are also it's best feature. Is there a better way?
- MichaelMoser123 8y agoffmpeg took a different approach (sort of like a single program/system with many command line options). Some say its easier to use (at least easier than gst-launch)
- kingosticks 8y agoI find the ffmpeg options dizzying whereas the single syntax for gst-launch and parse_bin_from_description is pretty neat. But then I guess you've still got a selection of randomly named properties to discover and correctly set.
- MichaelMoser123 8y agooccasionally they have events exchanged that can't be implemented with gst-launch, you have to do it in C. Go figure.
- viraptor 8y ago> Windows Powershell also has a pipeline where objects are exchanged, but it also somehow failed to become a huge success. I believe it is a huge success in the crowd it was aimed for. (non-trivial size Windows admins)
- milankovic 8y agoDoug be praised.. https://en.wikipedia.org/wiki/Douglas_McIlroy https://en.wikipedia.org/wiki/Douglas_McIlroy
- agumonkey 8y agohas there been proposals/research on having non linear / branching pipe syntax ? a |1 b | c | d, |1 e aka piping a to both b,c,d and e branches ?
- deleted 8y ago[deleted]
- masklinn 8y agoSmalltalk's "method cascading" (the `;` operator) did roughly that. I don't think you're forking an entire sequence of methods though, it just allows performing operations on the same root object e.g. a foo ; bar ; baz would sequentially send "foo", "bar" then "baz" to "a", then would return the result of the last call. The ability to fork iterators (which may be the semantics you're actually interested in) also exists in some languages e.g. Python (itertools.tee) or Rust (some iterators are clonable)
- gpderetta 8y agotee ? edit: a | tee >(b | c | d) | e
- agumonkey 8y agoprobably the closest solution.. thanks
- 8y ago
- mothsonasloth 8y ago[Quote] The Unix philosophy is documented by Doug McIlroy as: Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them. I really like the last two, if you can do them in development then you are then you have a great dev culture
- Waterluvian 8y agoThe last one is especially interesting to me these days. On a macro scale it sure sounds a whole lot like the robot revolution taking unskilled jobs. But of course that's probably not the author's intended context.
- marcosdumay 8y agoProgrammers have been destroying programmer jobs since those jobs exist. Up to now it has meant we have enough productivity for going into more markets, but that will not last forever.
- quietbritishjim 8y agoReformatted to be readable: > Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”. > Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input. > Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them. > Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.
- TuringTest 8y agoIt's sad that developers have had access to pipes for decades, yet the most similar feature they provide to end users is still copy/paste.
- sudhirj 8y agoSanjay Ghemawat (the other less visible half of Jeff Dean) wrote a pipe library in Go, learnt quite a bit from it. https://github.com/ghemawat/stream https://github.com/ghemawat/stream Edit: Jeff Dean, not James Dean
- lincpa 8y agoThe Pure Function Pipeline Data Flow https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow Using the input and output characteristics of pure functions, pure functions are used as pipelines. Dataflow is formed by a series of pure functions in series. A dataflow code block as a function, equivalent to an integrated circuit element (or board)。 A complete integrated system is formed by serial or parallel dataflow. data-flow is current-flow, function is chip, thread macro (->>, -> etc.) is a wire, and the entire system is an integrated circuit that is energized.
- jacquesm 8y agoYes, there is a strong connection between pipes and functional programming, they all transform what passes through them and should not retain state when implemented properly.
- kitd 8y agoThey also both rely on very generic data types as input and output. Which is why I was surprised to see the warning about avoiding columnar data. Tabular data is a basic way of expressing data and their relationships.
- timvisee 8y agoCool, the pipe command must be one of the most essential things in Unix/Linux based systems. I would have loved to see some awesome pipe examples though.
- YesThatTom2 8y agoOk, here are some example pipelines: A simple virus scanner in one line of pipe: https://everythingsysadmin.com/2004/10/whos-infected.html https://everythingsysadmin.com/2004/10/whos-infected.html And a bunch of pipe tricks that are oh so wrong but oh so useful: https://everythingsysadmin.com/2012/09/unorthodoxunix.html https://everythingsysadmin.com/2012/09/unorthodoxunix.html
- hoorayimhelping 8y agodestroyallsoftware's screencasts have pretty good pipe usage / unix-fu.
- bigger_cheese 8y agoBack when I first started using Linux you could pipe random data to /dev/dsp and the speakers would emit various beeps. Used to be a pretty cool trick I think when ASLA came out it stopped working.
- JustSomeNobody 8y agoPipes are awesome. Even in more general programming the pipes and filters pattern is very important and yet a lot of devs still don't know it.
- alkonaut 8y agoI love the idea of simple things that can be connected in any way. I'm not so much a fan of "everything is a soup of bytes with unspecified encoding and unknown formatting". It's an abstraction that held up quite well, but its starting to show its age.
- Jyaif 8y ago100% agree. Having to extract information with regular expressions is a waste of time. If the structure of the data was available, you would have type safety / auto-completion. You could even have GUIs to compose programs.
- bdamm 8y agoStructured data flows in pipes too. Json can even be line oriented. GUI programming fails when you get past a few hundred “lines” of complexity. What I’d love to see is a revolution of shells and terminals to more easily work and pull from piped data.
- jenscow 8y agoI hear what you're saying. However, how can you ensure the output type of one program matches the input type of another?
- testvox 8y agoAllow programs to specify the type of data they can consume and the type of the data they emit. This is how powershell does it (using the dotnet type system).
- jenscow 8y agoAnd the problem is how can you ensure the output type of one program matches the input type of another. A program emits one type, and the other program accepts another. Something will be needed to transform one type into another. Imagine doing that on the command line.
- yaakushi 8y agoI'm probably nitpicking, but if you're using cat to pipe a single file into the sdtin of another program, you most likely don't need the cat in the first place, you can just redirect the file to the process' stdin. Unless, of course, you're actually concatenating multiple files or maybe a file and stdin together. Disclaimer: I do cat-piping myself quite a bit out of habit, so I'm not trying to look down at the author or anything like that! :)
- tyingq 8y agoThe "useless use of cat" was a repeated gripe on Usenet back in the day: http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
- empath75 8y agoPeople use cat to look at the file first, then hit up arrow, add a pipe, etc.
- longwave 8y agoYes, this. Quite often I start writing out complex pipelines using head/tail to test with a small dataset and then switch it out for cat when I am done to run it on the full thing. And it's often not worth refactoring these things later unless you are really trying to squeeze performance out of them.
- Leace 8y agoI actually think that cat makes it more obvious what's happening in some cases. I had recently built a set of tools used primarily via pipes: (tool-a | tool-b | tool-c) and it looks clearer when I mock (for testing) one command (cat results | tool-b | tool-c) instead of re-flowing it just to avoid cat and use direct files.
- toxik 8y agoI think it's also a grammatical wart of shell syntax. Things going into a command are usually on the left, but piping in a file goes on the right.
- Kagerjay 8y agoPiping is fairly common when working with databases and importing CSV files
- LeeJMiller 8y agomy buddy's friend makes $96 hourly on the internet. She has been with out artwork for five months however final month her charge emerge as $12747 really on foot on the internet for some hours. study greater on this net internet site... HERE.......www.moneytechs.com
- j_b_s 8y agoI built a little tool to audit objects in your R pipeline here: https://jabustyerman.com/2017/09/29/auditing-your-r-data-pipeline/ https://jabustyerman.com/2017/09/29/auditing-your-r-data-pip... This basically allows you to tag (wrap) intermediate functions in your pipe, and then inspect the objects generated by that function later.
- talkingtab 8y agoI'm not a Clojure person, but there are transducers, https://clojure.org/reference/transducers https://clojure.org/reference/transducers.
- bjoli 8y agoWhich are in a way comparable, but I'd say that pipes are more like the arrow ->. Transducers are compostable algorithmic transformations that, in a way, generalize map, filter and friends but they can be used anywhere where you transform data. Transducers have to be invoked and handled in a way that pipes do not. Anyone interested should check out Hickeys talks about them. They are generally a lot more efficient than chaining higher order list processing functions and since they don't build.intermediate results they have a lot better GC performance.
- roelb 8y agoFully agree, pipes are awesome, only downside is the potential duplicate serialization/deserialization overhead. Streams in most decent languages closely adhere to this idea. I especially like how node does it, in my opinion one of the best things in node. Where you can simply create cli programs that have backpressure the same as you would work with binary/file streams, while also supporting object streams. process.stdin.pipe(byline()).pipe(through2(transformFunction)).pipe(process.stdout)
- mweberxyz 8y agoNode streams are excellent, but unfortunately don't get as much fanfare as Promises/async+await. A number of times I have gotten asked "how come my node script runs out of memory" -- due to the dev using await and storing the entirety of what is essentially streaming data in memory in between processing steps.
- kgr 8y agoI made a simulation and video which shows why the pipe is so powerful: https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4 I showed it to Doug McIlroy, and while he thought there was more to the story, he didn't disagree with it.
- cmsj 8y agopipe junkies might like to know about the following tools: * vipe (part of https://joeyh.name/code/moreutils/ https://joeyh.name/code/moreutils/ - lets you edit text part way through a complex series of piped commands) * pv (http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml - lets you visualise the flow of data through a pipe)
- foreigner 8y agoYes! I love pv. Besides that and tee, can anyone else suggest some more general pipe tools?
- _whiteCaps_ 8y agopee https://linux.die.net/man/1/pee https://linux.die.net/man/1/pee
- olejorgenb 8y agohttp://joeyh.name/code/moreutils/ http://joeyh.name/code/moreutils/ have a couple more: * pee: tee standard input to pipes (`pee "some-command" "another-command"`) * sponge: soak up standard input and write to a file Though in zsh and bash you can create pee using tee: `tee >(some-command) >(another-command) >/dev/(null`
- deleted 8y ago[deleted]
- peter_retief 8y agoI never really associated the Unix philosophy with unix pipes, now it seems so obvious
- jcelerier 8y agoIf you want to see what the endgame of this is when taking the reasoning to the maximum, look at visual dataflow languages such as Max/MSP, PureData, Reaktor, LabVIEW... Like always, simple stuff will be simple (http://write.flossmanuals.net/pure-data/wireless-connections/static/PureData-DataFlow-sendreceive1-en.png http://write.flossmanuals.net/pure-data/wireless-connections...) and complicated stuff will be complicated (https://ni.i.lithium.com/t5/image/serverpage/image-id/96294i47A75666708C8ECA?v=1.0 https://ni.i.lithium.com/t5/image/serverpage/image-id/96294i...). No silver bullet guys, sorry. If you take out the complexity of the actual blocks to have multiple small blocks then you just put that complexity at another layer in the system. Same for microservices, same for actor programming, same for JS callback hell...
- jancsika 8y ago> Like always, simple stuff will be simple (http://write.flossmanuals.net/pure-data/wireless-connections... http://write.flossmanuals.net/pure-data/wireless-connections...) That is not actually simple because the data is flowing across two completely different message passing paradigms. Many users of Max/MSP and Pd don't understand the rules for such dataflow, even though it is deterministic and laid out in the manual IIRC. The "silver bullet" in Max/MSP would be to only use the DSP message passing paradigm. There, all objects are guaranteed to receive their input before they compute their output. However, that would make a special case out of GUI building/looping/branching. For a visual language designed to accommodate non-programmers, the ease of handling larger amounts of complexity with impunity would not be worth the cost of a learning curve that excludes 99% of the userbase. Instead, Pd and Max/MSP has the objects with thin line connections. They are essentially little Rube Goldberg machines that end up being about as readable. But they can be used to do branching/looping/recursion/GUI building. So users typically end up writing as little DSP as they can get away with then uses thin line spaghetti to fill in the rest. That turns out to be much cheaper than paying a professional programmer to re-implement their prototype at scale. But that's a design decision in the language, not some natural law that visual programming languages are doomed to generate spaghetti. Edit: clarification
- jcelerier 8y ago
- wglb 8y agoThis is cool and useful, but not all unix programs follow this convention: * find * cal * vi * emacs * ls These don't use one of standard input/standard output. (edited) and are not fully pipeable. I don't recall seeing a list of programs--tools, in the original description--that distinguish between pipeable and not-pipeable programs. Also, none of the corrective cat/grep code in these threads point out that grep in fact takes file names, so "cat foo | grep stuff" is just a silly no-op.
- fooblat 8y agoWell find, cal, and ls don't take input at all. And vi (well vim) in fact does. If you invoke it with a - for the file name argument, it will read standard input into the buffer. I can't comment on emacs as I don't use it much.
- wglb 8y agoAh, correct on that point. What about standard output? That is, can vi be used in a pipe?
- rmilk 8y agoI use vim as a visual pipe or pipe debugger with undo when I need to perform a series of transformations (grep/sort/cut/lookup/map/run other data tool). Obviously geared towards text files because it is vim. The ! command sends the selection through external command(s) as STDIN and then replaces the selection with STDOUT from the command(s). For example, grep or sort, but can be any command that works with pipes. Buffer is replaced with output (sorted file for example). Undo with U to go back to original data. Redo with R to go forward to transformed data. Command line history is available to add more commmands or correct when you type ! again. Edit a file. Select block (Visual mode shift-V) and type ! or use the whole file with gg!G command. Type in the commands you need to run. Vim also reads stdin if you give - as the filename, like “ls -l | vim -“ so you we can use it at the end of a pipe instead of redirecting to a file. Like I said, I use it as an interactive debugger to assemble pipelines and see the results. vi / vim filter commands: http://vimdoc.sourceforge.net/htmldoc/change.html# http://vimdoc.sourceforge.net/htmldoc/change.html#!
- WhompingWindows 8y agoPipes have been a game-changer for me in R with the tidyverse suite of packages. Base R doesn't have pipes, requiring a bit more saving of objects or a compromise on code readability. One criticism would be that ggplot2 uses the "+" to add more graph features, whereas the rest of tidyverse uses "%>%" as its pipe, when ideally ggplot2 would also use it. One of my most common errors with ggplot2 is not utilizing the + or the %>% in the right places.
- vharuck 8y agoI've always thought of ggplot2's process as building a plot object. Most steps only add to the input. Of course, Hadley admitted it was because he wrote ggplot2 before adopting pipes into his packages.
- heinrichhartman 8y agoI have wondered for a long time why pipes are not used more often in production-grade applications. I have seen plenty pipe use in bash scripts, especially for build and ETL purposes. Other languages, have ways to do pipes as well (e.g. Python https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline https://docs.python.org/2/library/subprocess.html#replacing-...) but I have seen much less use of it. It appears to me, that for more complex applications one rather opts for TCP-/UDP-/UNIX-Domain sockets for IPC. - Has anyone here tried to plumb together large applications with pipes? - Was it successful? - Which problems did you run into?
- aidenn0 8y agoThe biggest issue is that pipes are unidirectional, while not all data flow is unidirectional. Some functional programming styles are pipe-like in the sense that data-flow is unidirectional: Foo(Bar(Baz(Bif(x)))) is analagous to: cat x | Bif| Baz |Bar| Foo Obviously the order of evaluation will depend on the semantics of the language used; most eager languages will fully evaluate each step before the next. (Actually this is one issue with Unix pipes; the flow-control semantics are tied to the concept of blocking I/O using a fixed-size buffer) The idea of dataflow programming[1] is closely related to pipes and has existed for a long time, but it has mostly remained a niche, at least outside of hardware-design languages 1: https://en.wikipedia.org/wiki/Dataflow_programming https://en.wikipedia.org/wiki/Dataflow_programming
- JdeBP 8y agoPipes are not unidirectional on FreeBSD, interestingly.
- crazygringo 8y agoPipes are awesome and infuriating. Sometimes they work great -- being able to dump from MySQL into gzip sending across the wire via ssh into gunzip and into my local MySQL without ever touching a file feels nothing short of magic... although the command/incantation to do so took quite a while to finally get right. But far too often they inexplicably fail. For example, I had an issue last year where piping curl to bunzip would just inexplicably stop after about 1GB, but it was at a different exact spot every time (between 1GB and 1.5GB). No error message, no exit, my network connection is fine, just an infinite timeout. (While curl by itself worked flawlessly every time.) And I've got another 10 stories like this (I do a lot of data processing). Any given combination of pipe tools, there's a kind of random chance they'll actually work in the end or not. And even more frustrating, they'll often work on your local machine but not on your server, or vice-versa. And I'm just running basic commodity macOS locally and out-of-the-box Ubuntu on my servers. I don't know why, but many times I've had to rewrite a piped command as streams in a Python script to get it to work reliably.
- oweiler 8y agoI've built a pipe which in very rare cases ran into a segfault. Never found out why.
- EE84M3i 8y agoPerhaps your example was contrived, but why would you pipe into gzip instead of using transparent ssh compression?
- crazygringo 8y agoBecause it simply never occurred to me to check if ssh would have compression built-in. Because why would it? If the UNIX philosophy is to separate out tools and pipe them, then the UNIX philosophy should be to pipe through gzip and gunzip, not for ssh to provide its own redundant compression option, right?
- acdha 8y agoThis is a good example of where that simple rule breaks down: piping it would only work when you are running a command and feeding its output to a different location whereas having it in SSH helps with everything so e.g. typing `ls` in a big directory, `cat`-ing a file, using `scp` on text, etc. benefits.
- elchief 8y agoI wish there was an operator to allow a branch then join. Like send output to a lookup, do the lookup, join results Can do with named pipes, but an operator would be nice
- timesntimes 8y agoIn Unix, everything is a file. /dev/null is a file but not an executable so it can't even become a process. So you can't pipe anything into it. If you consider this last sentence as a "figure of speech" then you should have probably avoided the use of "pipe" vs "redirect" and used send them to /dev/null.
- josemober 8y agoFantastic article!
- nojvek 8y agoI’m surprised JessFraz who is employed by Microsoft doesn’t talk about powershell pipes at all. Powershell pipes are an extension over Unix pipes. Rather than just being able to pipe a stream of bytes, powershell can pipe a stream of objects. It makes working with pipes so much fun. In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. In poweshell, ls outputs stream of file objects and you can get the field you want by piping to `Get-Item` or sum the file sizes, or filter only directories. It’s very expressive once you’re manipulating streams of objects with properties.
- sevensor 8y agoIf you're using cut and awk to get a field out of ls, maybe what you want is actually stat(1)?
- nijaru 8y agoShe is actually employed at GitHub now. https://twitter.com/jessfraz https://twitter.com/jessfraz
- schwartzworld 8y agoWhich is owned by...
- lordgrenville 8y agoStill MSFT, in a way!
- marpstar 8y agoI've been following her on twitter for awhile. She went to MSFT only a couple years ago. From what I understand her expertise is in Linux.
- npongratz 8y ago> In Unix you have to cut, awk and do all sorts of parsing to get some field out of `ls`. I'm guessing you've mentioned using `ls` as a simple, first-thing-that-comes-to-mind example, which is cool. I just wanted to point out that if a person is piping ls's output, there are probably other, far better alternatives, such as (per the link below) `find` and globs: https://unix.stackexchange.com/a/247973 https://unix.stackexchange.com/a/247973 That has been my experience, at least.
- ggm 8y agoDanny, boy, the pipes are calling! (I love pipes too btw)
- djhworld 8y agoI love writing little tools and scripts that use pipes, I've accumulated a lot of them over the years and some are daily drivers. It's a great learning tool for learning a new programming language as well as the interface between the boundaries of the program are very simple. For example I wrote this recently https://github.com/djhworld/zipit https://github.com/djhworld/zipit - I'm fully aware you could probably whip up some awk script to do the same, or chain some existing commands together, or someone else has written the same thing, but I've enjoyed the process of writing it and it's something to throw in the tool box - even if it's just for me!
- bonquesha99 8y agoElixir/Unix style pipe operations in Ruby! https://github.com/LendingHome/pipe_operator https://github.com/LendingHome/pipe_operator
- stirner 8y ago> “Perhaps surprisingly, in practice it turns out that the special case is the main use of the program.” This is, in fact, a Useless Use of Cat [1]. POSIX shells have the < operator for directing a single file to stdin: figlet <file [1] http://porkmail.org/era/unix/award.html http://porkmail.org/era/unix/award.html
- kradroy 8y agoI built an entire prototype ML system/pipeline using shell scripts that glued together two python scripts that did some heavy lifting not easily reproduced. I got the whole thing working from training to prediction in about 3 weeks. What I love about Unix shell commands is that you simply can't abstract beyond the input/output paradigm. You aren't going to create classes, types classes, tests, etc. It's not possible or not worth it. I'd like to see more devs use this approach, because it's a really nice way to get a project going in order to poke holes in it or see a general structure. I consider it a sketchpad of sorts.
- noir_lord 8y agoMy backup system at work is mostly bash scripts and some pipes. If you write them cleanly they don’t suck and crucially for me, bash today works basically the same way as it did 10 years ago and likely in 10 years, that static nature is a big damn win. I sometimes wish language vendors would just say ‘this language is complete, all future work will be bug fixes and libraries’ a static target for anything would be nice. Elixir did say that recently except for one last major change which moved it straight up my list of things to look at in future.
- anthk 8y ago- My IRC notification system is a shell script with entr, notify-send and dunst. - My mail setup uses NMH, everything is automated. I can mime-encode a directory and send the resulting mail in a breeze. - GF's photos from IG are being backed up with a python script and crontab. Non IG ones are geotagged too with a script. I just fire up some cli GPS tools if we hike some mountain route, and gpscorrelate runs on the GPX file. - Music is almost everything chiptunes, I felt interesent on any mainstream music since 2003-4. I mirror a site with wget and it's done. If they offered rsync... - Hell, even my podcasts are being fetch via cron(8). - My setup is CWM/cli based, except for mpv, emulators, links+ and vimb for JS needed sites. Noice is my fm, or the pure shell. find(1) and mpg123/xmp generate my music playlist. Street View is maybe the only service I use on vimb... The more you automate, the less tasks you need to do. I am starting to avoid even taskwarrior/timew, because I am almost task free as I don't have to track a trivial <5m script, and spt https://github.com/pickfire/spt https://github.com/pickfire/spt is everything I need. Also, now I can't stand any classical desktop, I find bloat on everything.
- flexer2 8y agoOne of my most used tools with pipes on the Mac is `pbcopy`, which copies whatever you're piping to the clipboard. Copying your public key somewhere? `cat ~/.ssh/id_rsa.pub | pbcopy` Pretty-printing some JSON you copied from a log file? `pbpaste | jq . | pbcopy` It's one of those little tools I use daily and don't think about much, but it's incredibly useful.
- saagarjha 8y agoAnd of course, its cousin pbpaste is also quite useful: pbpaste | clang -x c -; ./a.out
- zshrdlu 8y agoPedantic, but: you don't pipe to /dev/null, you redirect to it.