29 ms·

The Beauty of Unix Pipelines

• JadeNB 6y ago
  What does it mean to say that the video shows "Kernighan being a complete chad"?
  • Cerium 6y ago
    Based on some searching, it seems calling someone a "Chad" is evoking the cool and attractive image that the name presents. "Chad is a man who automatically and naturally turns girls on due to his appearance and demeanor" https://www.reddit.com/r/ForeverAlone/comments/6bgbc2/whats_a_chad/ https://www.reddit.com/r/ForeverAlone/comments/6bgbc2/whats_...
  • dodwer 6y ago
    It's an alt-right in-joke related to the incel subculture.
    • Aaronstotle 6y ago
      I wouldn't be so quick to say it's an alt-right joke. Plenty of my friends will describe something/someone as a Chad and it has nothing to do with incel/alt-right cultures. Are those kind of phrases thrown out in those circles? Yes, but that's more of the general meme/internet lingo as opposed to subscribing to an ideology.
      • sleepybrett 6y ago
        'Chad' originated in the incel subculture, be better.
        • goldenrhino 6y ago
          No, it really didn’t. Even if it did, it has since transcended into something more general that many people are familiar with. Ending your statement with “be better” is condescending and seems purposefully antagonistic. How about you lead by example?
      • junke 6y ago
        This is brogrammer speak (e.g. http://new_words.enacademic.com/662/brogrammer http://new_words.enacademic.com/662/brogrammer).
        • JadeNB 6y ago
          Thanks! In fact, your link suggests that it may have originated (in this context) from Rob Spectre calling his alter ego 'Chad': https://www.businessinsider.com/the-ultimate-guide-to-learning-brogramming-the-hard-way-2011-9 https://www.businessinsider.com/the-ultimate-guide-to-learni... https://twitter.com/ChadSpectre/ https://twitter.com/ChadSpectre/
        • junke 6y ago
          (can't believe this was downvoted)
      • fenwick67 6y ago
        "my friends say it all the time and they don't mean it that way" is not a super great justification for language...
        • goldenrhino 6y ago
          Apparently describing hyper masculine traits as “Chad” in a tongue-in-cheek way is on par with racial slurs? I’m assuming that’s what you are insinuating. Get a grip.
  • Aaronstotle 6y ago
    Essentially another way of saying that Kernighan exudes confidence in his demeanor and excellence in the subject matter
  • globular-toast 6y ago
    According to Urban Dictionary: "chad 1. Residue of faecal matter situated between arse cheeks after incomplete wiping and can spread to balls." I have no idea why the author decided to use that term. There is a related term "Chad" (capital C) which invokes the image of a man who is attractive to women. Again, I have no idea why the author decided to use that term.
• cowmix 6y ago
  I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~simsong/www/ugh.pdf
  • ColanR 6y ago
    I read the introduction. Sounds like Norman would have liked to hear about Plan 9.
    • dialamac 6y ago
      I have a hard time believing he would have been unfamiliar with Plan 9. It wasn’t exactly obscure in the research community at the time. See the USENIX proceedings in the late 80s and 90s. This is mere speculation, but I doubt he would have appreciated Plan 9.
      • ColanR 6y ago
        I did notice that he wrote about problems in Unix that were solved in P9; that's the reason I made the comment.
  • fenwick67 6y ago
    From the intro: > As for me? I switched to the Mac. It's amusing that Apple would go on to switch to a unix-based OS in '01.
    • pjmlp 6y ago
      Apple released their first UNIX based OS in 1988. Exercise for the reader to find out its name.
  • marcosdumay 6y ago
    Hum... Those other abstractions on that section are great and everything, but it really sucks to write them interactively. Pipelines also lack some concept similar to exceptions, but it would also suck to handle those interactively.
• geophile 6y ago
  I love pipelines. I don't know the elaborate sublanguages of find, awk, and others, to exploit them adequately. I also love Python, and would rather use Python than those sublanguages. I'm developing a shell based on these ideas: https://github.com/geophile/marcel https://github.com/geophile/marcel.
  • jraph 6y ago
    Your project looks really cool. I am pretty sure I've seen a Python-based interactive shell a few years ago but I can't remember the name. Have you heard of it?
    • geophile 6y ago
      Do you mean IPython? My understanding is that IPython more of a REPL for Python, less of a shell.
    • x1798DE 6y ago
      I imagine you are thinking of xonsh? https://xon.sh/ https://xon.sh/
    • ihumanable 6y ago
      Perhaps you are thinking of xonsh https://xon.sh/ https://xon.sh/ Edit: x1798DE beat me to it :D
      • jraph 6y ago
        Xonsh, that's it! Thanks to you and x1798DE, and to geophile for the attempt :-) I could not find the right keywords on two different search engines to find it. This never happens.
  • khimaros 6y ago
    The lispers/schemers in the audience may be interested in Rash https://docs.racket-lang.org/rash/index.html https://docs.racket-lang.org/rash/index.html which lets you combine an sh-like language with any other Racket syntax.
• ketanmaheshwari 6y ago
  Unix pipelines are cool and I am all for it. In recent times however, I see that sometimes they are taken too far without realizing that each stage in the pipeline is a process and a debugging overhead in case something goes wrong. A case in point is this pipeline that I came across in the wild: TOKEN=$(kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t' | tr -d " ") In this case, perhaps awk would have absorbed 3 to 4 stages.
  • salmo 6y ago
    I think people can go through a few stages of their shell-foo. The first involves a lot of single commands and temporary files. The second uses pipes, but only tacks on commands with no refactoring. The third would recognize that all the grep and cut should just be awk, that you can redirect the cumulative output of a control statement, that subprocesses and coroutines are your friend. We should all aspire to this. The fourth stage is rare. Some people start using weird file descriptors to chuck data around pipes, extensive process substitution, etc. This is the Perl of shell and should be avoided. The enlightened return back to terse readable scripts, but with greater wisdom. Also, I always hear of people hating awk and sed and preferring, say, python. Understanding awk and especially sed will make your python better. There really is a significant niche where the classic tools are a better fit than a full programming language.
    • young_unixer 6y ago
      My most complex use of the shell is defining aliases in .bashrc and have never felt the need to go further. Do you recommend learning all of that for someone like me? If so, what resource do you recommend?
      • every 6y ago
        You can get a lot of mileage out of aliases, especially compound ones. But eventually you may run up against something they don't handle very well. For that a simple shell script might be the next logical step. And fortunately, they aren't that much of a leap...
• nickthemagicman 6y ago
  I have problems understanding what commands I can pipe. Some work some don't.
  • ketanmaheshwari 6y ago
    The commands that read their input from standard input could be used as receiver in a pipeline. For instance: `A | B` is a short form of `A > tmp.txt; B < tmp.txt; rm tmp.txt` There are some commands that need literal arguments which is different than standard input. For instance the echo command. `echo < list.txt` will not work assuming you want to print the items inside the list. `echo itemA itemB itemC` will work. This is where xargs comes into play -- it converts the standard input stream to literal arguments among other things.
    • nickthemagicman 6y ago
      How do you know which ones receive input from standard input or how can you find out efficiently?
      • ketanmaheshwari 6y ago
        The man page of the command helps in most cases. Often the description section has info if the command reads from standard input, a file or both.
  • mehrdadn 6y ago
    --help is a great one. When you want to search the help output you can either feed it into grep or not... depending on whether the program writes to stdout or stderr. You can force writing to stdout via adding 2>&1 at the end of your command. (It's possible that's what you're running into.)
  • cptnapalm 6y ago
    As everything gets tossed into bin, there's no easy way to find out what is and what isn't a tool or filter rather than an application. It makes discovery difficult.
• not2b 6y ago
  I've been using pipes for decades to get my work done, but it was cool to learn about jq as I have much less experience with JSON. It's a very cool program. Thanks.
• sedatk 6y ago
  It's ironic that the article ends with Python code. You could have done everything in Python in the first place and it would have probably been much more readable.
  • unnouinceput 6y ago
    But slower
  • enriquto 6y ago
    > You could have done everything in Python in the first place and it would have probably been much more readable. Python scripts that call a few third party programs are notoriously unreadable, full of subprocess call getouptut decode(utf8) bullshit. Python is alright as a language, but it is a very bad team player: it only really works when everything is written in Python, if you want to use things written in other languages it becomes icky really fast. Python projects that use parts written in other languages inevitably gravitate to being 100% Python. Another way to put it, is that Python is a cancer.
• rzmnzm 6y ago
  Powershell is the ultimate expression of the Unix pipeline imo. Passing objects through the pipeline and being able to access this data without awk/sed incantations is a blessing for me. I think anyone who appreciates shell pipelines and python can grok the advantages of the approach taken by Powershell, in a large way it is directly built upon existing an Unix heritage. I'm not so good at explaining why, but for anyone curious please have a look at the Monad manifesto by Jeffrey Snover https://devblogs.microsoft.com/powershell/monad-manifesto-the-origin-of-windows-powershell/ https://devblogs.microsoft.com/powershell/monad-manifesto-th... You may not agree with the implementation, but the ideas being it, I think, are worth considering.
• gorgoiler 6y ago
  Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in the environment A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary. It just helps. Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.
  • ianai 6y ago
    How’s it uncool? Just not “ mobile”?
    • gorgoiler 6y ago
      Exactly. “Linux, ew!” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. Just a keyboard.
      • carlmr 6y ago
        >No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. They're full of magnets, and with electron microscope you can analyze what's going on in the hardware.
        • yjftsjthsd-h 6y ago
          Yeah, but it's not required for a lot of investigation.
• CalmStorm 6y ago
  Unix pipelines are indeed beautiful, especially when you consider its similarity to Haskell's monadic I/O: http://okmij.org/ftp/Computation/monadic-shell.html http://okmij.org/ftp/Computation/monadic-shell.html Unix pipelines actually helped me make sense of Haskell's monad.
  • AnimalMuppet 6y ago
    Could you be more specific? I don't get it.
    • CalmStorm 6y ago
      In Unix: a; b # Execute command b after a. The result of a is not used by b. In Haskell: a >> b # Run function b after a. The result of a is not used by b. In Unix: a | b # Execute command b after a. The result of a is used by b. In Haskell: a >>= b # Run function b after a. The result of a is used by b.
      • cnity 6y ago
        Nitpick, sort of, but in unix, a | b executes a and b in parallel, not sequentially. This somewhat complicates the analogy with programming. They're more like functions operating on a data stream. Edit: I might be misinterpreting your use of "execute _after_". You may not mean "after the completion of a" but instead "operating on the results of a, asynchronously" in which case, apologies.
    • msla 6y ago
      It helps if you avoid the syntactic sugar of do-notation: main = getArgs >>= processData >>= displayData main is in the IO monad. The >>= function takes a value wrapped in a monad and a function which accepts a value and returns a value wrapped in the same type of monad, and returns the same monad-wrapped value the function did. It can be used as an infix operator because Haskell allows that if you specify precedence, in which case its left side is the first argument (the value-in-a-monad) and its right side is the second argument (the function). The (imaginary) function getArgs takes no arguments and returns a list of command-line arguments wrapped in an IO monad. The first >>= takes that list and feeds it into the function on its right, processData, which accepts it and returns some other value in an IO monad, which the third >>= accepts and feeds into displayData, which accepts it and returns nothing (or, technically, IO (), the unit (empty) value wrapped in an IO monad) which is main's return value. See? Everything is still function application, but the mental model becomes feeding data through a pipeline of functions, each of which operates on the data and passes it along.
• mehrdadn 6y ago
  What kills me about pipelines is when I pipe into xargs and then suddenly can't kill things properly with Ctrl+C. Often I have to jump through hoops to parse arguments into arrays and avoid xargs just for this. (This has to do with stdin being redirected. I don't recall if there's anything particular about xargs here, but that's where it usually comes up.)
• hi41 6y ago
  Thank you! I did not know you get to a "sql like group by" using uniq -c. That's so cool! I think I used to pipe it to awk and count using an array and then display but your method is far better than mine.
  • quicklime 6y ago
    I use "sort | uniq -c" all the time, but I find it annoying that it does it in worse-than-linear time. It gets super slow every time I accidentally pass it a large input. At that point I usually fall back to "awk '{ xs[$0]++ } END { for (x in xs) print xs[x], x }'", but it's quite long and awkward to type, so I don't do it by default every time. At some point I'll add an alias or a script to do this. One day. edit: OK I finally did it; it's literally been on my to-do list for about 10 years now, so thanks :) $ cat ~/bin/count #!/usr/bin/awk -f { xs[$0]++ } END { for (x in xs) print xs[x], x }
• lexpar 6y ago
  Interesting comment in the header of this site. https://github.com/prithugoswami/personal-website/blob/master/layouts/partials/head.html https://github.com/prithugoswami/personal-website/blob/maste...
  • nsajko 6y ago
    It's a meme: https://www.reddit.com/r/copypasta/comments/5we0ny/if_youre_reading_this_youve_been_in_a_coma_for/ https://www.reddit.com/r/copypasta/comments/5we0ny/if_youre_...
    • giantrobot 6y ago
      It's from the naughty strings list. It's meant to be a "meat exploit" instead of an exploit for the computer.
• stormdennis 6y ago
  The video from 1982 is brilliant great explanations from an era before certain knowledge was assumed as generally known
• enriquto 6y ago
  If you eval this simple pipeline file -b `echo $PATH:|sed 's/:/\/* /g'`|cut -d\ -f-2|sort|uniq -c|sort -n it prints a histogram of the types of all the programs on your path (e.g., whether they are shell, python, perl scripts or executable binaries). How can you ever write such a cute thing in e.g., python or, god forbid, java?
  • viraptor 6y ago
    You don't. If you use it once, meh. If you share it with anyone or preserve for the future, why would you want it to be cute? It's just a few lines in python, probably takes just as long to write because you don't have to play with what needs to be escaped and what doesn't. You can actually tell what's the intent of each line and it doesn't fail on paths starting with minuses or including spaces. Outside of one time use or a code golf challenge, it's not cute. https://pastebin.com/FnBCgHUm https://pastebin.com/FnBCgHUm
    • enriquto 6y ago
      This is not about "code golfing", the "sort|uniq -c|sort" combo is in the hall of fame of great code lines. The PATH thing in my script was an unnecessary distractor, consider this: file -b /bin/* /usr/bin/* |cut -d' ' -f-2|sort|uniq -c|sort -n There are no bizarre escapes nor anything. Besides, the "file" program is called only once. The python equivalent that you wrote may be better if you want to store it somewhere, but it takes a lot to type, and it serves a different purpose. The shell line is something that you write once and run it because it falls naturally from your fingers. Moreover, you can run it everywhere, as it works in bash, zsh, pdksh and any old shell. The python version requires an appropriate python version installed (3 not 2).
      • viraptor 6y ago
        > Moreover, you can run it everywhere Issue 1: `zsh: argument list too long: file` Issue 2: `file -b /usr/bin/apt` returns 3 lines, so the entries you count end up being: 2 /usr/bin/apt (for 1 Mach-O universal Yes, it looks easier, but there are traps.
• nojito 6y ago
  Debugging Linux pipelines is not a fun experience. This is one clear area where Powershell with its object model has got it right.
  • jandrese 6y ago
    I don't find it too bad most of the time. If the pipeline isn't working you can cut it off at any juncture and watch standard out to see whats going on. You can even redirect some half processed data to a file so you don't have to re-run the first half of the pipe over and over while you work out what's going wrong in the tail end.
  • pwdisswordfish2 6y ago
    Please provide an example. Thank you.
    • fomine3 6y ago
      PowerShell supports breakpoint. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/set-psbreakpoint?view=powershell-7 https://docs.microsoft.com/en-us/powershell/module/microsoft...
      • pwdisswordfish2 6y ago
        Can you give a working example? This way you can watch a UNIX user "struggle/fail" as they "attempt" to debug a script using set -x, set -e, $?, LINENO, utilties like ktrace, strace, recordio, etc. and your argument will be taken seriously.
        • pjmlp 6y ago
          Some people also insist to keep debuging with printf(), whereas the rest of the world is happily using InteliJ and friends.
          • pwdisswordfish2 6y ago
• adamnemecek 6y ago
  Unix pipes are the a 1970's construct, the same way bell bottom pants are. It's a construct that doesn't take into account the problems and scale of today's computing. Unicode? Hope your pipes process it fine. Video buffers? High perf? Fuggetaboutit. Piping the output of ls to idk what? Nice, I'll put it on the fridge.
  • a4wtaw4tawtawt 6y ago
    Shhh, you're upsetting them!
  • JdeBP 6y ago
    An informed criticism would be based upon the knowledge that Doug McIlroy came up with the idea in the 1960s. (-:
• billfor 6y ago
  I think systemd can do all of that.
• russellbeattie 6y ago
  I think there's an interesting inflection point between piping different utilities together to get something done, and just whipping up a script to do the same thing instead. First I'll use the command line to, say, grab a file from a URL, parse, sort and format it. If I find myself doing the same commands a lot, I'll make a .sh file and pop the commands in there. But then there's that next step, which is where Bash in particular falls down: Branching and loops or any real logic. I've tried it enough times to know it's not worth it. So at this point, I load up a text editor and write a NodeJS script which does the same thing (Used to be Perl, or Python). If I need more functionality than what's in the standard library, I'll make a folder and do an npm init -y and npm install a few packages for what I need. This is not as elegant as pipes, but I have more fine grained control over the data, and the end result is a folder I can zip and send to someone else in case they want to use the same script. There is a way to make a NodeJS script listen to STDIO and act like another Unix utility, but I never do that. Once I'm in a scripting environment, I might as well just put it all in there so it's in one place.
  • mongol 6y ago
    I have never come across a NodeJS script instead of a shell script. Can you suggest an example to look at that shows the advantages of this?
    • falafel 6y ago
      There are plenty of CLI tools built on Node: https://github.com/sindresorhus/awesome-nodejs#command-line-apps https://github.com/sindresorhus/awesome-nodejs#command-line-...
    • russellbeattie 6y ago
      Here's a JS script [1] I wrote a little while ago just for my own use that queries the CDC for the latest virus numbers, then calcs the average, formats the data and prints to the command line. You can pass in a number of days, otherwise it pulls the last 14 days. $ node query-cdc.js 7 It's nothing special, but I wouldn't want to try to do this with command line utilities. (And yes, it was a bit uglier, but I cleaned it up of random crap I had thrown in before posting it.) 1. https://gist.github.com/russellbeattie/f9edf91115b43d6d7ca3cf0b4f9eb060 https://gist.github.com/russellbeattie/f9edf91115b43d6d7ca3c...
• parliament32 6y ago
  Related: Pipelines can be 235x faster than a Hadoop cluster https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
• staycoolboy 6y ago
  The first time I saw pipes in action I was coming from Apple and PC DOS land. It blew my mind. The re-usability of so many /bin tools, and being able to stuff my own into that pattern was amazing. If you like pipes and multimedia, checkout gstreamer, it has taken the custom pipeline example to real-time.
• Silamoth 6y ago
  That was a great article! Pipes can definitely be very powerful. I will say, though, that I often find myself reading pages of documentation in order to actually get anything with Unix and its many commands.
• niko0221 6y ago
  Esta genial Laexplicacion. Se aprende mucho aqui Soy nuevo y este articulo es bastante genial.
• akavel 6y ago
  I think it will be on topic if I let myself take this occasion to once again plug in a short public service announcement of an open-source tool I built, that helps interactively build Unix/Linux pipelines, dubbed "The Ultimate Plumber": https://github.com/akavel/up/ https://github.com/akavel/up/ I've also recently seen it being described in shorter words as a "sticky REPL for shell". Hope you like it, and it makes your life easier!
  • edeion 6y ago
    I'm surprised this doesn't get up-voted. The tool automates very nicely my shell command writing process: pipe one step at a time and check the incremental results by using head to get a sample. Looks cool to me!
• fogetti 6y ago
  I am sorry for playing the devil's advocate. I also think that pipes are extremely useful and a very strong paradigm, and I use them daily in my work. Also it is not an accident that it is fundamental and integral part of powershell too. But is this really HN top page worthy? I have seen this horse beaten to death for decades now. These kind of articles have been around since the very beginning of the internet. Am I missing something newsworthy which makes this article different from the hundreds of thousands of similar articles?
  • hu3 6y ago
    Perhaps it's useful to reinforce pipes' utility to those who are not aware. As for being top most article, if I had to guess it's because people feel related to the tool's power and enjoy sharing anecdotes of it.
• atombender 6y ago
  Pipes are a great idea, but are severely hampered by the many edge cases around escaping, quoting, and, my pet peeve, error handling. By default, in modern shells, this will actually succeed with no error: $ alias fail=exit 1 $ find / | fail | wc -l; echo $? 0 0 You can turn on the "pipefail" option to remedy this: $ set -o pipefail $ find / | fail | wc -l; echo $? 0 1 Most scripts don't, because the option makes everything much stricter, and requires more error handling. Of course, a lot of scripts also forget to enable the similarly strict "errexit" (-e) and "nounset" options (-u), which are also important in modern scripting. There's another error that hardly anyone bothers to handle correctly: x=$(find / | fail | wc -l) This sets x to "" because the command failed. The only way to test if this succeeded is to check $?, or use an if statement around it: if ! x=$(find / | fail | wc -l); then echo "Fail!" >&2 exit 1 fi I don't think I've seen a script ever bother do this. Of course, if you also want the error message from the command. If you want that, you have to start using name pipes or temporary files, with the attendant cleanup. Shell scripting is suddenly much more complicated, and the resulting scripts become much less fun to write. And that's why shell scripts are so brittle.
  • codemac 6y ago
    Just use a better shell. rc handles this wonderfully, $? is actually called $status, and it's an array, depending on the number of pipes.
  • fomine3 6y ago
    set -e makes another pain for command that nonzero isn't mean failed (ex. diff). It changes semantics for whole script.
    • atombender 6y ago
      If by pain you mean you have to handle errors, yes, that's what you have to do. It's no different from checking the return code of functions in C.
      • fomine3 6y ago
• hbarka 6y ago
  abinitio.com was borne from these principles.
• unnouinceput 6y ago
  Cygwin and you can do exactly this on Windows too. For me Cygwin is a bless.
• nsajko 6y ago
  Surprised there is no mention of Doug McIlroy: https://wiki.c2.com/?DougMcIlroy https://wiki.c2.com/?DougMcIlroy
• sigjuice 6y ago
  Minor nitpick: It is Unix "pipes" (not pipelines).
  • lsofzz 6y ago
    I think this reply needs more up votes! ;)
• antipaul 6y ago
  A similar philosophy has made the "tidyverse" a much-loved extension of the statistical language R. Compare the following 2 equivalent snippets. Which one seems more understandable? iris_data %>% names() %>% tolower() %>% gsub(".", "_", ., fixed=TRUE) %>% paste0("(", ., ")") or: paste0("(", gsub(".", "_", tolower(names(iris_data)), fixed=TRUE), ")")
  • chubot 6y ago
    Yup, I'm working on Oil shell and I wrote an article about data frames, comparing tidyverse, Pandas, and raw Python: What Is a Data Frame? (In Python, R, and SQL) http://www.oilshell.org/blog/2018/11/30.html http://www.oilshell.org/blog/2018/11/30.html The idea is essentially to combine tidyverse and shell -- i.e. put structured data in pipes. I use both shell and R for data cleaning. My approach is different than say PowerShell because data still gets serialized; it's just more strictly defined and easily parseable. It's more like JSON than in-memory objects. The left-to-right syntax is nicer and more composable IMO, and many functional languages are growing this feature (Scala, maybe Haskell?). Although I think Unix pipes serve a distinct use case.
• stevefan1999 6y ago
  I think it is terrible, as if everything is a text you can't precisely describe some data structure, e.g. circular list.
• known 6y ago
  The capacity of pipe buffer is important for doing serious producer|consumer work;
• tekert 6y ago
  To my understanding, this is the same pattern where every "object" outputs the same data type for other "objects" to consume. This pattern can have a text or gui representation that is really powerful in its own nature if you think about it, its why automation agents with their events consumption/emition are so powerful, its why the web itself shifts towards this pattern (json as comunication of data, code as object), The thing is, this will always be a higher level of abstraction, i think that a gui of this pattern should exist as a default method in most operating systems, its would solve a lot learning problems like learning all the names and options of objects, i would be the perfect default gui tool, actually, sites like zapier, or tools like huginn already do this pattern, i always wondered why this pattern expands so slowly being so useful.
• 0xsnowcrash 6y ago
  Aka The Ugliness of Powershell. Would be fun to see equivalents to these in Powershell.
• Dilu8 6y ago
  Good article. Thanks for information
• tarkin2 6y ago
  I wish there were gui pipes. Pipes are wonderful but they’re neither interactive nor continuous. Pipes that loop, outputting a declarative gui text format, and listen for events from the gui, would be marvellous. I can’t think how to do that without sockets and a bash loop. And that seems to create the kind of complexity that pipes manage to avoid.
  • deleted 6y ago
    [deleted]
• miclill 6y ago
  Great write up. One thing I would add is how pipes do buffering / apply backpressure. To my understanding this is the "magic" that makes pipes fast and failsafe(r).
• benjaminoakes 6y ago
  I find gron to be much more Unix-y than jq. It "explodes" JSON into single lines for use with grep, sed, etc and can recombine back into JSON as well. https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron
  • majewsky 6y ago
    Since jq is sed for JSON, by the transitive property, you're saying that sed is not Unix-y. ;) Seriously though, I use both, and IMO they serve different purposes. gron is incredibly useful for exploring unknown data formats, especially with any form of something | gron | grep something Once you've figured out how the data format in question works, a jq script is usually more succinct and precise than a chain of gron/{grep,awk,sed,...}/ungron. So in practice, gron for prompts and jq for scripts.
• praveen9920 6y ago
  I recently wrote a golang code for fetching rss feed and displaying gist based on requirement. Looking at this code, I am tempted to reimplement this using pipes but saner mind took over and said "don't fix something that is not broken" I probably would be still do it and get some benchmark numbers to compare both.
• kazinator 6y ago
  > If you append /export to the url you get the list in a .csv format. Text filtering approach narrowly rescued by website feature. Phew, that was close!
• jakubnarebski 6y ago
  The first example in the article is what `git shortlog -n` does; no need for the pipeline.
• gajjuhacker 6y ago
  hiii bro
• theshadowmonkey 6y ago
  Pipes are like one of the best experiences you’ll have whatever you were doing. I was debugging a remote server logging millions of Logs a day and was aggregating a little on the server. Then all it required was wget, jq, sed and awk. And I had a powerful log analyzer than splunk or any other similar solution on a developer Mac. Which you think is awesome when you’re paying a fortune to use Splunk. And for getting some insights quick, Unix pipes are a godsend.