6 ms·

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipeline

by cowmix 6y ago

I use pipelines as much as the next guy but every time I see post praise how awesome they are, I'm reminded of the Unix Hater's Handbook. Their take on pipelines is pretty spot on too. http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~simsong/www/ugh.pdf

ColanR 6y ago
I read the introduction. Sounds like Norman would have liked to hear about Plan 9.
- dialamac 6y ago
  I have a hard time believing he would have been unfamiliar with Plan 9. It wasn’t exactly obscure in the research community at the time. See the USENIX proceedings in the late 80s and 90s. This is mere speculation, but I doubt he would have appreciated Plan 9.
  - ColanR 6y ago
    I did notice that he wrote about problems in Unix that were solved in P9; that's the reason I made the comment.
fenwick67 6y ago
From the intro: > As for me? I switched to the Mac. It's amusing that Apple would go on to switch to a unix-based OS in '01.
- pjmlp 6y ago
  Apple released their first UNIX based OS in 1988. Exercise for the reader to find out its name.
marcosdumay 6y ago
Hum... Those other abstractions on that section are great and everything, but it really sucks to write them interactively. Pipelines also lack some concept similar to exceptions, but it would also suck to handle those interactively.
- cryptonector 6y ago
  set -o pipefail
  - RMPR 6y ago
    Let's be more hardcore set -euo pipefail
    - lsofzz 6y ago
      Why not a littel bit more? ;) set -euf -o pipefail
    - cryptonector 6y ago
      Sadly set -e is busted, though, I do use it.
deleted 6y ago
[deleted]
ken 6y ago
> The Macintosh model, on the other hand, is the exact opposite. The system doesn’t deal with character streams. Data files are extremely high level, usually assuming that they are specific to an application. When was the last time you piped the output of one program to another on a Mac? And yet, at least it's possible to have more than one Macintosh application use the same data. Half the world has migrated to web apps, which are far worse. As a user, it's virtually impossible to connect two web apps at all, or access your data in any way except what the designers decided you should be able to do. Data doesn't get any more "specific to an application" than with web apps.
- junke 6y ago
  Command-line tools where you glue byte streams are, in spirit, very much like web scraping. Sure, tools can be good or bad, but by design you are going to have ad-hoc input/output formats, and for some tools this is interleaved with presentation/visual stuff (colors, alignment, headers, whitespaces, etc.). In a way web apps are then more alike standard unix stuff, where you parse whatever output you get, hoping that it has enough usable structure to do an acceptable job. The most reusable web apps are those that offer an API, with JSON/XML data formats where you can easily automate your work, and connect them together.
  - jandrese 6y ago
    > The most reusable web apps are those that offer an API, with JSON/XML data formats where you can easily automate your work, and connect them together. Sadly these kind are also the most rare.
cuddlybacon 6y ago
I mostly like what they wrote about pipes. I think the example of bloating they talked about in ls at the start of the shell programming section is a good example: if pipelines are so great, why have so many unix utilities felt the need to bloat? I think it a result of there being just a bit too much friction in building a pipeline. A good portion tends to be massaging text formats. The standard unix commands for doing that tend to have infamously bad readability. Fish Shell seems to be making this better by making a string which has a syntax that makes it clear what it is doing: http://fishshell.com/docs/current/cmds/string.html http://fishshell.com/docs/current/cmds/string.html I use fish shell, and I can usually read and often write text manipulations with the string command without needing to consult the docs. Nushell seems to take a different approach: add structure to command output. By doing that, it seems that a bunch of stuff that is super finicky in the more traditional shells ends up being simple and easy commands with one clear job in nushell. I have never tried it, but it does seem to be movement in the correct direction.
- code-faster 6y ago
  It's less that pipelines are friction, they're really not. It's more that people like building features and people don't like saying no to features. The original unix guys had a rare culture that was happy to knock off unnecessary features.
- rnestler 6y ago
  > Nushell seems to take a different approach: add structure to command output. By doing that, it seems that a bunch of stuff that is super finicky in the more traditional shells ends up being simple and easy commands with one clear job in nushell. I have never tried it, but it does seem to be movement in the correct direction. I tried nushell a few times and the commands really compose better due to the structured approach. How would one sort the output of ls by size in bash without letting ls do the sorting? In nushell it is as simple as "ls | sort-by size".
the_af 6y ago
I think of the Unix Hater's Handbook as a kind of loving roast to Unix, that hackers of the time understood to be humorous (you know, how people complain about the tools they use every day, much like people would later complain endlessly about Windows) and which was widely misunderstood later to be a real scathing attack. It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades...
- msla 6y ago
  No, I think it's mostly just as angry and bitter as it sounds, coming from people who backed the wrong horse (ITS, Lisp Machines, the various things Xerox was utterly determined to kill off... ) and got extremely pissy about Unix being the last OS concept standing outside of, like, VMS or IBM mainframe crap, neither of which they'd see as improvements.
  - Analemma_ 6y ago
    Just because Unix was the last man standing, doesn't mean it's good. Bad products win in the marketplace all the time and for all kinds of reasons. In Unix's case I'd argue the damage is particularly severe because hackers have elevated Unix to a religion and insisted that all its flaws are actually virtues.
- Quekid5 6y ago
  > It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades... You're not wrong, but that's only because people wrote extensions for direct access to the graphics hardware... which obviously don't work remotely, and so aren't really in the spirit of X. It's great that that was possible, but OTOH it probably delayed the invention of something like Wayland for a decade+. It's been ages since I've used X for a remote application, but I sometimes wonder how many of them actually really still work to any reasonable degree. I remember some of them working when I last tried it, albeit with abysmal performance compared to Remote desktop, for example.
zamalek 6y ago
I am firmly in the "ugh" camp. I strongly suspect that the fawning that occurs over pipelines is because of sentimentality more than practicality. Extracting information from text using a regular expression is fragile. Writing fragile code brings me absolutely no joy as a programmer - unless I am trying to flex my regex skills. If you really look at how pipelines are typically used is: lines are analogous to objects and [whatever the executable happens to use] delimiters are analogous to fields. Piping bare objects ("plain old shell objects") makes far more sense, involves far less typing and is far less fragile.
- shakna 6y ago
  If you're having to extract your data using a regex, then the data probably isn't well-formed enough for a shell pipeline. It's doable, but a bad idea. Regex should not be the first hammer you reach for, because it's a scalpel. I recently wanted cpu cores + 1. That could be a single regex. But this is more maintainable, and readable: echo '1 + '"$(grep 'cpu cores' /proc/cpuinfo | tail -n1 | awk -F ':' '{print $2}')" | bc There's room for a regex there. Grep could have done it, and then I wouldn't need the others... But I wouldn't be able to come back in twelve months and instantly be able to tell you what it is doing.
  - zamalek 6y ago
    Would anyone be able to approach this, without knowledge of /proc/cpuinfo, and understand what `$2` refers to? > Regex should not be the first hammer you reach for, because it's a scalpel. You used a regex tool: grep.
    - jandrese 6y ago
      It's plaintext, you could just look at it. That's the beauty of text based systems.
  - jandrese 6y ago
    That seems overly complicated to me. Can you not just: grep '^processor' /proc/cpuinfo | wc -l
Silamoth 6y ago
Thanks for sharing! I'll have to give that a read sometime. It looks quite interesting.
julianeon 6y ago
I love UNIX but I think on this point I'm with the haters. Practically speaking, as an individual, I don't have the needs of the DMV. For my personal use I'm combing through pip-squeaky data file downloads and CSV's. So even though using Python or some other language causes a gigantic performance hit, it's just the difference between 0.002s and 0.02 seconds: a unit of time/inefficiency so small I can't ever perceive it. So I might also well use a language to do my processing, because at my level it's easier to understand and practically the same speed.

by cowmix 6y ago

ColanR 6y ago

I read the introduction. Sounds like Norman would have liked to hear about Plan 9.

dialamac 6y ago
I have a hard time believing he would have been unfamiliar with Plan 9. It wasn’t exactly obscure in the research community at the time. See the USENIX proceedings in the late 80s and 90s. This is mere speculation, but I doubt he would have appreciated Plan 9.
- ColanR 6y ago
  I did notice that he wrote about problems in Unix that were solved in P9; that's the reason I made the comment.

fenwick67 6y ago

From the intro: > As for me? I switched to the Mac. It's amusing that Apple would go on to switch to a unix-based OS in '01.

pjmlp 6y ago
Apple released their first UNIX based OS in 1988. Exercise for the reader to find out its name.

marcosdumay 6y ago

Hum... Those other abstractions on that section are great and everything, but it really sucks to write them interactively. Pipelines also lack some concept similar to exceptions, but it would also suck to handle those interactively.

cryptonector 6y ago
set -o pipefail
- RMPR 6y ago
  Let's be more hardcore set -euo pipefail
  - lsofzz 6y ago
    Why not a littel bit more? ;) set -euf -o pipefail
  - cryptonector 6y ago
    Sadly set -e is busted, though, I do use it.

deleted 6y ago

[deleted]

ken 6y ago

> The Macintosh model, on the other hand, is the exact opposite. The system doesn’t deal with character streams. Data files are extremely high level, usually assuming that they are specific to an application. When was the last time you piped the output of one program to another on a Mac? And yet, at least it's possible to have more than one Macintosh application use the same data. Half the world has migrated to web apps, which are far worse. As a user, it's virtually impossible to connect two web apps at all, or access your data in any way except what the designers decided you should be able to do. Data doesn't get any more "specific to an application" than with web apps.

junke 6y ago
Command-line tools where you glue byte streams are, in spirit, very much like web scraping. Sure, tools can be good or bad, but by design you are going to have ad-hoc input/output formats, and for some tools this is interleaved with presentation/visual stuff (colors, alignment, headers, whitespaces, etc.). In a way web apps are then more alike standard unix stuff, where you parse whatever output you get, hoping that it has enough usable structure to do an acceptable job. The most reusable web apps are those that offer an API, with JSON/XML data formats where you can easily automate your work, and connect them together.
- jandrese 6y ago
  > The most reusable web apps are those that offer an API, with JSON/XML data formats where you can easily automate your work, and connect them together. Sadly these kind are also the most rare.

cuddlybacon 6y ago

I mostly like what they wrote about pipes. I think the example of bloating they talked about in ls at the start of the shell programming section is a good example: if pipelines are so great, why have so many unix utilities felt the need to bloat? I think it a result of there being just a bit too much friction in building a pipeline. A good portion tends to be massaging text formats. The standard unix commands for doing that tend to have infamously bad readability. Fish Shell seems to be making this better by making a string which has a syntax that makes it clear what it is doing: http://fishshell.com/docs/current/cmds/string.html http://fishshell.com/docs/current/cmds/string.html I use fish shell, and I can usually read and often write text manipulations with the string command without needing to consult the docs. Nushell seems to take a different approach: add structure to command output. By doing that, it seems that a bunch of stuff that is super finicky in the more traditional shells ends up being simple and easy commands with one clear job in nushell. I have never tried it, but it does seem to be movement in the correct direction.

code-faster 6y ago
It's less that pipelines are friction, they're really not. It's more that people like building features and people don't like saying no to features. The original unix guys had a rare culture that was happy to knock off unnecessary features.
rnestler 6y ago
> Nushell seems to take a different approach: add structure to command output. By doing that, it seems that a bunch of stuff that is super finicky in the more traditional shells ends up being simple and easy commands with one clear job in nushell. I have never tried it, but it does seem to be movement in the correct direction. I tried nushell a few times and the commands really compose better due to the structured approach. How would one sort the output of ls by size in bash without letting ls do the sorting? In nushell it is as simple as "ls | sort-by size".

the_af 6y ago

I think of the Unix Hater's Handbook as a kind of loving roast to Unix, that hackers of the time understood to be humorous (you know, how people complain about the tools they use every day, much like people would later complain endlessly about Windows) and which was widely misunderstood later to be a real scathing attack. It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades...

msla 6y ago
No, I think it's mostly just as angry and bitter as it sounds, coming from people who backed the wrong horse (ITS, Lisp Machines, the various things Xerox was utterly determined to kill off... ) and got extremely pissy about Unix being the last OS concept standing outside of, like, VMS or IBM mainframe crap, neither of which they'd see as improvements.
- Analemma_ 6y ago
  Just because Unix was the last man standing, doesn't mean it's good. Bad products win in the marketplace all the time and for all kinds of reasons. In Unix's case I'd argue the damage is particularly severe because hackers have elevated Unix to a religion and insisted that all its flaws are actually virtues.
Quekid5 6y ago
> It hasn't aged very well, either. "Even today, the X server turns fast computers into dumb terminals" hasn't been true for at least a couple of decades... You're not wrong, but that's only because people wrote extensions for direct access to the graphics hardware... which obviously don't work remotely, and so aren't really in the spirit of X. It's great that that was possible, but OTOH it probably delayed the invention of something like Wayland for a decade+. It's been ages since I've used X for a remote application, but I sometimes wonder how many of them actually really still work to any reasonable degree. I remember some of them working when I last tried it, albeit with abysmal performance compared to Remote desktop, for example.

zamalek 6y ago

I am firmly in the "ugh" camp. I strongly suspect that the fawning that occurs over pipelines is because of sentimentality more than practicality. Extracting information from text using a regular expression is fragile. Writing fragile code brings me absolutely no joy as a programmer - unless I am trying to flex my regex skills. If you really look at how pipelines are typically used is: lines are analogous to objects and [whatever the executable happens to use] delimiters are analogous to fields. Piping bare objects ("plain old shell objects") makes far more sense, involves far less typing and is far less fragile.

shakna 6y ago
If you're having to extract your data using a regex, then the data probably isn't well-formed enough for a shell pipeline. It's doable, but a bad idea. Regex should not be the first hammer you reach for, because it's a scalpel. I recently wanted cpu cores + 1. That could be a single regex. But this is more maintainable, and readable: echo '1 + '"$(grep 'cpu cores' /proc/cpuinfo | tail -n1 | awk -F ':' '{print $2}')" | bc There's room for a regex there. Grep could have done it, and then I wouldn't need the others... But I wouldn't be able to come back in twelve months and instantly be able to tell you what it is doing.
- zamalek 6y ago
  Would anyone be able to approach this, without knowledge of /proc/cpuinfo, and understand what `$2` refers to? > Regex should not be the first hammer you reach for, because it's a scalpel. You used a regex tool: grep.
  - jandrese 6y ago
    It's plaintext, you could just look at it. That's the beauty of text based systems.
- jandrese 6y ago
  That seems overly complicated to me. Can you not just: grep '^processor' /proc/cpuinfo | wc -l

Silamoth 6y ago

Thanks for sharing! I'll have to give that a read sometime. It looks quite interesting.

julianeon 6y ago

I love UNIX but I think on this point I'm with the haters. Practically speaking, as an individual, I don't have the needs of the DMV. For my personal use I'm combing through pip-squeaky data file downloads and CSV's. So even though using Python or some other language causes a gigantic performance hit, it's just the difference between 0.002s and 0.02 seconds: a unit of time/inefficiency so small I can't ever perceive it. So I might also well use a language to do my processing, because at my level it's easier to understand and practically the same speed.