5 ms·

For an alternative view, don't forget to read the section on Pipes of The Unix-Haters Handbook: http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~sims

For an alternative view, don't forget to read the section on Pipes of The Unix-Haters Handbook: http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~simsong/www/ugh.pdf (page 198)

bagrow 8y ago
> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)
- deleted 8y ago
  [deleted]
- pytester 8y ago
  The core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the object. You can’t send pointers into another process’s address space. You can’t send file handles or tcp connections or permissions to access particular files or resources.
  - majewsky 8y ago
    > You can’t send pointers into another process’s address space. Thank goodness.
  - jimktrains2 8y ago
    To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.
    - AnIdiotOnTheNet 8y ago
      Yes. The compromise of just using an untyped byte stream in a single linear pipeline was a fair tradeoff in the 70s, but it is nearly 2020 and we can do better.
      - laumars 8y ago
        We have done better. The shell I'm writing is typed and I know I'm not the only person to do this (eg Powershell). The issue here is really more with POSIX compatibility but if you're willing to step away from that then you might find an alternative that better suits your needs. Thankfully switching shells is as painless as switching text editors.
        da_chicken 8y ago
        > Thankfully switching shells is as painless as switching text editors. So, somewhere between, "That wasn't as bad as I feared," and, "Sweet Jesus, what fresh new hell have I found myself in"?
        laumars 8y ago
        haha yes. I was thinking more about launching the shell but you're absolutely right that learning the syntax of a new shell is often non-trivial.
    - __MatrixMan__ 8y ago
      I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a proof of concept: https://unix.stackexchange.com/a/495338/146169 https://unix.stackexchange.com/a/495338/146169 Do you think this is a reasonable way to achieve the magic that some users want in their pipelines? Or are ancient Unix gods going to smite me for tampering with the functional consistency of tools by making their behavior different in different contexts?
  - laumars 8y ago
    I'm not going to argue that UNIX got everything right because I don't believe that to be the case either but I don't agree with those specific points: > This means that you can’t send an object and the code for the class definition necessary to implement the object. To some degree you can and I do just this with my own shell I've written. You just have to ensure that both ends of the pipe understands what is being sent (eg is it JSON, text, binary data, etc)? Even with typed terminals (such as Powershell), you still need both ends of the pipe to understand what to expect to some extent. Having this whole thing happen automatically with a class definition is a little optimistic though. Not least of all because not every tool would be suited for every data format (eg a text processor wouldn't be able to do much with a GIF even if it has a class definition). > You can’t send pointers into another process’s address space. Good job too. That seems a very easy path for exploit. Thankfully these days it's less of an issue though because copying memory is comparatively quick and cheap compared to when that handbook was written. > You can’t send file handles Actually that's exactly how piping works as technically the standard streams are just files. So you could launch a program with STDIN being a different file from the previous processes STDOUT. > or tcp connections You can if you pass it as a UNIX socket (where you define a network connection as a file). > or permissions to access particular files or resources. This is a little ambiguous. For example you can pass strings that are credentials. However you cannot alter the running state of another program via it's pipeline (aside what files it has access to). To be honest I prefer the `sudo` type approach but I don't know how much of that is because it's better and how much of that is because it's what I am used to.
    - zwp 8y ago
      >> You can’t send file handles > Actually that's exactly how piping works Also SCM_RIGHTS, which exists exactly for this purpose (see cmsg(3), unix(7) or https://blog.cloudflare.com/know-your-scm_rights/ https://blog.cloudflare.com/know-your-scm_rights/ for a gentler introduction and application). That's been around since BSD 4.3, which predates the Hater's Handbook 1ed by 4 years or so.
      - laumars 8y ago
        Yeah I had mentioned UNIX domain sockets. However your post does add a lot of good detail on them which I had left off.
  - rumcajz 8y ago
    Look at the alternatives though. Would you really want to use something like Spring in shell scripting?
    - pytester 8y ago
      No. I typically use python as a drop in replacement for shell scripts > ~10 lines of code.
- OliverJones 8y ago
  MacOs is layered on a UNIX-like OS. You can use pipes in your command windows.
  - amdavidson 8y ago
    The Unix Haters Handbook was published in 1994, when System 7 was decidedly not unix-like.
  - DavidWoof 8y ago
    This comment makes me feel really old. MacOs wasn't always layered on unix, and the unix-haters' handbook predates the switch to the unix-based MacOs X.
    - chrisfinazzo 8y ago
      Of course not, but the switch to BSD fixed a bunch of the underpinnings in the OS and was a sane base to work off of. Not to put too fine a point on it, but they found religion. Unlike Classic (and early versions of Windows for that matter), there was more to be gained by ceding some control to the broader community. Microsoft has gotten better (PowerShell - adapting UNIX tools to Windows, and later WSL, where they went all in) Still, for Apple it meant they had to serve two masters for a while - old school Classic enthusiasts and UNIX nerds. Reading the back catalog of John Siracusa's (one of my personal nerd heroes) old macOS reviews gives you some sense of just how weird this transition was.
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
  - JdeBP 8y ago
    You can also drop the "-like". (-: * https://unix.stackexchange.com/questions/1489/ https://unix.stackexchange.com/questions/1489/
- jcelerier 8y ago
  ... has it ? most people using macs never ever open a terminal.
randallsquared 8y ago
Or p160 by internal numbering.
Asooka 8y ago
The section on find after pipes has also not aged well. I can see why GNU and later GNU/Linux replaced most of the old Unices (I mean imagine having a find that doesn't follow symlinks!). If I may, a bit of code golf on the problem of "print all .el files without a matching .elc" find . -name '*.el' | while read el; do [ -f "${el}c" ] || echo $el; done Of course this uses the dreaded pipes and doesn't support the extremely common filenames with a newline in them, so let's do it without them find . -name '*.el' -exec bash -c 'el=$0; elc="${el}c"; [ -f "$elc" ] || echo "$el"' '{}' ';'
- wglb 8y ago
  So the dreaded space-in-filenames is a problem when you pass the '{}' to a script. The following works very nicely for me: * find . -name '*.el' -exec file {}c ';' 2>&1 | grep cannot
  - wglb 8y ago
    I should have said "works very nicely for me, including on file names with spaces"

For an alternative view, don't forget to read the section on Pipes of The Unix-Haters Handbook: http://web.mit.edu/~simsong/www/ugh.pdf http://web.mit.edu/~simsong/www/ugh.pdf (page 198)

bagrow 8y ago

> When was the last time your Unix workstation was as useful as a Macintosh? Some of that discussion has not aged well :)

deleted 8y ago
[deleted]
pytester 8y ago
The core critique - that everything is stringly typed - still holds pretty well though. >The receiving and sending processes must use a stream of bytes. Any object more complex than a byte cannot be sent until the object is first transmuted into a string of bytes that the receiving end knows how to reassemble. This means that you can’t send an object and the code for the class definition necessary to implement the object. You can’t send pointers into another process’s address space. You can’t send file handles or tcp connections or permissions to access particular files or resources.
- majewsky 8y ago
  > You can’t send pointers into another process’s address space. Thank goodness.
- jimktrains2 8y ago
  To be fair, the same critisim could be used for a socket? I think the issue is that some people want pipes to be something magical that connects their software, not a dumb connection between them.
  - AnIdiotOnTheNet 8y ago
    Yes. The compromise of just using an untyped byte stream in a single linear pipeline was a fair tradeoff in the 70s, but it is nearly 2020 and we can do better.
    - laumars 8y ago
      We have done better. The shell I'm writing is typed and I know I'm not the only person to do this (eg Powershell). The issue here is really more with POSIX compatibility but if you're willing to step away from that then you might find an alternative that better suits your needs. Thankfully switching shells is as painless as switching text editors.
      - da_chicken 8y ago
        > Thankfully switching shells is as painless as switching text editors. So, somewhere between, "That wasn't as bad as I feared," and, "Sweet Jesus, what fresh new hell have I found myself in"?
        laumars 8y ago
        haha yes. I was thinking more about launching the shell but you're absolutely right that learning the syntax of a new shell is often non-trivial.
  - __MatrixMan__ 8y ago
    I don't want all my pipes to be magical all the time, but occasionally I do want to write a utility that is "pipeline aware" in some sense. For example, I'd like to pipe mysql to jq and have one utility or the other realize that a conversion to json is needed in the middle for it work. Im working on a library for this kind of intra-pipeline negitiation. It's all drawing-board stuff right now but I coobbled together a proof of concept: https://unix.stackexchange.com/a/495338/146169 https://unix.stackexchange.com/a/495338/146169 Do you think this is a reasonable way to achieve the magic that some users want in their pipelines? Or are ancient Unix gods going to smite me for tampering with the functional consistency of tools by making their behavior different in different contexts?
- laumars 8y ago
  I'm not going to argue that UNIX got everything right because I don't believe that to be the case either but I don't agree with those specific points: > This means that you can’t send an object and the code for the class definition necessary to implement the object. To some degree you can and I do just this with my own shell I've written. You just have to ensure that both ends of the pipe understands what is being sent (eg is it JSON, text, binary data, etc)? Even with typed terminals (such as Powershell), you still need both ends of the pipe to understand what to expect to some extent. Having this whole thing happen automatically with a class definition is a little optimistic though. Not least of all because not every tool would be suited for every data format (eg a text processor wouldn't be able to do much with a GIF even if it has a class definition). > You can’t send pointers into another process’s address space. Good job too. That seems a very easy path for exploit. Thankfully these days it's less of an issue though because copying memory is comparatively quick and cheap compared to when that handbook was written. > You can’t send file handles Actually that's exactly how piping works as technically the standard streams are just files. So you could launch a program with STDIN being a different file from the previous processes STDOUT. > or tcp connections You can if you pass it as a UNIX socket (where you define a network connection as a file). > or permissions to access particular files or resources. This is a little ambiguous. For example you can pass strings that are credentials. However you cannot alter the running state of another program via it's pipeline (aside what files it has access to). To be honest I prefer the `sudo` type approach but I don't know how much of that is because it's better and how much of that is because it's what I am used to.
  - zwp 8y ago
    >> You can’t send file handles > Actually that's exactly how piping works Also SCM_RIGHTS, which exists exactly for this purpose (see cmsg(3), unix(7) or https://blog.cloudflare.com/know-your-scm_rights/ https://blog.cloudflare.com/know-your-scm_rights/ for a gentler introduction and application). That's been around since BSD 4.3, which predates the Hater's Handbook 1ed by 4 years or so.
    - laumars 8y ago
      Yeah I had mentioned UNIX domain sockets. However your post does add a lot of good detail on them which I had left off.
- rumcajz 8y ago
  Look at the alternatives though. Would you really want to use something like Spring in shell scripting?
  - pytester 8y ago
    No. I typically use python as a drop in replacement for shell scripts > ~10 lines of code.
OliverJones 8y ago
MacOs is layered on a UNIX-like OS. You can use pipes in your command windows.
- amdavidson 8y ago
  The Unix Haters Handbook was published in 1994, when System 7 was decidedly not unix-like.
- DavidWoof 8y ago
  This comment makes me feel really old. MacOs wasn't always layered on unix, and the unix-haters' handbook predates the switch to the unix-based MacOs X.
  - chrisfinazzo 8y ago
    Of course not, but the switch to BSD fixed a bunch of the underpinnings in the OS and was a sane base to work off of. Not to put too fine a point on it, but they found religion. Unlike Classic (and early versions of Windows for that matter), there was more to be gained by ceding some control to the broader community. Microsoft has gotten better (PowerShell - adapting UNIX tools to Windows, and later WSL, where they went all in) Still, for Apple it meant they had to serve two masters for a while - old school Classic enthusiasts and UNIX nerds. Reading the back catalog of John Siracusa's (one of my personal nerd heroes) old macOS reviews gives you some sense of just how weird this transition was.
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- JdeBP 8y ago
  You can also drop the "-like". (-: * https://unix.stackexchange.com/questions/1489/ https://unix.stackexchange.com/questions/1489/
jcelerier 8y ago
... has it ? most people using macs never ever open a terminal.

randallsquared 8y ago

Or p160 by internal numbering.

Asooka 8y ago

The section on find after pipes has also not aged well. I can see why GNU and later GNU/Linux replaced most of the old Unices (I mean imagine having a find that doesn't follow symlinks!). If I may, a bit of code golf on the problem of "print all .el files without a matching .elc" find . -name '*.el' | while read el; do [ -f "${el}c" ] || echo $el; done Of course this uses the dreaded pipes and doesn't support the extremely common filenames with a newline in them, so let's do it without them find . -name '*.el' -exec bash -c 'el=$0; elc="${el}c"; [ -f "$elc" ] || echo "$el"' '{}' ';'

wglb 8y ago
So the dreaded space-in-filenames is a problem when you pass the '{}' to a script. The following works very nicely for me: * find . -name '*.el' -exec file {}c ';' 2>&1 | grep cannot
- wglb 8y ago
  I should have said "works very nicely for me, including on file names with spaces"