7 ms·

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Uni

To criticize sh semantics without acknowledging that C was always there when you needed something serious is a bit short sighted. There are two uses of the Unix “api”: [A] Long lived tools for other people to use. [B] Short lived tools one throws together oneself. The fact that most things work most of the time is why the shell works so well for B, and why it is indeed a poor choice for the sort of stable tools designed for others to use, in A. The ubiquity of the C APIs of course solved [A] use cases in the past, when it was unconscionable to operate a system without cc(1). It’s part of why they get first class treatment in the Unix man pages, as old fashioned as that seems nowadays.

jcranmer 6y ago
There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.
- coliveira 6y ago
  The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.
  - viraptor 6y ago
    Please don't ever assume you can avoid spaces and quotes. It's just a time bomb. It's like people saying you don't need to escape SQL values because they come from constants. Yes, they do... today. It's not just quoting either. It's setting the separator value and reverting it correctly. It's making sure you're still correct when you're in a function in an undefined state. It's a lot of overhead in larger projects.
    - CountSessine 6y ago
      Sure, but in a larger project your won’t be coding in shell script, right?
      - viraptor 6y ago
        Larger projects will often include small shell scripts for deployment / maintenance / various tasks.
    - coliveira 6y ago
      What I am saying is nothing new. This has always been the policy of UNIX developers. If you use only file names without space, then you can make it easier to use tools that depend on this. Of course, if you're writing applications in shell then you cannot rely just on convention.
  - thekaleb 6y ago
    Some shells handle this quoting better than others.
  - gorgoiler 6y ago
    Indeed, and the shell lives and breathes spaces. Arrays of arguments are created every time one types a command. Without spaces-as-magic we’d be typing things like this all the time: $ exec(‘ls’, ‘-l’, ‘A B C’) Maybe that’s unrealistic? I mean, if the shell was like that, it probably wouldn’t have exec semantics and would be more like this with direct function calls: $ ls(ls::LONG, ‘A B C’) Maybe we would drop the parentheses though — they can be reasonably implied given the first token is an unspaced identifier: $ ls ls::LONG, ‘A B C’ And really, given that unquoted identifiers don’t have spaces, we don’t really need the commas either. Could also use ‘-‘ instead of ‘ls::’ to indicate that an identifier is to be interpreted locally in the specific context of the function we are calling, rather than as a generic argument. $ ls -LONG ‘A B C’ If arguments didn’t have spaces, you could make the quotes optional too. QED
    - tsimionescu 6y ago
      I don't think this is the problem people usually complain about. The much bigger problem is that spaces make text-only commands compose badly. $ ls -l `find ./ -name *abc*` Is going to work very nicely if file names have certain properties (no spaces, no special chars), and it's going to break badly if they don't. Quoting is also very simple in simple cases, but explodes in complexity when you add variables and other commands into the mix.
      - hnlmorg 6y ago
        That's still just a shell problem since the shell is expanding `find` and placing the unquoted result into the command line. Some non-POSIX shells don't have this issue. For example with murex you could use either of these two syntaxes: ls -l ${find ./ name *abc*} # return the result as a single string ls -l @{find ./ name *abc*} # return the result as an array (like Bourn shell) So in the first example, the command run would look something like: ls -l "foo abc bar" whereas in the second it would behave more like Bourne shell: ls -l foo abc bar As an aside, the example you provided also wouldn't work in Bourne shell because the asterisks would be expanded by the shell rather than `find`. So you'd need to quote that string: ls -l `find ./ -name "*abc*"` This also isn't a problem in murex because you can specify whether to audo-expand globbing or not. With POSIX shells you end up with quotation mark soup: ls -l "`find ./ -name '*abc*'`" (and we're lucky in this example that we don't need to nest any of the same quotation marks. It often gets uglier than this!) Murex also solves this problem by supporting open and close quote marks via S-Expression-style parentheses: ls -l (${find ./ -name (*abc*)}) Which is massively more convenient when you'd normally end up having to escape double quotes in POSIX shells. --- Now I'm not trying to advocate murex as a Bash-killer, my point is that it's specifically the design of POSIX shells that cause these problems and not the way Unix pipelines work.
  - tsimionescu 6y ago
    The shell (bash, anyway) has a weird mix of support and lack of support for spaces/lists/etc. For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syntax. This isn't necessarily a big hurdle, but it indicates that the concepts are hard to grasp and possibly inconsistent.
    - teddyh 6y ago
      That has nothing to do with spaces, but more to do with the rather lackluster support for arrays in most Bourne-style shells.
    - account42 6y ago
      Bash has arrays: var=( a b "c d" ) ls "${var[@]}" POSIX shell can get you what you want with eval: var='a b "c d"' eval "ls $var" There is also one array variable in POSIX shells, the argument list: set -- a b "c d" ls "$@"
- kbenson 6y ago
  That's when it's useful to use Perl as a substitute for a portion of or the entire pipeline.
- gorgoiler 6y ago
  The biggest pain comes from untarring some source code and trying to build it while in ~/My Tools Directory. Spaces in directories that scupper build code that isn’t expecting them is the fatal mixing of worlds. In most other cases I’ve never really had a problem with “this is a place where spaces are ok” (e.g. notes, documents, photos) and “this is a place where they are not ok” — usually in parts of my filesystem where I’m developing code. It’s fine to make simplifying assumptions if it’s your own code. Command history aside, most one liners we type at the shell are literally throwaways. I think I was clear that they aren’t the only category of program one writes and that, traditionally on Unix systems, the counterpart to sh was C.
- de_watcher 6y ago
  Choosing space as a separator is a useful convention that saves you typing in additional code afterwards if you want to run a short command. It's like there is a shortcut that some people use that you want to wall in because it doesn't look pretty to you.
- enriquto 6y ago
  > because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.
  - outadoc 6y ago
    You should take a stroll in the real world sometime, where spaces and Unicode exists :)
    - enriquto 6y ago
      Yeah, any unicode character is OK on a filename, except maybe space and slash.
      - skissane 6y ago
        I don't think C0 or C1 controls should be allowed in filenames. Why allow them? And they potentially pose security issues. With some, admittedly somewhat obscure [1][2], terminal emulators, C0 and C1 controls can even be used to execute arbitrary code. You could say these terminal emulators are insecure, and you may well be right, but the fact is they exist. [1] See for example page 211 of https://static.zumasys.com/zumasys/atfiles/manuals/at7/AccuTerm-7-User-Manual.pdf https://static.zumasys.com/zumasys/atfiles/manuals/at7/AccuT... which documents the ESC STX < command CR escape sequence which tells the AccuTerm terminal emulator to run an arbitrary command [2] Also the APC C1 control can make Kermit run arbitrary commands, if you've executed "SET APC UNCHECKED" (which is not the default setting) – see http://www.kermitproject.org/onlinebooks/usingckermit3e.pdf http://www.kermitproject.org/onlinebooks/usingckermit3e.pdf page numbered 300 (page 310 of PDF)
        nerdponx 6y ago
        I agree. Personally I think the Linux kernel should have a compilation-time option to disallow various special characters in filenames such as ASCII control characters and colons, so users who want to maintain a sane system can give themselves the option to do so.
nerdponx 6y ago
C was always there when you needed something serious There is a world of stuff in between "I need relatively low-level memory management" and "I need a script to just glue some shit together". For that we have Python and Perl and Ruby and Go, or even Rust.
- 1MachineElf 6y ago
  Going with the same theme, C itself was an innovation meant to fill the space between Assembler (in this context, "A") and B[0]. [0] https://en.wikipedia.org/wiki/B_(programming_language) https://en.wikipedia.org/wiki/B_(programming_language)
  - klyrs 6y ago
    That isn't really supported by the article you linked -- it describes C as an extension of B multiple times.
    - jacquesm 6y ago
      To be more precise: A->BCPL->B->C
    - 1MachineElf 6y ago
      I only linked that because most people don't know what B is, and I suppose the difference between "innovation meant to fill the space" and "extention" wasn't so important to me. My source for the claim itself is from 14m30s of Brian Cantrill's talk "Is It Time to Rewrite the Opersting System in Rust" https://youtu.be/HgtRAbE1nBM https://youtu.be/HgtRAbE1nBM
- gorgoiler 6y ago
  Yes! I love 2020! (My point about C was in the historical context of Unix, which is relevant when talking about its design principles.)
viraptor 6y ago
This doesn't always hold in Linux though. Some proc entries are available only via text entries you have to parse whether that's in sh or C. There's simply no public structured interface.
- JdeBP 6y ago
  This is one of the superior aspects of FreeBSD: no parsing of human-readable strings to get back machine-readable information about the process table. It's available directly in machine-readable form via sysctl().
pwdisswordfish2 6y ago
[C] Long lived tools one throws together oneself Stable tools designed for oneself
- Koshkin 6y ago
  The unknown knowns?
tsimionescu 6y ago
The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.
- diegocg 6y ago
  That's not the fault of pipes, that's the fault of shells like bash. There are shells that deal with spaces and quotes perfectly fine.
- necrotic_comp 6y ago
  I've been scripting stuff on the pipeline for over a decade and haven't really run into this much. You can define a field separator in the command line with the environment variable IFS - i.e. 'IFS=$(echo -en "\n\b");' for newlines - which takes care of the basic cases like spaces in filenames/directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script. I haven't found it too much of a burden, even when dealing with exceptionally large files.
- mbreese 6y ago
  Pipelines are mainly for short-lived, one-off quick scripts. But they are also really useful for when you control the input data. For example, if you need a summary report based on the results of a sql query. If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you. If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea. Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data. But if you mix shell scripts with CSV, you’re just asking for trouble.
- nerdponx 6y ago

jcranmer 6y ago

There's a certain irony in responding to criticism of that which you're extolling by saying not to use it. And the only reason I might be pushed down that path is because the task I'm working on happens to involve filenames with spaces in them (without those spaces, the code would work fine!), because spaces are a reasonable thing to put in a filename unless you're on a Unix system.

coliveira 6y ago
The shell works with spaces, you just need to be careful to quote every file name. The advantage of using file names without spaces is that you can avoid the quotes.
- viraptor 6y ago
  Please don't ever assume you can avoid spaces and quotes. It's just a time bomb. It's like people saying you don't need to escape SQL values because they come from constants. Yes, they do... today. It's not just quoting either. It's setting the separator value and reverting it correctly. It's making sure you're still correct when you're in a function in an undefined state. It's a lot of overhead in larger projects.
  - CountSessine 6y ago
    Sure, but in a larger project your won’t be coding in shell script, right?
    - viraptor 6y ago
      Larger projects will often include small shell scripts for deployment / maintenance / various tasks.
  - coliveira 6y ago
    What I am saying is nothing new. This has always been the policy of UNIX developers. If you use only file names without space, then you can make it easier to use tools that depend on this. Of course, if you're writing applications in shell then you cannot rely just on convention.
- thekaleb 6y ago
  Some shells handle this quoting better than others.
- gorgoiler 6y ago
  Indeed, and the shell lives and breathes spaces. Arrays of arguments are created every time one types a command. Without spaces-as-magic we’d be typing things like this all the time: $ exec(‘ls’, ‘-l’, ‘A B C’) Maybe that’s unrealistic? I mean, if the shell was like that, it probably wouldn’t have exec semantics and would be more like this with direct function calls: $ ls(ls::LONG, ‘A B C’) Maybe we would drop the parentheses though — they can be reasonably implied given the first token is an unspaced identifier: $ ls ls::LONG, ‘A B C’ And really, given that unquoted identifiers don’t have spaces, we don’t really need the commas either. Could also use ‘-‘ instead of ‘ls::’ to indicate that an identifier is to be interpreted locally in the specific context of the function we are calling, rather than as a generic argument. $ ls -LONG ‘A B C’ If arguments didn’t have spaces, you could make the quotes optional too. QED
  - tsimionescu 6y ago
    I don't think this is the problem people usually complain about. The much bigger problem is that spaces make text-only commands compose badly. $ ls -l `find ./ -name *abc*` Is going to work very nicely if file names have certain properties (no spaces, no special chars), and it's going to break badly if they don't. Quoting is also very simple in simple cases, but explodes in complexity when you add variables and other commands into the mix.
    - hnlmorg 6y ago
      That's still just a shell problem since the shell is expanding `find` and placing the unquoted result into the command line. Some non-POSIX shells don't have this issue. For example with murex you could use either of these two syntaxes: ls -l ${find ./ name *abc*} # return the result as a single string ls -l @{find ./ name *abc*} # return the result as an array (like Bourn shell) So in the first example, the command run would look something like: ls -l "foo abc bar" whereas in the second it would behave more like Bourne shell: ls -l foo abc bar As an aside, the example you provided also wouldn't work in Bourne shell because the asterisks would be expanded by the shell rather than `find`. So you'd need to quote that string: ls -l `find ./ -name "*abc*"` This also isn't a problem in murex because you can specify whether to audo-expand globbing or not. With POSIX shells you end up with quotation mark soup: ls -l "`find ./ -name '*abc*'`" (and we're lucky in this example that we don't need to nest any of the same quotation marks. It often gets uglier than this!) Murex also solves this problem by supporting open and close quote marks via S-Expression-style parentheses: ls -l (${find ./ -name (*abc*)}) Which is massively more convenient when you'd normally end up having to escape double quotes in POSIX shells. --- Now I'm not trying to advocate murex as a Bash-killer, my point is that it's specifically the design of POSIX shells that cause these problems and not the way Unix pipelines work.
- tsimionescu 6y ago
  The shell (bash, anyway) has a weird mix of support and lack of support for spaces/lists/etc. For example, there is no way to store the `a b "c d"` in a regular variable in a way where you can then call something similar to `ls $VAR` and get the equivalent of `ls a b "c d"`. You can either get the behavior of `ls a b c d` or `ls "a b c d"`, but if you need `ls a b "c d"` you must go for an array variable with new syntax. This isn't necessarily a big hurdle, but it indicates that the concepts are hard to grasp and possibly inconsistent.
  - teddyh 6y ago
    That has nothing to do with spaces, but more to do with the rather lackluster support for arrays in most Bourne-style shells.
  - account42 6y ago
    Bash has arrays: var=( a b "c d" ) ls "${var[@]}" POSIX shell can get you what you want with eval: var='a b "c d"' eval "ls $var" There is also one array variable in POSIX shells, the argument list: set -- a b "c d" ls "$@"
kbenson 6y ago
That's when it's useful to use Perl as a substitute for a portion of or the entire pipeline.
gorgoiler 6y ago
The biggest pain comes from untarring some source code and trying to build it while in ~/My Tools Directory. Spaces in directories that scupper build code that isn’t expecting them is the fatal mixing of worlds. In most other cases I’ve never really had a problem with “this is a place where spaces are ok” (e.g. notes, documents, photos) and “this is a place where they are not ok” — usually in parts of my filesystem where I’m developing code. It’s fine to make simplifying assumptions if it’s your own code. Command history aside, most one liners we type at the shell are literally throwaways. I think I was clear that they aren’t the only category of program one writes and that, traditionally on Unix systems, the counterpart to sh was C.
de_watcher 6y ago
Choosing space as a separator is a useful convention that saves you typing in additional code afterwards if you want to run a short command. It's like there is a shortcut that some people use that you want to wall in because it doesn't look pretty to you.
enriquto 6y ago
> because spaces are a reasonable thing to put in a filename unless you're on a Unix system. Putting spaces on a filename is atrocious and should be disallowed by modern filesystems. It is like if you could put spaces inside variable names in Python. Ridiculous.
- outadoc 6y ago
  You should take a stroll in the real world sometime, where spaces and Unicode exists :)
  - enriquto 6y ago
    Yeah, any unicode character is OK on a filename, except maybe space and slash.
    - skissane 6y ago
      I don't think C0 or C1 controls should be allowed in filenames. Why allow them? And they potentially pose security issues. With some, admittedly somewhat obscure [1][2], terminal emulators, C0 and C1 controls can even be used to execute arbitrary code. You could say these terminal emulators are insecure, and you may well be right, but the fact is they exist. [1] See for example page 211 of https://static.zumasys.com/zumasys/atfiles/manuals/at7/AccuTerm-7-User-Manual.pdf https://static.zumasys.com/zumasys/atfiles/manuals/at7/AccuT... which documents the ESC STX < command CR escape sequence which tells the AccuTerm terminal emulator to run an arbitrary command [2] Also the APC C1 control can make Kermit run arbitrary commands, if you've executed "SET APC UNCHECKED" (which is not the default setting) – see http://www.kermitproject.org/onlinebooks/usingckermit3e.pdf http://www.kermitproject.org/onlinebooks/usingckermit3e.pdf page numbered 300 (page 310 of PDF)
      - nerdponx 6y ago
        I agree. Personally I think the Linux kernel should have a compilation-time option to disallow various special characters in filenames such as ASCII control characters and colons, so users who want to maintain a sane system can give themselves the option to do so.

nerdponx 6y ago

C was always there when you needed something serious There is a world of stuff in between "I need relatively low-level memory management" and "I need a script to just glue some shit together". For that we have Python and Perl and Ruby and Go, or even Rust.

1MachineElf 6y ago
Going with the same theme, C itself was an innovation meant to fill the space between Assembler (in this context, "A") and B[0]. [0] https://en.wikipedia.org/wiki/B_(programming_language) https://en.wikipedia.org/wiki/B_(programming_language)
- klyrs 6y ago
  That isn't really supported by the article you linked -- it describes C as an extension of B multiple times.
  - jacquesm 6y ago
    To be more precise: A->BCPL->B->C
  - 1MachineElf 6y ago
    I only linked that because most people don't know what B is, and I suppose the difference between "innovation meant to fill the space" and "extention" wasn't so important to me. My source for the claim itself is from 14m30s of Brian Cantrill's talk "Is It Time to Rewrite the Opersting System in Rust" https://youtu.be/HgtRAbE1nBM https://youtu.be/HgtRAbE1nBM
gorgoiler 6y ago
Yes! I love 2020! (My point about C was in the historical context of Unix, which is relevant when talking about its design principles.)

viraptor 6y ago

This doesn't always hold in Linux though. Some proc entries are available only via text entries you have to parse whether that's in sh or C. There's simply no public structured interface.

JdeBP 6y ago
This is one of the superior aspects of FreeBSD: no parsing of human-readable strings to get back machine-readable information about the process table. It's available directly in machine-readable form via sysctl().

pwdisswordfish2 6y ago

[C] Long lived tools one throws together oneself Stable tools designed for oneself

Koshkin 6y ago
The unknown knowns?

tsimionescu 6y ago

The problem is that the pipeline model is extremely fragile and breaks in unexpected ways in unexpected places when hit with the real world. The need to handle spaces and quotes can take you from a 20 character pipeline to a 10 line script, or a C program. That is not a good model whichever way you look at it.

diegocg 6y ago
That's not the fault of pipes, that's the fault of shells like bash. There are shells that deal with spaces and quotes perfectly fine.
necrotic_comp 6y ago
I've been scripting stuff on the pipeline for over a decade and haven't really run into this much. You can define a field separator in the command line with the environment variable IFS - i.e. 'IFS=$(echo -en "\n\b");' for newlines - which takes care of the basic cases like spaces in filenames/directory names when doing a for loop, and if I have other highly structured data that is heavily quoted or has some other sort of structure to it, then I either normalize it in some fashion or, as you suggest, write a perl script. I haven't found it too much of a burden, even when dealing with exceptionally large files.
mbreese 6y ago
Pipelines are mainly for short-lived, one-off quick scripts. But they are also really useful for when you control the input data. For example, if you need a summary report based on the results of a sql query. If you control the inputs and you need to support quotes, spaces, non-white space delimiters, etc... in shell script, then that’s on you. If you don’t control the inputs, then shell scripts are generally a poor match. For example, if you need summary reports from a client, but they sometimes provide the table in xlxs or csv format — shell might not be a good idea. Might be controversial, but I think you can tell who works with shell pipes the most by looking at who uses CSV vs tab-delimited text files. Tabs can still be a pain if you have spaces in data. But if you mix shell scripts with CSV, you’re just asking for trouble.
nerdponx 6y ago