6 ms·

The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. If you call a function i

The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. If you call a function in a programming language and an erroneous output happens, you get a crash or exception. In a shell, you'll get empty lines or, worse, incorrect lines, that will propagate to the rest of the script. This makes it impractical to write large scripts and debugging them gets more and more complicated. The shell works well for a few lines of script, any more than that and it becomes a frustrating experience.

theamk 2y ago
that's why the rule #1 of robust shell scripts is "set -e", exit on any error. This is not perfect, but helps with most of the errors.
- mananaysiempre 2y ago
  set -euo pipefail, if you're OK with the Bashism.
  - thristian 2y ago
    Since POSIX 2024, `set -o pipefail` is no longer a bashism!
hggigg 2y ago
It's even worse than that. Most non-trivial, and some trivial scripts and one liners rely on naive parsing (regex/cut etc) because that's the only tool in the toolbox. This resulted in some horrific problems over the years. I take a somewhat hard line that scripts and terminals are for executing sequential commands naively only. Call it "glue". If you're writing a program, use a higher level programming language and parse things properly. This problem of course does tend to turn up in higher level languages but at least you can pull a proper parser in off the shelf there if you need to. Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"...
- chasil 2y ago
  You might have enjoyed DCL under VMS. It did not immediately succumb to envy of the Korn shell.
- Bluecobra 2y ago
  > Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"... How do you tackle this? Would you count the numbers of commas in each line then manually fix the lines that contain more fields?
  - hggigg 2y ago
    You parse them and reject CSVs that do not conform. There is absolutely no way to reason about a malformed CSV.
  - harry8 2y ago
    http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html paragraph titled DSV style. (Yeah esr, not a fan, whatever...) Csv sucks no matter what, there is no one csv spec. Then even if you assume the file is "MS Excel style csv" you can't validate it conforms. There's a bunch of things the libraries do that cope with at least some of it that you will not replicate with cut or an awk one liner.
  - ReleaseCandidat 2y ago
    Yes, as in "check the number of parsed fields for each line" and don't forget about empty fields. Throw an error and stop the program if the number of columns isn't consistent. Which doesn't mean that you can't parse the whole file and output all errors at once (which is the preferred way, we don't live in the 90s any more ;), just don't process the wrong result. And with usable error messages, not just "invalid line N".
- aadhavans 2y ago
  What if you had constraints on the CSV files? Suppose you knew that they don't contain spaces, for example. In that case, I don't see the problem with using UNIX tools.
userbinator 2y ago
The && and || operators let you branch on errors.
- anthk 2y ago
  Not like a Lisp REPL, you have to start over with pipes. That's my main issue with Unix.
chasil 2y ago
At the same time, the POSIX shell can be implemented in a tiny binary (dash compiles to 80k on i386). Shells that implement advanced objects and error handling cannot sink this low, and thus the embedded realm is not accessible to them.
- kragen 2y ago
  that's dramatically larger than any pdp-11 executable, including the original bourne shell, and also, for example, xlisp, which was an object-oriented lisp for cp/m advanced objects and error handling do not require tens of kilobytes of machine code. a lot of why the bourne shell is so error-prone is just design errors, many of them corrected in es and rc
  - chasil 2y ago
    David Korn would not agree with you. "A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K. "ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach. I gave up on size minimization with ksh93." https://m.slashdot.org/story/16351 https://m.slashdot.org/story/16351
    - kragen 2y ago
      it sounds like he did agree, even laboring under the heavy burden of bourne shell compatibility
  - ronjakoi 2y ago
    What are es and rc? Can you give some links?
    - chubot 2y ago
      Search for "rc shell" and "es shell" https://en.wikipedia.org/wiki/Rc_(Unix_shell) https://en.wikipedia.org/wiki/Rc_(Unix_shell) https://wryun.github.io/es-shell/ https://wryun.github.io/es-shell/ They are alternative shells, both from the 90's I believe. POSIX was good in some ways, but bad in that it froze a defective shell design It has been acknowledged as defective for >30 years https://www.oilshell.org/blog/2019/01/18.html#slogans-to-explain-the-project https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp... --- es shell is heavily influenced by Lisp. And actually I just wrote a comment that said my project YSH has garbage collection, but the es shell paper has a nice section on garbage collection (which is required for Lisp-y data structures) And I took some influence from it Trivia: one of the authors of es shell, Paul Haahr, went on to be a key engineer in the creation of Google
throw10920 2y ago
> The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. That's pretty bad, but isn't the complete lack of support for structured data an even bigger one? After all, if you can't even represent your data, then throwing errors is kind of moot.
- chubot 2y ago
  Oils/YSH has structured data and JSON! This was finished earlier this year Garbage Collection Makes YSH Different (than POSIX shell, awk, cmake, make, ...) - https://www.oilshell.org/blog/2024/09/gc.html https://www.oilshell.org/blog/2024/09/gc.html You need GC for arbitrary recursive data structures, and traditionally Unix didn't have those languages. Lisp was the first GC language, and pre-dated Unix, and then Java made GC popular, and Java was not integrated with Unix (it wanted to be its own OS) ---- So now you can do # create some JSON ysh-0.23.0$ echo '{"foo":[1,2,3]}' > x.json # read it into the variable x -- you will get a syntax error if it's malformed ysh-0.23.0$ json read (&x) < x.json # pretty print the resulting data structure, = comes from Lua ysh-0.23.0$ = x (Dict) {foo: [1, 2, 3]} # use it in some computation ysh-0.23.0$ var y = x.foo[1] ysh ysh-0.23.0$ = y (Int) 2
  - throw10920 2y ago
    Structured shells are neat and I love them, but the Unix philosophy is explicitly built around plain text - the "structured" part of structured shells isn't Unixy.
    - chubot 2y ago
      It's not either-or -- I'd think of it as LAYERED - JSON denotes a data structure, but it is also text - you can use grep and sed on it, or jq - TSV denotes a data structure [1], but it is also text - you can use grep on it, or xsv or recutils or ... (on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly. But that doesn't mean they're bad or not useful, just not interoperable in a Unix style. The way you use them with Unix is to "project" onto text) etc. That is the layered philosophy of Oils, as shown in this diagram - https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-flat-files https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-fl... IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM what I call "interior vs. exterior" processes and files/JSON/TSV are "exterior", while cmdlets and objects inside a .NET VM are "interior" Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html --- [1] Oils fixes some flaws in the common text formats with "J8 Notation", an optional and compatible upgrade. Both JSON and TSV have some "text-y" quirks, like UTF-16 legacy and inablity to represent tabs So J8 Notation cleans up those rough edges, and makes them more like "real" data structures with clean / composable semantics

theamk 2y ago

that's why the rule #1 of robust shell scripts is "set -e", exit on any error. This is not perfect, but helps with most of the errors.

mananaysiempre 2y ago
set -euo pipefail, if you're OK with the Bashism.
- thristian 2y ago
  Since POSIX 2024, `set -o pipefail` is no longer a bashism!

hggigg 2y ago

It's even worse than that. Most non-trivial, and some trivial scripts and one liners rely on naive parsing (regex/cut etc) because that's the only tool in the toolbox. This resulted in some horrific problems over the years. I take a somewhat hard line that scripts and terminals are for executing sequential commands naively only. Call it "glue". If you're writing a program, use a higher level programming language and parse things properly. This problem of course does tend to turn up in higher level languages but at least you can pull a proper parser in off the shelf there if you need to. Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"...

chasil 2y ago
You might have enjoyed DCL under VMS. It did not immediately succumb to envy of the Korn shell.
Bluecobra 2y ago
> Notably if I see anyone parsing CSVs with cut again I'm going to die inside. Try unpicking a problem where someone put in the name field "Smith, Bob"... How do you tackle this? Would you count the numbers of commas in each line then manually fix the lines that contain more fields?
- hggigg 2y ago
  You parse them and reject CSVs that do not conform. There is absolutely no way to reason about a malformed CSV.
- harry8 2y ago
  http://www.catb.org/~esr/writings/taoup/html/ch05s02.html http://www.catb.org/~esr/writings/taoup/html/ch05s02.html paragraph titled DSV style. (Yeah esr, not a fan, whatever...) Csv sucks no matter what, there is no one csv spec. Then even if you assume the file is "MS Excel style csv" you can't validate it conforms. There's a bunch of things the libraries do that cope with at least some of it that you will not replicate with cut or an awk one liner.
- ReleaseCandidat 2y ago
  Yes, as in "check the number of parsed fields for each line" and don't forget about empty fields. Throw an error and stop the program if the number of columns isn't consistent. Which doesn't mean that you can't parse the whole file and output all errors at once (which is the preferred way, we don't live in the 90s any more ;), just don't process the wrong result. And with usable error messages, not just "invalid line N".
aadhavans 2y ago
What if you had constraints on the CSV files? Suppose you knew that they don't contain spaces, for example. In that case, I don't see the problem with using UNIX tools.

userbinator 2y ago

The && and || operators let you branch on errors.

anthk 2y ago
Not like a Lisp REPL, you have to start over with pipes. That's my main issue with Unix.

chasil 2y ago

At the same time, the POSIX shell can be implemented in a tiny binary (dash compiles to 80k on i386). Shells that implement advanced objects and error handling cannot sink this low, and thus the embedded realm is not accessible to them.

kragen 2y ago
that's dramatically larger than any pdp-11 executable, including the original bourne shell, and also, for example, xlisp, which was an object-oriented lisp for cp/m advanced objects and error handling do not require tens of kilobytes of machine code. a lot of why the bourne shell is so error-prone is just design errors, many of them corrected in es and rc
- chasil 2y ago
  David Korn would not agree with you. "A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K. "ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach. I gave up on size minimization with ksh93." https://m.slashdot.org/story/16351 https://m.slashdot.org/story/16351
  - kragen 2y ago
    it sounds like he did agree, even laboring under the heavy burden of bourne shell compatibility
- ronjakoi 2y ago
  What are es and rc? Can you give some links?
  - chubot 2y ago
    Search for "rc shell" and "es shell" https://en.wikipedia.org/wiki/Rc_(Unix_shell) https://en.wikipedia.org/wiki/Rc_(Unix_shell) https://wryun.github.io/es-shell/ https://wryun.github.io/es-shell/ They are alternative shells, both from the 90's I believe. POSIX was good in some ways, but bad in that it froze a defective shell design It has been acknowledged as defective for >30 years https://www.oilshell.org/blog/2019/01/18.html#slogans-to-explain-the-project https://www.oilshell.org/blog/2019/01/18.html#slogans-to-exp... --- es shell is heavily influenced by Lisp. And actually I just wrote a comment that said my project YSH has garbage collection, but the es shell paper has a nice section on garbage collection (which is required for Lisp-y data structures) And I took some influence from it Trivia: one of the authors of es shell, Paul Haahr, went on to be a key engineer in the creation of Google

throw10920 2y ago

> The biggest disadvantage of the shell is that, by exchanging data using text, you lose opportunities to check for errors in the output. That's pretty bad, but isn't the complete lack of support for structured data an even bigger one? After all, if you can't even represent your data, then throwing errors is kind of moot.

chubot 2y ago
Oils/YSH has structured data and JSON! This was finished earlier this year Garbage Collection Makes YSH Different (than POSIX shell, awk, cmake, make, ...) - https://www.oilshell.org/blog/2024/09/gc.html https://www.oilshell.org/blog/2024/09/gc.html You need GC for arbitrary recursive data structures, and traditionally Unix didn't have those languages. Lisp was the first GC language, and pre-dated Unix, and then Java made GC popular, and Java was not integrated with Unix (it wanted to be its own OS) ---- So now you can do # create some JSON ysh-0.23.0$ echo '{"foo":[1,2,3]}' > x.json # read it into the variable x -- you will get a syntax error if it's malformed ysh-0.23.0$ json read (&x) < x.json # pretty print the resulting data structure, = comes from Lua ysh-0.23.0$ = x (Dict) {foo: [1, 2, 3]} # use it in some computation ysh-0.23.0$ var y = x.foo[1] ysh ysh-0.23.0$ = y (Int) 2
- throw10920 2y ago
  Structured shells are neat and I love them, but the Unix philosophy is explicitly built around plain text - the "structured" part of structured shells isn't Unixy.
  - chubot 2y ago
    It's not either-or -- I'd think of it as LAYERED - JSON denotes a data structure, but it is also text - you can use grep and sed on it, or jq - TSV denotes a data structure [1], but it is also text - you can use grep on it, or xsv or recutils or ... (on the other hand, protobuf or Apache arrow not text, and you can't use grep on them directly. But that doesn't mean they're bad or not useful, just not interoperable in a Unix style. The way you use them with Unix is to "project" onto text) etc. That is the layered philosophy of Oils, as shown in this diagram - https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-flat-files https://www.oilshell.org/blog/2022/02/diagrams.html#bytes-fl... IMO this is significantly different and better than say PowerShell, which is all about objects inside a VM what I call "interior vs. exterior" processes and files/JSON/TSV are "exterior", while cmdlets and objects inside a .NET VM are "interior" Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html --- [1] Oils fixes some flaws in the common text formats with "J8 Notation", an optional and compatible upgrade. Both JSON and TSV have some "text-y" quirks, like UTF-16 legacy and inablity to represent tabs So J8 Notation cleans up those rough edges, and makes them more like "real" data structures with clean / composable semantics