6 ms·

I've written many, many shell scripts, back as a sysadmin, and because I wanted it to not be a cute little hack. I know all the tricks described on the blogpost

I've written many, many shell scripts, back as a sysadmin, and because I wanted it to not be a cute little hack. I know all the tricks described on the blogpost and many more. I have an intimate understanding how the quoting works and how we can work around its limitations. Let me tell you: shell is sort of a local optimum for one-off throwaway stuff, because it makes commonly needed thing accessible with so few keystrokes. But also, shell is a cute little hack. It's literally impossible to write robust programs in it.

cabalamat 7y ago
> It's literally impossible to write robust programs in it. This is why I never write shell scripts more than about 4 lines long or with any control flow statements in it -- I use Python for that. The Python might be a bit longer, but at least I'll understand it when I come back to it 6 months later.
- mjevans 7y ago
  I would _like_ to use Python (2) for that, but it's EOL/effectively EOL, and Python 3 made the _wrong_ call with respect to Unicode support. Source code is Unicode? Great. Forcing _everything_ to have an Encoding when the surrounding world isn't an academically clean environment? Nope, this means Unicode Enforcement and Error Handling in what should be simple programs. I don't mind have __optional__ libraries for handling Encoding conversions, corrections, and errors... but if I'm working with a legacy filesystem, human input, or anything else, the only basic language support I want or need is transparency of bits/bytes. Let a programmer decide if or where things are validated and what to do about handling invalid input at that moment; do not force the issue upon them.
  - kragen 7y ago
    I agree; at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler. Python did eventually acquire a sensible (lossless, errorless) way to convert back and forth between binary and Unicode — UTF-8B, known as surrogateescape, described in PEP 383. Unfortunately, it isn't the default everywhere. PEP 540 makes it default in some cases. This is the reason Oil is written in its own private fork of Python 2 instead of in Python 3.
    - chubot 7y ago
      Yes those kinds of errors are extremely common... In fact 2 days ago I was visiting a well-known company in SF, and someone mentioned the reason for an aborted Python 3 rollout: a database contained utf-16 surrogate pairs mis-encoded in utf-8. To me this is the same problem I had with HTML and DOM mentioned on the other thread (which I will reply to!). The analogy is HTML files : DOM :: string data :: Python strings. I've long had a rant about "models/abstraction vs. reality", which is basically: Reality always wins :) The DOM is an overabstraction of HTML to the point of causing bugs and performance degradation; ditto for Python strings and "real world" textual data. I'd rather solve "honest problems" than self-inflicted problems caused by the wrong abstraction. A lot of software is basically "undoing" problems caused by other software, not addressing something in the application domain. PEP 383 and 540 seem to be clear instances of that. (And note that I said Python is an amazing achievement in the appendix of this post, which is simultaneously true :) )
    - adwn 7y ago
      > at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler. Woah. If you're using a dynamic programming language for mission-critical software and your tests don't properly cover "coercing binary data into a Unicode string", then your problems run way deeper than Python 2 vs 3. This isn't restricted to on-orbit software. Your GSE ("ground support equipment", for those not in the industry) also shouldn't have that kind of quality problems.
      - kragen 7y ago
        The problem, as I said, happened inside an exception handler; if we had written tests for that error case, they might have passed if it had thrown a UnicodeDecodeError instead of an IOError or whatever it actually was throwing. In defense of Satellogic, it never would have gotten off the ground using traditional risk-minimizing aerospace engineering practices. And it's arguable whether the software in question was "mission-critical" — the mission did continue, after all. My efforts to raise the bar on software, process, and documentation quality were not appreciated.
  - chubot 7y ago
    (author here) Yes, that is exactly why I wanted to provide an upgrade path from Python 2 to Oil (reusing a portion of its CPython). But I mentioned in this post that I'm explicitly cutting it out for reasons of engineering effort. But yeah I still think that idea has merit... Somewhere on my blog I made the observation that Python 3 and Perl 6 are both worse than their predecessors for the shell scripting domain. They're better languages in general, but worse for that domain -- which is still important and getting bigger along with the rest of computing, IMO. I think Oil would be useful in any case, but both Python 3 and Perl 6 opened up that gap in the design space even wider. I mean the string is the CORE TYPE of such programs, so it's important that it be logically applicable to the problem. I'm sure many people saw the Mercurial / Python 3 thread on HN last week: https://lobste.rs/s/pnuak4/mercurial_s_journey_reflections_on https://lobste.rs/s/pnuak4/mercurial_s_journey_reflections_o... Basically the same story, although I questioned their decision of not embedding the Python 2 interpreter instead. I think the only reason not to do that is "memes".
    - lizmat 7y ago
      Please note that Perl 6 has been renamed to Raku (https://raku.org https://raku.org using the #rakulang tag on social media). With regards to being better / worse in the shell scripting domain, I would say that currently performance wise, Perl is better for shell scripting. But development of shell script-wise, I think Raku has superior handling of command line parameters built in: https://docs.raku.org/language/create-cli https://docs.raku.org/language/create-cli
  - bonoboTP 7y ago
    Use the "bytes" data type for raw bytes.
    - mjevans 7y ago
      All the standard libraries need to use it too. golang gets this part right. In golang strings are immutable sequences of bytes (which can be concatenated) or have 'slices' taken from within them (I believe addressed to a rune; which IIRC is a complete UTF-8 sequence). Standard libraries typically operate with bytes[] for low level operations (file I/O) but strings and bytes can be coerced between each other with ease, and notably many objects (such as a File https://golang.org/pkg/os/#File.Write https://golang.org/pkg/os/#File.Write ) have methods for interacting with bytes[] and strings. There's a core library specifically for low level Unicode operations and external libraries for more complex tasks.
      - int_19h 7y ago
        Go gets this very wrong, because you can still index strings as arrays - and you get bytes back, which 99% of the time is not what you want. But which also happens to "just work" on common ASCII inputs, so you get code floating around that is broken without its authors' knowledge. (This, coincidentally, was also the problem in Python 2, and why Python 3 breaking the world was necessary, if inconvenient for some.) At this point, the only sensible design for strings is as fully encapsulated, opaque values that are emphatically not arrays of anything. If you want to index them, you should be specifically saying what you want - nth byte, nth codepoint, nth grapheme etc. If you want content as an array of bytes in a particular encoding, then you should ask for that specifically. Internally, yeah, it should use UTF-8, because it makes the most sense - but that's just an implementation detail. So far, the only language I know that got this right is Swift.
        adwn 7y ago
        Rust also gets this right: Every string and every string slice is guaranteed to be a valid UTF-8 sequence. If you slice a string at invalid UTF-8 boundaries, then – depending on the operation – the thread either panics, or None is returned.
chubot 7y ago
It's literally impossible to write robust programs in it. Right, that's the whole point of Oil! You can make those programs robust without rewriting them entirely. Run them with Oil, and then flip a few switches to enable stricter behavior. Here are a few examples: Oil's Stricter Semantics Solve Real Problems http://www.oilshell.org/blog/2019/08/16.html http://www.oilshell.org/blog/2019/08/16.html But there are a dozen more. The key point is that bash's codebase has kind of "topped out" in the sense that it's hard to add anything without breaking something else. On the other hand, the Oil interpreter is reasonable, small, and you can add features to it, and fix things. Anyone who knows about programming languages might be able to read the type definitions in osh/runtime.asdl (and frontend/syntax.asdl) and learn a lot about how shell works.
- ashkankiani 7y ago
  At that point, just use an actual programming language instead of a shell...
  - coldtea 7y ago
    That misses the whole point - that there shouldn't be any difference. Shells already behave like programming languages just bad ones. The reason people use them over a "bona fide" programming language is their command line editing, easy process handling, first class support for job control, access to environment and pipes. One could use a non-shell programming language, but then he'd have a much harder time with all the later - even though he'll have a much better time with abstraction, programming correctness, error handling, first class data structures, and so on. So we're in an either (shell+easy command line use+process handling/job control/pipes etc) or (programming language+better syntax-cohesion-correctness-better error handling-libraries-first class data structures-and on on). There are things in IT that can't be combined because one is a tradeoff to the other. This is not such a case - this is just a historical accident: crappy language features in shells and crappy job control features in programming languages. But there's no reason one can't have all those latter things in a proper programming language. Which is what Oil is.
chalst 7y ago
> It's literally impossible to write robust programs in it. It's hard, verbose, and littered with snafus, but it is also a solved problem. POSIX and C shell are a source of quoting difficulties. The semantics of Plan9's rc and now fish are much more straightforward. I think that shells could be a robust technology if we were less fixated on faithfully capturing familiar idioms.
coldtea 7y ago
>Let me tell you: shell is sort of a local optimum for one-off throwaway stuff, because it makes commonly needed thing accessible with so few keystrokes. But also, shell is a cute little hack. It's literally impossible to write robust programs in it. But there's no essential reason why it has to be this way. It's just that the first UNIX shell was a minimal and ad-hoc design, and the rest just accumulated BS craft in an unguided and even more ad-hoc way, without any vision or cohesion. Then POSIX came, and we got stuck with those (or hardly better versions, like zsh). If by shell we define it as a programming language that has: first class support for job management, first class support for pipes, handy primitives for one-liners and file/text/job processing then there's absolutely no reason why such a language can't be much better than what we have. For example, just off the top of my head: - proper argument parsing (not just the BS hacks e.g. Bash allows) - proper vectors and maps (not the primitive hacks bash allows - "shift" is so idiotic...) - proper syntax (consider the BS [ vs [[ dichotomy in bash, or the millions of edge cases) - proper exception/error handling (no just the primitive "traps" or checking $?) - better expandability (e.g. consider the crappy kludge that autocomplete options are set) - proper functions (not the arg1=$1 BS - shell could e.g. automatically assign to named arguments) - namespacing - packages, libraries. How many times people write and rewrite the same BS shell functions or primitives? "source" doesn't cut it.

cabalamat 7y ago

> It's literally impossible to write robust programs in it. This is why I never write shell scripts more than about 4 lines long or with any control flow statements in it -- I use Python for that. The Python might be a bit longer, but at least I'll understand it when I come back to it 6 months later.

mjevans 7y ago
I would _like_ to use Python (2) for that, but it's EOL/effectively EOL, and Python 3 made the _wrong_ call with respect to Unicode support. Source code is Unicode? Great. Forcing _everything_ to have an Encoding when the surrounding world isn't an academically clean environment? Nope, this means Unicode Enforcement and Error Handling in what should be simple programs. I don't mind have __optional__ libraries for handling Encoding conversions, corrections, and errors... but if I'm working with a legacy filesystem, human input, or anything else, the only basic language support I want or need is transparency of bits/bytes. Let a programmer decide if or where things are validated and what to do about handling invalid input at that moment; do not force the issue upon them.
- kragen 7y ago
  I agree; at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler. Python did eventually acquire a sensible (lossless, errorless) way to convert back and forth between binary and Unicode — UTF-8B, known as surrogateescape, described in PEP 383. Unfortunately, it isn't the default everywhere. PEP 540 makes it default in some cases. This is the reason Oil is written in its own private fork of Python 2 instead of in Python 3.
  - chubot 7y ago
    Yes those kinds of errors are extremely common... In fact 2 days ago I was visiting a well-known company in SF, and someone mentioned the reason for an aborted Python 3 rollout: a database contained utf-16 surrogate pairs mis-encoded in utf-8. To me this is the same problem I had with HTML and DOM mentioned on the other thread (which I will reply to!). The analogy is HTML files : DOM :: string data :: Python strings. I've long had a rant about "models/abstraction vs. reality", which is basically: Reality always wins :) The DOM is an overabstraction of HTML to the point of causing bugs and performance degradation; ditto for Python strings and "real world" textual data. I'd rather solve "honest problems" than self-inflicted problems caused by the wrong abstraction. A lot of software is basically "undoing" problems caused by other software, not addressing something in the application domain. PEP 383 and 540 seem to be clear instances of that. (And note that I said Python is an amazing achievement in the appendix of this post, which is simultaneously true :) )
  - adwn 7y ago
    > at Satellogic we had on-orbit error-handling problems due to Python 3's string encoding, where coercing binary data into a Unicode string caused an exception in our exception handler. Woah. If you're using a dynamic programming language for mission-critical software and your tests don't properly cover "coercing binary data into a Unicode string", then your problems run way deeper than Python 2 vs 3. This isn't restricted to on-orbit software. Your GSE ("ground support equipment", for those not in the industry) also shouldn't have that kind of quality problems.
    - kragen 7y ago
      The problem, as I said, happened inside an exception handler; if we had written tests for that error case, they might have passed if it had thrown a UnicodeDecodeError instead of an IOError or whatever it actually was throwing. In defense of Satellogic, it never would have gotten off the ground using traditional risk-minimizing aerospace engineering practices. And it's arguable whether the software in question was "mission-critical" — the mission did continue, after all. My efforts to raise the bar on software, process, and documentation quality were not appreciated.
- chubot 7y ago
  (author here) Yes, that is exactly why I wanted to provide an upgrade path from Python 2 to Oil (reusing a portion of its CPython). But I mentioned in this post that I'm explicitly cutting it out for reasons of engineering effort. But yeah I still think that idea has merit... Somewhere on my blog I made the observation that Python 3 and Perl 6 are both worse than their predecessors for the shell scripting domain. They're better languages in general, but worse for that domain -- which is still important and getting bigger along with the rest of computing, IMO. I think Oil would be useful in any case, but both Python 3 and Perl 6 opened up that gap in the design space even wider. I mean the string is the CORE TYPE of such programs, so it's important that it be logically applicable to the problem. I'm sure many people saw the Mercurial / Python 3 thread on HN last week: https://lobste.rs/s/pnuak4/mercurial_s_journey_reflections_on https://lobste.rs/s/pnuak4/mercurial_s_journey_reflections_o... Basically the same story, although I questioned their decision of not embedding the Python 2 interpreter instead. I think the only reason not to do that is "memes".
  - lizmat 7y ago
    Please note that Perl 6 has been renamed to Raku (https://raku.org https://raku.org using the #rakulang tag on social media). With regards to being better / worse in the shell scripting domain, I would say that currently performance wise, Perl is better for shell scripting. But development of shell script-wise, I think Raku has superior handling of command line parameters built in: https://docs.raku.org/language/create-cli https://docs.raku.org/language/create-cli
- bonoboTP 7y ago
  Use the "bytes" data type for raw bytes.
  - mjevans 7y ago
    All the standard libraries need to use it too. golang gets this part right. In golang strings are immutable sequences of bytes (which can be concatenated) or have 'slices' taken from within them (I believe addressed to a rune; which IIRC is a complete UTF-8 sequence). Standard libraries typically operate with bytes[] for low level operations (file I/O) but strings and bytes can be coerced between each other with ease, and notably many objects (such as a File https://golang.org/pkg/os/#File.Write https://golang.org/pkg/os/#File.Write ) have methods for interacting with bytes[] and strings. There's a core library specifically for low level Unicode operations and external libraries for more complex tasks.
    - int_19h 7y ago
      Go gets this very wrong, because you can still index strings as arrays - and you get bytes back, which 99% of the time is not what you want. But which also happens to "just work" on common ASCII inputs, so you get code floating around that is broken without its authors' knowledge. (This, coincidentally, was also the problem in Python 2, and why Python 3 breaking the world was necessary, if inconvenient for some.) At this point, the only sensible design for strings is as fully encapsulated, opaque values that are emphatically not arrays of anything. If you want to index them, you should be specifically saying what you want - nth byte, nth codepoint, nth grapheme etc. If you want content as an array of bytes in a particular encoding, then you should ask for that specifically. Internally, yeah, it should use UTF-8, because it makes the most sense - but that's just an implementation detail. So far, the only language I know that got this right is Swift.
      - adwn 7y ago
        Rust also gets this right: Every string and every string slice is guaranteed to be a valid UTF-8 sequence. If you slice a string at invalid UTF-8 boundaries, then – depending on the operation – the thread either panics, or None is returned.

chubot 7y ago

It's literally impossible to write robust programs in it. Right, that's the whole point of Oil! You can make those programs robust without rewriting them entirely. Run them with Oil, and then flip a few switches to enable stricter behavior. Here are a few examples: Oil's Stricter Semantics Solve Real Problems http://www.oilshell.org/blog/2019/08/16.html http://www.oilshell.org/blog/2019/08/16.html But there are a dozen more. The key point is that bash's codebase has kind of "topped out" in the sense that it's hard to add anything without breaking something else. On the other hand, the Oil interpreter is reasonable, small, and you can add features to it, and fix things. Anyone who knows about programming languages might be able to read the type definitions in osh/runtime.asdl (and frontend/syntax.asdl) and learn a lot about how shell works.

ashkankiani 7y ago
At that point, just use an actual programming language instead of a shell...
- coldtea 7y ago
  That misses the whole point - that there shouldn't be any difference. Shells already behave like programming languages just bad ones. The reason people use them over a "bona fide" programming language is their command line editing, easy process handling, first class support for job control, access to environment and pipes. One could use a non-shell programming language, but then he'd have a much harder time with all the later - even though he'll have a much better time with abstraction, programming correctness, error handling, first class data structures, and so on. So we're in an either (shell+easy command line use+process handling/job control/pipes etc) or (programming language+better syntax-cohesion-correctness-better error handling-libraries-first class data structures-and on on). There are things in IT that can't be combined because one is a tradeoff to the other. This is not such a case - this is just a historical accident: crappy language features in shells and crappy job control features in programming languages. But there's no reason one can't have all those latter things in a proper programming language. Which is what Oil is.

chalst 7y ago

> It's literally impossible to write robust programs in it. It's hard, verbose, and littered with snafus, but it is also a solved problem. POSIX and C shell are a source of quoting difficulties. The semantics of Plan9's rc and now fish are much more straightforward. I think that shells could be a robust technology if we were less fixated on faithfully capturing familiar idioms.

coldtea 7y ago

>Let me tell you: shell is sort of a local optimum for one-off throwaway stuff, because it makes commonly needed thing accessible with so few keystrokes. But also, shell is a cute little hack. It's literally impossible to write robust programs in it. But there's no essential reason why it has to be this way. It's just that the first UNIX shell was a minimal and ad-hoc design, and the rest just accumulated BS craft in an unguided and even more ad-hoc way, without any vision or cohesion. Then POSIX came, and we got stuck with those (or hardly better versions, like zsh). If by shell we define it as a programming language that has: first class support for job management, first class support for pipes, handy primitives for one-liners and file/text/job processing then there's absolutely no reason why such a language can't be much better than what we have. For example, just off the top of my head: - proper argument parsing (not just the BS hacks e.g. Bash allows) - proper vectors and maps (not the primitive hacks bash allows - "shift" is so idiotic...) - proper syntax (consider the BS [ vs [[ dichotomy in bash, or the millions of edge cases) - proper exception/error handling (no just the primitive "traps" or checking $?) - better expandability (e.g. consider the crappy kludge that autocomplete options are set) - proper functions (not the arg1=$1 BS - shell could e.g. automatically assign to named arguments) - namespacing - packages, libraries. How many times people write and rewrite the same BS shell functions or primitives? "source" doesn't cut it.