7 ms·

> Automation was glued together in one of these with a series of grep, awk, sed, ls, test, commands glued together. Anything more complicated was written in C a

• Scubabear68 3y ago
  I don’t think you’ve seen the kind of scripts the person you are responding to is talking about. I have, and mentioned one lower down in the comments. Unix philosophy was great but does not scale well in terms of maintainability or efficiency. Invoking processes over and over again loops is godawful slow. And the horror of complicated shell scripts is legendary.
  • imiric 3y ago
    I certainly have, and might have written a few of those myself. But this doesn't make this approach inherently wrong or obsolete. The programmer is wrong for trying to use the tools beyond their capabilities. Where that line is drawn is subjective, as is the concept of maintainability, but if you feel that you're struggling to accomplish something, and that it's becoming a chore to maintain, the path forward is choosing a more capable tool, like a programming language.
    • Scubabear68 3y ago
      I think we are actually mostly in agreement there then. Perl was invented because the gap from shell to more capable languages was (and is) really big. Languages like Python and Ruby didn’t exist yet, and Perl had a really, really strong sweet spot in text processing.
      • ryl00 3y ago
        > Perl had a really, really strong sweet spot in text processing. Still does.
        • cutler 3y ago
          Since Ruby took the best bits of Perl what advantage does Perl retain?
          • thesuperbigfrog 3y ago
            >> what advantage does Perl retain? Ubiquity, speed, and conciseness. Perl is usually installed by default on Linux and Unix systems. Ruby might be there, it depends. Perl is faster than Ruby. Ruby has been one of the slower scripting languages. But Ruby has been working on performance improvements in the past few releases. I have not seen any benchmarks of the current Perl versus the current Ruby, so this may have changed. Perl is more concise than Ruby allowing more functionality for less code.
            • cutler 3y ago
              I don't know about conciseness. Ruby excels here. As for speed it's useful to distinguish between startup and runtime.
              • thesuperbigfrog 3y ago
                >> As for speed it's useful to distinguish between startup and runtime. The JVM would like a word. (It has slow startup, but can be very fast at runtime due to JIT optimization and cacheing.) Scripts should start and run quickly. Ruby has historically been fairly slow which is why Ruby 3 focused heavily on performance. It has been improving a lot, but I have not seen any benchmarks against other programming languages.
  • mulmen 3y ago
    Programming environments including shells and operating systems are just tools. And every tool can been misused. I opened a can of beans with a screwdriver once. Reality is messy. That doesn’t make the tool bad.
  • sigg3 3y ago
    As a self-taught coder, I've experienced many times how highly skilled software engineers groan and sweat when they encounter shell scripts. I don't understand why, but it seems like people with a CS background are never really taught shell scripts and have come to irrationally fear them. It's sort of taboo. This results in weird behavior, such as writing a groovy (Java?) script for Jenkins to execute bazel in order to build a go binary that runs the very same commands in an exec.Command() construct. Or people who download and import pandas to grab the third field in a csv file. During the course of learning, I've naturally written code in bash that should have been written in another language. I replaced if statements with case because they turned out to be more performant. It's a great learning experience and why I got into python and go. IMO we should use the right tool for the job. Sometimes that tool is a combination of unix utilities that you can put in a shell script for easier maintenance. It's just procedural execution of (usually very efficient) binaries, akin to a jenkins script or gitlab pipeline. Just mind the exceptions and use exit codes.
    • Scene_Cast2 3y ago
      I'm the type of person that would grab pandas to parse a CSV. Here's my reasoning * often times, it's not just the third column I want. Sometimes it becomes "third column unless the first column is 'b' then instead grab the fourth column". Having a good data representation makes sure that I'm not mixing logic code with representation parsing code * I don't have to care about CSV parsing edge cases. Escaped comma? Quotes? I don't care, the library will either handle it or throw an explicit error. With custom parsing code, instead of an error, I'll get some mangled result in the middle of the file that I won't even catch / notice until later down the line * when working with CSVs, in my area (ML / scientific compute), Python is often the right context to be in.
    • stanleydrew 3y ago
      > I've experienced many times how highly skilled software engineers groan and sweat when they encounter shell scripts. I don't understand why, but it seems like people with a CS background are never really taught shell scripts and have come to irrationally fear them. It's sort of taboo. It's not a lack of being "taught" shell scripts. It's the fact that shell programming constructs aren't well documented, your "standard library" is basically dependent on whatever binaries happen to be available on the filesystem, error handling is almost non-existent, etc. It's very easy to write a bad shell script that "solves" a problem as long as a bunch of assumptions aren't violated. In my experience, senior software engineers are extremely averse to hidden assumptions and very concerned with reliability of the systems they build.
  • moomin 3y ago
    Yeah, the actual experience of leaky abstractions and non-portable code is forgotten. Perl solved a very real problem in the 90s. Grief, I shudder to think back to the sheer complexity of my bashrc file back then.
  • beebeepka 3y ago
    Yes, exactly. I have seen entire backend systems written in bash. Everything was shell script, sed and awk. The owner didn't want python or perl because he only knew bash and the related tools. Everything was needlessly hard because these tools were not built for that. Easy to talk about philosophy and the "classic Unix approach" if you don't have to build modern applications this way.
  • unpythonic 3y ago
    Horror is apt. You hit it on the head with the slowness of loops when the body comprises a series of program invocations. The horror really seeps in when you realize the original author wasn't stopped by the lack of data structures: they could get around that with some creative variable names.
• stubish 3y ago
  At the time, before everything became Linux, all these tools and the shells used to glue them together were an incoherent mess. Was your glue sh, ksh, csh, tcsh, bash or something uncommon like zsh? Did your grep, awk and sed use the same regexp syntax as your text editor? Single letter command line options, all meaning something different to each tool. Dozens of domain specific languages (shells, awk, sed etc.) meant dozens to learn and keep in your head. And you needed it in your head, because finding the information you needed in the massive single document man file reference was a pain because hypertext links had not been invented yet (well, probably in Emacs, which was another tool like Perl that people used to avoid the command line nightmare).
  • jancsika 3y ago
    Honest question-- why weren't the tools glued together or perhaps replaced entirely with the lisp inside Emacs? What was it missing?
    • ChoHag 3y ago
      [dead]
    • turndown 3y ago
      While there are many possible answers, it eventually boils down to Unix being a C runtime and thus has a C culture. Lisp is from outside of this section of the world, so it simply had less adoption and support inside Unix land. Other languages, like sed, awk, and shell are not C but share its heritage(essentially, they were made by people close to the making of C.)
    • shrubble 3y ago
      Memory and CPU efficiency, since most systems were memory constrained. My first server had 128MB of RAM...
      • shaftoe 3y ago
        There's an old joke that emacs stands for Eight Megs And Constantly Swapping.
• em-bee 3y ago
  you probably shouldn't abuse shell scripts to build a complex system, and beyond a certain level of complexity, a programming language is the better tool but the only free programming languages available at the time were C/C++, various shells, and awk. everything else was expensive or not generally usable for other reasons. all the really useful languages to build complex systems didn't really appear or become freely available until the 90s. and perl was first among those.
  • imiric 3y ago
    I'm not saying that Perl didn't have its time and place. It certainly fulfilled a need at the time for a language more capable than shell scripts, but less cumbersome than C/C++. But the thing is that today the shell landscape is much more mature for solving simple problems, and we have C/C++ alternatives that are saner and more capable than Perl (e.g. Go). So it arguably has lost its place, as shell tools are still in widespread use, while Perl is mostly underused. Raku is interesting, but it goes in a different direction, and its adoption is practically zero.
    • lolive 3y ago
      Python killed the game.
      • cutler 3y ago
        Unfortunately. Ruby should have been Perl's natural successor. Python is the VHS of scripting languages. For a start it doesn't have a decent answer to Perl or Ruby's one-liners. Then there's the crippled lambda implementation. Python is a sad case of worse is better.
        • tharne 3y ago
          > For a start it doesn't have a decent answer to Perl or Ruby's one-liners. This is by design. Readability is core to the design and philosophy of python. One liners are cool and fun to write, but trying to decipher someone else's incredibly dense bash or perl one-liner is absolutely awful.
• NoMoreNicksLeft 3y ago
  > This doesn't sound that horrific to me. I think it's possible that things that seem normal and inoffensive can become horrific simply from scale. You'll climb the stepladder without complaint, but then there's that radio tower in Canada... Others like the idea of the family cow, then they see the 10,000 head feedlot from the highway. Scale is sometimes sufficient by itself to induce horror.
• aidenn0 3y ago
  csh was a decent interactive tool, but not great for scripting. Bourne shell had the right idea but there were so many bugs in various corners of it (I still sometimes end up writing "test "x$foo" = "xbar" even though shells that need that are long gone). If you can depend on a recent bash and use shellcheck, then it's actually quite a pleasant programming environment, with fewer footguns than one might think. (I want a @#$@# "set -e" equivalent that returns non-zero from a function if any statement in the function results in non-zero). There are some things that are more awkward than they should be though (e.g. given a glob, does it match 0, 1, or many files, or the way array expansions work). Also, there's no builtin way to manage libraries (I don't know about Perl, but Python suffers from this as well). This results in me pasting a few dozen lines of shell at the top of any of my significant shell scripts, for quality-of-life functions. Then I have to use "command -v" to check if the various external programs I'm going to use are present. Say what you will about C, but a statically-linked C program can be dropped in anywhere.
  • imiric 3y ago
    Mostly agree. The modern shell scripting environment is much more robust than 30 years ago, with ShellCheck and some sane defaults, as you say. I also find it pleasant, once you get over some of its quirks. As for managing libraries, that's true, but you can certainly import and reuse some common util functions. For example, this is at the top of most of my scripts: set -eEuxo pipefail _scriptdir="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "${_scriptdir}/lib.sh" This loads `lib.sh` from a common directory where my shell scripts live, which has some logging and error handling functions, so it cuts down on repetition just like a programming language would.
    • orra 3y ago
      I haven't seen -E (aka -o errtrace) before, but it looks like a useful addition to the standard-ish -euxo pipefail. Moreover, it's existence being required explains why my error handling, recently, wasn't working as expected.
• deleted 3y ago
  [deleted]
• j16sdiz 3y ago
  > This doesn't sound that horrific to me. It's the classic Unix approach of building small tools that do one thing well, and composing them in novel ways to solve problems. This works really well if your problem can be solved in one or two liners. It go bad very quickly when, say, you have two CSV files and want to join them the sql-way. In sed, you have to use positional variables and think about shell escaping. In perl, you can at least name those variables and use \Q
  • imiric 3y ago
    > This works really well if your problem can be solved in one or two liners. My personal comfort threshold is around the 100-line mark. It's even possible to write maintainable shell scripts up to 500 lines, but it mostly depends on the problem you're trying to solve, and the discipline of the programmer to follow best practices (use sane defaults, ShellCheck, etc.). > It go bad very quickly when, say, you have two CSV files and want to join them the sql-way. In that case we're talking about structured data, and, yeah, Perl or Python would be easier to work with. That said, depending on the complexity of the CSV, you can still go a long way with plain Bash with IFS/read(1) or tr(1) to split CSV columns. This wouldn't be very robust, but there are tools that handle CSV specifically[1], which can be composed in a shell script just fine. So it's always a balancing act of being productive quickly with a shell script, or reaching out for a programming language once the tools aren't a good fit, or maintenance becomes an issue. [1]: https://miller.readthedocs.io/ https://miller.readthedocs.io/
    • hnlmorg 3y ago
      You’re discussing modern tooling in a conversation about early UNIX tooling. Back in the period being discussed, even ‘read’ was less functional. Ksh introduced a lot of the stuff we now take for granted, some of which wasn’t even available until the Ksh93 (long after Perl was released). Bash itself is a younger project than Perl. Albeit not by much.
      • imiric 3y ago
        Fair point. I'm not arguing that Perl wasn't an improvement back then, but that the approach of composing Unix tools is not inherently bad. And as the shell ecosystem evolved since then, and more capable programming languages appeared, Perl has been left by the wayside as a historical relic, rather than the replacement of Unix tools that Wall envisioned. So I don't disagree that it was needed back then, but it's important to mention the modern context it struggles to exist in.
• pjmlp 3y ago
  The "philosophy" that I never saw in commercial UNIXes, starting with my first experience with Xenix in 1993, beyond being endless repeated in FOSS circles, which is ironic given how GNU and BSD applications work, with their endless amount of command line parameters.
• unpythonic 3y ago
  The effortless composition of complex commands out of simple standalone programs is one of the best features of Unix. And yes, I admire and love it as well. That said, imagine a metrics system for a huge networking company that used these methods to cover all automated testing or defect analysis. Those inner loops were made of greps and seds and so forth, and each one is the invocation of a new program. It wasn't uncommon for these runs to take almost a day. Besides performance, the other nightmare was was someone described below: each script was a one-off that didn't leverage the work from others. If the author only new C shell, then you know you're going to be doing gymnastics to catch the stderr of some of those programs (you can't capture it in the same manner that Bourne variants do). Anyway, yes, we all adore the Unix philosophy, but there are limits.