8 ms·

Most of the justifications for using collections of command-line Unix tools are no longer valid today. Instead you should be using a proper programming language

Most of the justifications for using collections of command-line Unix tools are no longer valid today. Instead you should be using a proper programming language. Note that people who still do use complex solutions built from cat, head, cut, etc, and who know what they're doing, will typically either write a shell script (which won't be structured particularly differently from the equivalent Python or whatever) or will rely heavily on awk (itself a full-featured programming language, no easier to learn than any other scripting language), or both. One-liners which pipe text between four or five different commands are the equivalent of hand-soldered boards or bitwise arithmetic. Interesting to learn about for historical reasons but of no practical utility. The use of things like xargs and jq in this solution, difficult to invoke Unix utilities for doing things that are trivial in any reasonable language, makes this even more clear.

3np 5y ago
I guess it depends on what you do. For a subset of tasks, shell scripting is a lot faster to implement than the equivalent python/js/go/ruby/rust.
Kinrany 5y ago
No "proper programming language" is capable of ergonomically piping between programs. Shell is indeed very old and it's time for a replacement, but it's not there yet. Oilshell might get there eventually or at least spark interest in this area.
- ilyash 5y ago
  > No "proper programming language" is capable of ergonomically piping between programs. Solved. https://github.com/ngs-lang/ngs https://github.com/ngs-lang/ngs I'm the author. Frustrated with exactly this situation I created Next Generation Shell. It's a "proper programming language" on one hand but domain-specific for "DevOps"y scripting on another. So sane syntax, data structures, error handling, multiple dispatch on one hand but also syntax for running external programs, pipes and redirects. You are welcome!
- ccalloway 5y ago
  > No "proper programming language" is capable of ergonomically piping between programs. You cherry-picked the one thing that shell script is somewhat better at than other languages (not even really needed for this task). Meanwhile the article uses shell for both making HTTP requests and mangling JSON data, both of which are easy in all modern languages, and extremely painful in shell.
  - actually_a_dog 5y ago
    It's not cherry picking to mention the thing that makes a fundamental component of the UNIX philosophy work. > (ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input. https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s...
    - ccalloway 5y ago
      OK, the Unix philosophy holds that this is one of the most important things, and therefore it's easy to do in shell. But what if it's not? It's perfectly easy to write code to do the same thing in a programming language without invoking any other executables, let alone piping the output of one into the input of another. So the claim that this functionality is what makes Unix and shell scripting effective is vacuous. They are good at doing something which would not be necessary if one was not using them.
      - Kinrany 5y ago
        Being able to compose programs is not a necessity, but it saves effort. A single pipe replaces M*N different programs with M+N different programs. This is how Unix won.
      - deleted 5y ago
        [deleted]
- unixbane 5y ago
  *sh is only ergonomic if you're using it insecurely. wait, even then it's not ergonomic. you're just insane. the day i figured out about python repl over 10 years ago, i realized *sh is utterly obsolete (and python isn't even good, a repl like that just gives you some perspective of how things could be).
smitty1e 5y ago
> command-line Unix tools are no longer valid today Strongly disagree. The understanding of the OS, the data, and how to checkmate the problem with minimal effort is timeless. Programming languages are relatively ephemeral compared to POSIX utilities. Invest in knowledge of the enduring.
akho 5y ago
> Instead you should be using a proper programming language. Why use many word when few word do trick? Shell scripts are very compact for what they do, and generally as clear as a similarly quick-and-dirty solution in another language. Ease of learning and use comes from having used the programs you are running in your script. E. g. my small script piping stuff from nmcli to fzf so I can choose a wifi network would be much more difficult to write in Python: I’d need to find a Python library for interacting with NetworkManager, and a library for interactively fuzzy searching in a list, read the docs, spend a while setting up a venv to run it, … I don’t have time for any of that. xargs, in particular, is not difficult to invoke once you’ve done it once or twice, and does a lot. Apart from just being a loop, it also parallelizes execution, and can ask for confirmation from the user for each invocation. Implementing either of these features will take you more than 2-4 chars you need to use it with xargs.
spekcular 5y ago
I don't understand why you're being so harshly downvoted. This seems ... plausibly correct to me? Some questions: 1) Where do you draw the line, and move away from shell to a "real" language. If I just want want to view a directory, surely I use ls, right? What about if I want to remove all leading whitespace from a text file? This can be done with a short but fairly opaque awk one-liner. Probably takes way more time to write and run the Python equivalent, but I don't know Python so well, so maybe it's also a one-liner. 2) What's the optimal "real language" for replacing shell? Python? Perl? Raku? 2a) xargs automatically parallelizes programs. How can this be done efficiently (meaning I don't have to write much additional code) in your proposed "real language"?
- dagw 5y ago
  xargs automatically parallelizes programs. How can this be done efficiently (meaning I don't have to write much additional code) in your proposed "real language"? By using xargs :) Or more likely gnu parallel. But seriously, in so many cases using GNU parallel to parallelize a process is the quickest and easiest way to approach the problem, and I use it all the time. If I need to process 10k+ images in a folder, rather than try to parallelize the process in my python/C script I'll write the fastest possible single threaded script that takes its input args as command line arguments and then uses gnu parallel to distribute the workload. The added advantage is that I can distribute this work on a cluster of machines with only a few changes to GNU Parallel's command line arguments.
- actually_a_dog 5y ago
  We had a rule at my old workplace that any shell script that's over a page of code was to be replaced with a Python script. We chose Python because guaranteed there would always be an up to date version of Python on any cloud machine we used, but you could certainly make a case for some other language. In a vacuum, my choice might be Scheme.
aulin 5y ago
wait, what's wrong with hand soldering and bitwise arithmetic? hand soldering is still of very much practical utility for building prototypes, small production batches, electronics repair... Bitwise arithmetic... seriously? tell that to an embedded developer or anyone who works with low-level stuff.
- ccalloway 5y ago
  Nothing is wrong with them if 1) you have a specifically suited, niche problem and you understand the complexities and tradeoffs of the tool OR 2) it's not a critical requirement to solve the problem using modern, efficient tools AND you want to use something else for fun or learning or whatever. If this isn't the case, stay away.
  - aulin 5y ago
    Either you're just trolling at this point OR you haven't the slightest idea of what you're talking about.
2143 5y ago
> Most of the justifications for using collections of command-line Unix tools are no longer valid today. Why is it not valid today? > One-liners which pipe text between four or five different commands are the equivalent of hand-soldered boards or bitwise arithmetic. Why is that bad? Don't deploy a supercomputer to do what a hand soldered board can. Keep it simple. I don't see how you equate piping commands to bitwise arithmetic, but bitwise arithmetic is easy anyway. > but of no practical utility Says you. Just because you don't find something useful doesn't mean nobody else finds it useful. Turning one liner piped commands to a program in what you might consider a "proper programming language" usually ends up turning a declarative program into something prodecural. Not that that's necessarily a bad thing. Just saying. Use the right tool for the job. In some cases (not all) the shell is indeed the right tool. I get a feeling you don't understand the Unix philosophy. Read The Art of Unix Programming by Eric S. Raymond. Go learn bitwise arithmetic. The uneducated play with pictures. Educated people read and write :) Have a great day (or night, depending on your timezone — night here).
- Chris2048 5y ago
  > Turning one liner piped commands to a program in what you might consider a "proper programming language" usually ends up turning a declarative program into something prodecural. I don't understand - isn't bash shell procedural?
- Chris2048 5y ago
  > I get a feeling you don't understand the Unix philosophy. Read The Art of Unix Programming by Eric S. Raymond. Go learn bitwise arithmetic. These kind of comments aren't useful, and come off smug to me. Why doesn't he understand something, what it the point in reading/learning something you think relevant? Why would they invest in your suggestions if you don't provide any reasons what is missing? Also: > The uneducated play with pictures. Educated people read and write :) This sounds insulting to me (name-calling), and passive-aggressively so when combined with "Have a great day".
smcameron 5y ago
This is akin to saying if you want to hang a picture in your house, you shouldn't just grab a hammer and a nail, instead, you should get a nice piece of wood, and a nice hunk of metal, go to the workshop, fire up the forge, the mill and various wood and metal working machines, and forge a special picture-hanging-hammerer-thing.
- ccalloway 5y ago
  Umm, no, chaining together 5 or more somewhat arcane single-purpose tools is the unrealistic solution here. If the article was about using ls to just list files, and I had said "actually you should use Python's os.listdir() and filter the results by whatever" you would be right. For most simple problems it's correct to use a simple tool. For the overwhelming majority of complex problems you should use a well-understood, well-designed, common general-purpose tool.
  - pessimizer 5y ago
    What's supposed to be the difference between chaining simple UNIX commands and chaining simple python functions again?
    - ccalloway 5y ago
      For example, the latter is testable.
    - Chris2048 5y ago
      How many executables do the two methods require? Do all those executables/programs share a common type system?
herbst 5y ago
This is only partially true. Working on your own machine this may be fully the case, but debugging and maintaining random servers is a complete different beast. Shell is portable in a way nothing else is, same reason people use Excel instead of code or PHP instead of literally anything.
- ccalloway 5y ago
  Firstly the model of sshing into your server and trying to run commands on it directly is more or less obsolete. Secondly the presence of all these utilities on a machine is far from guaranteed - expecting Python to be present is no more or less likely.
  - cranekam 5y ago
    > Firstly the model of sshing into your server and trying to run commands on it directly is more or less obsolete. Why do you get to declare what is acceptable (using UNIX tools) or what's now obsolete (SSHing into a host)? It's rather presumptuous to believe you can speak for everyone. Perhaps you only ever deploy code using k8s and never need to use a command line but that doesn't mean we all do. There are many reasons one would use these tools and approaches (fun, small scale, investigating problems, anything).
    - ccalloway 5y ago
      You might think 'fun, small scale, investigating problems' are good justifications for your working practices, but your boss's boss's boss does not. They would prefer you to have less fun. Or perhaps to have fun on your own time after being laid off.
      - herbst 5y ago
        My boss (me) doesn't mind me dabbling around in my servers via SSH. Not everyone of us works for evil cooperate.
        5y ago
Lamad123 5y ago
I heard awk is extremely fast, much faster than Python.
- hansel_der 5y ago
  i heard python is among of the slowest
- ccalloway 5y ago
  Yes, but execution speed (of things like creating a lookup table from the second and third fields in each line) is rarely the operative constraint.
  - jes 5y ago
    What is most often the operative constraint?
    - ccalloway 5y ago
      Time spent maintaining your script by your future co-workers.
- revscat 5y ago
  It is, but it's much closer to perl when it comes to maintainability.
high_5 5y ago
> Interesting to learn about for historical reasons but of no practical utility. The practical utility has just been demonstrated in this particular article? Historical? I think that Unix shell are like crocodiles - outliving the dinosaurs and lurking in the murky water the unsuspecting sysadmin to come close enough to fix the script that ain't broken.
- oblio 5y ago
  Your analogy is quite interesting, in the sense that crocodiles are around, but they're not the dominant species. That would be an interesting continuation to your analogy, actually.
  - high_5 5y ago
    Sooner or later one has to take a sip from the unixy pond :-)
    - oblio 5y ago
      For now. At some point we're going to build proper water management and sanitation systems. What those are in the software world, I, as a swamp dweller, cannot say.
- ccalloway 5y ago
  > The practical utility has just been demonstrated in this particular article? How? The author is doing something which could be done much more easily and elegantly in a programming language.
flohofwoe 5y ago
Fundamentally it's the same thing. The UNIX tools are the "batteries included" standard library of the integrated shell programming environment. And if you need to extend that library you quickly whip up a new minimal command line tool in C (or any other language which allows to write small and quick and dirty command line tools, like Python). The only downside of shell scripting is that it isn't trivially portable to Windows (or even macOS because of the differences between GNU and BSD tools), so it often makes sense to create big Python scripts that do more than "one thing right". If the whole world would run on UNIX, shell scripting would make much more sense.
- actually_a_dog 5y ago
  > The only downside of shell scripting is.... The only downside? Let's add that no major *NIX shell that I'm aware of has any good way to modularize code while enforcing encapsulation of state. At my previous job, we had a rule that any shell script longer than about a page of code had to be replaced with a Python script ASAP. That was a good rule, IMO, because once you've exceeded a certain size, a shell script starts getting brittle and hard to work with. I don't know if 1 page of code is the threshold size or not, but it seems like as good a cutoff as any.
hawski 5y ago
Shell is a very high level language that is undoubtedly bizarre at times. Its routines can be written in any programming language you want without much of a boilerplate or any special bindings, because supporting for command line arguments, environment variables, reading and writing files is a bare minimum that most languages support. It is a tradeoff - it has its immense advantages and a couple of often hard to navigate disadvantages. But the ability to easily compose whatever you want is something you can't ignore. I think the perfect very high level language is closer to shell than to python for example. The power of Tcl/Tk (still it has some big weaknesses) or Rebol/Red is something that I admire. The following statement is probably controversial: shell is more akin to Lisp for human beings. I dabbled at Scheme, but it is harder for me to grasp than Shell, but in the end they are more similar than not. I hold a candle for Oil shell for example.
pkrumins 5y ago
Sir, you are absolutely and totally wrong. No sysadmin has time or interest to write programs or use "proper programming languages" to get the job done. Sysadmins know their tools and are fast and efficient. Zero sysadmins will ever write "proper programs" when they can pipe find, sed, and awk.
- ccalloway 5y ago
  Yes, but the number of sysadmins in the world is converging rapidly to zero. Mainly because the invisible overhead of having your boxes managed by people who use sed and awk is much higher than the alternatives (containerisation, cloud vendors, cattle not pets, infrastructure as code). I think you actually confirmed my point when you said 'no sysadmin has time or interest'. I didn't say that no one uses shell anymore. I said that there was no practical justification for doing so. Lots of people still use it and don't have any interest in finding a better solution. These people are choosing, for their own reasons, to double down on an obsolete skillset. Good for them, I suppose, but I don't think that demand for their niche is going to be around for very long. I also pointed out that sysadmins who know what they're doing use awk. Based on your mentioning awk it seems like you agree with this.
  - DarylZero 5y ago
    >containerisation, cloud vendors, cattle not pets, infrastructure as code Lots of shell script in that part of the code world.
  - ozim 5y ago
    I care to disagree because you just move one abstraction layer higher and again you will use sed and awk. To move stuff to that cattle and to maintain cloud environments, to prepare k8s clusters and manage those. To create docker images or to maintain servers that are running k8s. You still have AWS console and Azure CLI or GCP console and configurations that need to be search/replaced. You still need to parse logs to understand what is happening in multiple environments, which I would say makes it even more important to be good with unix-fu because you have to find that one broken cow to put it out of misery. Those command line tools are still useful and used.
gattilorenz 5y ago
> Most of the justifications for using collections of command-line Unix tools are no longer valid today. Instead you should be using a proper programming language. That's just, like, your opinion man... > people who still do use complex solutions built from cat, head, cut, etc, and who know what they're doing, will typically either write a shell script or use awk No, I still use cat, head, cut etc., because it's easier to see at each step what is happening and incrementally add to that, because they're literally everywhere, because it's quick, and because I like it. Not for major projects, granted, but why would I need to write a python file for something that takes a single line of piped commands?
- unixbane 5y ago
  It's not hard: Imagine if your terminal executed Python repl instead of bash shell. Literally anything is better than bash. It's amazing how people still haven't figured out bash is basically PHP.
  - gattilorenz 5y ago
    > Imagine if your terminal executed Python repl instead of bash shell. Ugh.
  - deleted 5y ago
    [deleted]
- pkrumins 5y ago
  Amen.

3np 5y ago

I guess it depends on what you do. For a subset of tasks, shell scripting is a lot faster to implement than the equivalent python/js/go/ruby/rust.

Kinrany 5y ago

No "proper programming language" is capable of ergonomically piping between programs. Shell is indeed very old and it's time for a replacement, but it's not there yet. Oilshell might get there eventually or at least spark interest in this area.

ilyash 5y ago
> No "proper programming language" is capable of ergonomically piping between programs. Solved. https://github.com/ngs-lang/ngs https://github.com/ngs-lang/ngs I'm the author. Frustrated with exactly this situation I created Next Generation Shell. It's a "proper programming language" on one hand but domain-specific for "DevOps"y scripting on another. So sane syntax, data structures, error handling, multiple dispatch on one hand but also syntax for running external programs, pipes and redirects. You are welcome!
ccalloway 5y ago
> No "proper programming language" is capable of ergonomically piping between programs. You cherry-picked the one thing that shell script is somewhat better at than other languages (not even really needed for this task). Meanwhile the article uses shell for both making HTTP requests and mangling JSON data, both of which are easy in all modern languages, and extremely painful in shell.
- actually_a_dog 5y ago
  It's not cherry picking to mention the thing that makes a fundamental component of the UNIX philosophy work. > (ii) Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input. https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s06.html https://homepage.cs.uri.edu/~thenry/resources/unix_art/ch01s...
  - ccalloway 5y ago
    OK, the Unix philosophy holds that this is one of the most important things, and therefore it's easy to do in shell. But what if it's not? It's perfectly easy to write code to do the same thing in a programming language without invoking any other executables, let alone piping the output of one into the input of another. So the claim that this functionality is what makes Unix and shell scripting effective is vacuous. They are good at doing something which would not be necessary if one was not using them.
    - Kinrany 5y ago
      Being able to compose programs is not a necessity, but it saves effort. A single pipe replaces M*N different programs with M+N different programs. This is how Unix won.
    - deleted 5y ago
      [deleted]
unixbane 5y ago
*sh is only ergonomic if you're using it insecurely. wait, even then it's not ergonomic. you're just insane. the day i figured out about python repl over 10 years ago, i realized *sh is utterly obsolete (and python isn't even good, a repl like that just gives you some perspective of how things could be).

smitty1e 5y ago

> command-line Unix tools are no longer valid today Strongly disagree. The understanding of the OS, the data, and how to checkmate the problem with minimal effort is timeless. Programming languages are relatively ephemeral compared to POSIX utilities. Invest in knowledge of the enduring.

akho 5y ago

> Instead you should be using a proper programming language. Why use many word when few word do trick? Shell scripts are very compact for what they do, and generally as clear as a similarly quick-and-dirty solution in another language. Ease of learning and use comes from having used the programs you are running in your script. E. g. my small script piping stuff from nmcli to fzf so I can choose a wifi network would be much more difficult to write in Python: I’d need to find a Python library for interacting with NetworkManager, and a library for interactively fuzzy searching in a list, read the docs, spend a while setting up a venv to run it, … I don’t have time for any of that. xargs, in particular, is not difficult to invoke once you’ve done it once or twice, and does a lot. Apart from just being a loop, it also parallelizes execution, and can ask for confirmation from the user for each invocation. Implementing either of these features will take you more than 2-4 chars you need to use it with xargs.

spekcular 5y ago

I don't understand why you're being so harshly downvoted. This seems ... plausibly correct to me? Some questions: 1) Where do you draw the line, and move away from shell to a "real" language. If I just want want to view a directory, surely I use ls, right? What about if I want to remove all leading whitespace from a text file? This can be done with a short but fairly opaque awk one-liner. Probably takes way more time to write and run the Python equivalent, but I don't know Python so well, so maybe it's also a one-liner. 2) What's the optimal "real language" for replacing shell? Python? Perl? Raku? 2a) xargs automatically parallelizes programs. How can this be done efficiently (meaning I don't have to write much additional code) in your proposed "real language"?

dagw 5y ago
xargs automatically parallelizes programs. How can this be done efficiently (meaning I don't have to write much additional code) in your proposed "real language"? By using xargs :) Or more likely gnu parallel. But seriously, in so many cases using GNU parallel to parallelize a process is the quickest and easiest way to approach the problem, and I use it all the time. If I need to process 10k+ images in a folder, rather than try to parallelize the process in my python/C script I'll write the fastest possible single threaded script that takes its input args as command line arguments and then uses gnu parallel to distribute the workload. The added advantage is that I can distribute this work on a cluster of machines with only a few changes to GNU Parallel's command line arguments.
actually_a_dog 5y ago
We had a rule at my old workplace that any shell script that's over a page of code was to be replaced with a Python script. We chose Python because guaranteed there would always be an up to date version of Python on any cloud machine we used, but you could certainly make a case for some other language. In a vacuum, my choice might be Scheme.

aulin 5y ago

wait, what's wrong with hand soldering and bitwise arithmetic? hand soldering is still of very much practical utility for building prototypes, small production batches, electronics repair... Bitwise arithmetic... seriously? tell that to an embedded developer or anyone who works with low-level stuff.

ccalloway 5y ago
Nothing is wrong with them if 1) you have a specifically suited, niche problem and you understand the complexities and tradeoffs of the tool OR 2) it's not a critical requirement to solve the problem using modern, efficient tools AND you want to use something else for fun or learning or whatever. If this isn't the case, stay away.
- aulin 5y ago
  Either you're just trolling at this point OR you haven't the slightest idea of what you're talking about.

2143 5y ago

> Most of the justifications for using collections of command-line Unix tools are no longer valid today. Why is it not valid today? > One-liners which pipe text between four or five different commands are the equivalent of hand-soldered boards or bitwise arithmetic. Why is that bad? Don't deploy a supercomputer to do what a hand soldered board can. Keep it simple. I don't see how you equate piping commands to bitwise arithmetic, but bitwise arithmetic is easy anyway. > but of no practical utility Says you. Just because you don't find something useful doesn't mean nobody else finds it useful. Turning one liner piped commands to a program in what you might consider a "proper programming language" usually ends up turning a declarative program into something prodecural. Not that that's necessarily a bad thing. Just saying. Use the right tool for the job. In some cases (not all) the shell is indeed the right tool. I get a feeling you don't understand the Unix philosophy. Read The Art of Unix Programming by Eric S. Raymond. Go learn bitwise arithmetic. The uneducated play with pictures. Educated people read and write :) Have a great day (or night, depending on your timezone — night here).

Chris2048 5y ago
> Turning one liner piped commands to a program in what you might consider a "proper programming language" usually ends up turning a declarative program into something prodecural. I don't understand - isn't bash shell procedural?
Chris2048 5y ago
> I get a feeling you don't understand the Unix philosophy. Read The Art of Unix Programming by Eric S. Raymond. Go learn bitwise arithmetic. These kind of comments aren't useful, and come off smug to me. Why doesn't he understand something, what it the point in reading/learning something you think relevant? Why would they invest in your suggestions if you don't provide any reasons what is missing? Also: > The uneducated play with pictures. Educated people read and write :) This sounds insulting to me (name-calling), and passive-aggressively so when combined with "Have a great day".

smcameron 5y ago

This is akin to saying if you want to hang a picture in your house, you shouldn't just grab a hammer and a nail, instead, you should get a nice piece of wood, and a nice hunk of metal, go to the workshop, fire up the forge, the mill and various wood and metal working machines, and forge a special picture-hanging-hammerer-thing.

ccalloway 5y ago
Umm, no, chaining together 5 or more somewhat arcane single-purpose tools is the unrealistic solution here. If the article was about using ls to just list files, and I had said "actually you should use Python's os.listdir() and filter the results by whatever" you would be right. For most simple problems it's correct to use a simple tool. For the overwhelming majority of complex problems you should use a well-understood, well-designed, common general-purpose tool.
- pessimizer 5y ago
  What's supposed to be the difference between chaining simple UNIX commands and chaining simple python functions again?
  - ccalloway 5y ago
    For example, the latter is testable.
  - Chris2048 5y ago
    How many executables do the two methods require? Do all those executables/programs share a common type system?

herbst 5y ago

This is only partially true. Working on your own machine this may be fully the case, but debugging and maintaining random servers is a complete different beast. Shell is portable in a way nothing else is, same reason people use Excel instead of code or PHP instead of literally anything.

ccalloway 5y ago
Firstly the model of sshing into your server and trying to run commands on it directly is more or less obsolete. Secondly the presence of all these utilities on a machine is far from guaranteed - expecting Python to be present is no more or less likely.
- cranekam 5y ago
  > Firstly the model of sshing into your server and trying to run commands on it directly is more or less obsolete. Why do you get to declare what is acceptable (using UNIX tools) or what's now obsolete (SSHing into a host)? It's rather presumptuous to believe you can speak for everyone. Perhaps you only ever deploy code using k8s and never need to use a command line but that doesn't mean we all do. There are many reasons one would use these tools and approaches (fun, small scale, investigating problems, anything).
  - ccalloway 5y ago
    You might think 'fun, small scale, investigating problems' are good justifications for your working practices, but your boss's boss's boss does not. They would prefer you to have less fun. Or perhaps to have fun on your own time after being laid off.
    - herbst 5y ago
      My boss (me) doesn't mind me dabbling around in my servers via SSH. Not everyone of us works for evil cooperate.
      - 5y ago

Lamad123 5y ago

I heard awk is extremely fast, much faster than Python.

hansel_der 5y ago
i heard python is among of the slowest
ccalloway 5y ago
Yes, but execution speed (of things like creating a lookup table from the second and third fields in each line) is rarely the operative constraint.
- jes 5y ago
  What is most often the operative constraint?
  - ccalloway 5y ago
    Time spent maintaining your script by your future co-workers.
revscat 5y ago
It is, but it's much closer to perl when it comes to maintainability.

high_5 5y ago

> Interesting to learn about for historical reasons but of no practical utility. The practical utility has just been demonstrated in this particular article? Historical? I think that Unix shell are like crocodiles - outliving the dinosaurs and lurking in the murky water the unsuspecting sysadmin to come close enough to fix the script that ain't broken.

oblio 5y ago
Your analogy is quite interesting, in the sense that crocodiles are around, but they're not the dominant species. That would be an interesting continuation to your analogy, actually.
- high_5 5y ago
  Sooner or later one has to take a sip from the unixy pond :-)
  - oblio 5y ago
    For now. At some point we're going to build proper water management and sanitation systems. What those are in the software world, I, as a swamp dweller, cannot say.
ccalloway 5y ago
> The practical utility has just been demonstrated in this particular article? How? The author is doing something which could be done much more easily and elegantly in a programming language.

flohofwoe 5y ago

Fundamentally it's the same thing. The UNIX tools are the "batteries included" standard library of the integrated shell programming environment. And if you need to extend that library you quickly whip up a new minimal command line tool in C (or any other language which allows to write small and quick and dirty command line tools, like Python). The only downside of shell scripting is that it isn't trivially portable to Windows (or even macOS because of the differences between GNU and BSD tools), so it often makes sense to create big Python scripts that do more than "one thing right". If the whole world would run on UNIX, shell scripting would make much more sense.

actually_a_dog 5y ago
> The only downside of shell scripting is.... The only downside? Let's add that no major *NIX shell that I'm aware of has any good way to modularize code while enforcing encapsulation of state. At my previous job, we had a rule that any shell script longer than about a page of code had to be replaced with a Python script ASAP. That was a good rule, IMO, because once you've exceeded a certain size, a shell script starts getting brittle and hard to work with. I don't know if 1 page of code is the threshold size or not, but it seems like as good a cutoff as any.

hawski 5y ago

Shell is a very high level language that is undoubtedly bizarre at times. Its routines can be written in any programming language you want without much of a boilerplate or any special bindings, because supporting for command line arguments, environment variables, reading and writing files is a bare minimum that most languages support. It is a tradeoff - it has its immense advantages and a couple of often hard to navigate disadvantages. But the ability to easily compose whatever you want is something you can't ignore. I think the perfect very high level language is closer to shell than to python for example. The power of Tcl/Tk (still it has some big weaknesses) or Rebol/Red is something that I admire. The following statement is probably controversial: shell is more akin to Lisp for human beings. I dabbled at Scheme, but it is harder for me to grasp than Shell, but in the end they are more similar than not. I hold a candle for Oil shell for example.

pkrumins 5y ago

Sir, you are absolutely and totally wrong. No sysadmin has time or interest to write programs or use "proper programming languages" to get the job done. Sysadmins know their tools and are fast and efficient. Zero sysadmins will ever write "proper programs" when they can pipe find, sed, and awk.

ccalloway 5y ago
Yes, but the number of sysadmins in the world is converging rapidly to zero. Mainly because the invisible overhead of having your boxes managed by people who use sed and awk is much higher than the alternatives (containerisation, cloud vendors, cattle not pets, infrastructure as code). I think you actually confirmed my point when you said 'no sysadmin has time or interest'. I didn't say that no one uses shell anymore. I said that there was no practical justification for doing so. Lots of people still use it and don't have any interest in finding a better solution. These people are choosing, for their own reasons, to double down on an obsolete skillset. Good for them, I suppose, but I don't think that demand for their niche is going to be around for very long. I also pointed out that sysadmins who know what they're doing use awk. Based on your mentioning awk it seems like you agree with this.
- DarylZero 5y ago
  >containerisation, cloud vendors, cattle not pets, infrastructure as code Lots of shell script in that part of the code world.
- ozim 5y ago
  I care to disagree because you just move one abstraction layer higher and again you will use sed and awk. To move stuff to that cattle and to maintain cloud environments, to prepare k8s clusters and manage those. To create docker images or to maintain servers that are running k8s. You still have AWS console and Azure CLI or GCP console and configurations that need to be search/replaced. You still need to parse logs to understand what is happening in multiple environments, which I would say makes it even more important to be good with unix-fu because you have to find that one broken cow to put it out of misery. Those command line tools are still useful and used.

gattilorenz 5y ago

> Most of the justifications for using collections of command-line Unix tools are no longer valid today. Instead you should be using a proper programming language. That's just, like, your opinion man... > people who still do use complex solutions built from cat, head, cut, etc, and who know what they're doing, will typically either write a shell script or use awk No, I still use cat, head, cut etc., because it's easier to see at each step what is happening and incrementally add to that, because they're literally everywhere, because it's quick, and because I like it. Not for major projects, granted, but why would I need to write a python file for something that takes a single line of piped commands?

unixbane 5y ago
It's not hard: Imagine if your terminal executed Python repl instead of bash shell. Literally anything is better than bash. It's amazing how people still haven't figured out bash is basically PHP.
- gattilorenz 5y ago
  > Imagine if your terminal executed Python repl instead of bash shell. Ugh.
- deleted 5y ago
  [deleted]
pkrumins 5y ago
Amen.