9 ms·

Thinking over this, it just (finally) struck me that awk (and sed, grep, etc.) are basically just libraries (more like DSLs), and the shell is just another scri

Thinking over this, it just (finally) struck me that awk (and sed, grep, etc.) are basically just libraries (more like DSLs), and the shell is just another scripting language with a REPL/notebook environment. When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? I don't think ubiquity/portability is a compelling answer, since (for many/most people who care about this) it is relatively easy to install the environment of their choice on their personal computing device. BTW, we must also strive to maintain that flexibility in the next generation of computing devices. As an aside, folks who do like the task-focused nature of shell DSLs might appreciate language-oriented programming with Racket.

mordechai9000 7y ago
Awk is part of BusyBox, thus it's present by default on a wide range of embedded systems/systems with limited resources. Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.
- Annatar 7y ago
  There is no inconsistency if one programs in straight Bourne shell from 1978, not using any additional features, and frankly, I've been programming in it for 25+ years and never needed "bashisms". ksh is also extremely consistent and portable. One of the main hidden strengths of AWK is that there is one and only one way how to do something in AWK syntax, so from that point, as long as one doesn't use GNU AWK-isms, it's programming luxury and paradise. Programs written in non-GNU AWK run everywhere (one would have to go out of one's way to use Thompson specific features, for example). As far as I'm concerned, it's a matter of understanding UNIX, not GNU/Linux, but UNIX, so that it is clear what is what and what specifically to avoid (like any GNU features). People who grew up on GNU/Linux often struggle with understanding what's what because they have a huge knowledge gap. The fact that GNU/Linux was their first UNIXoid OS is actually highly detrimental now.
snagglegaggle 7y ago
I suspect shell languages see widespread use because they are optimized for interactivity. The barrier between thinking of an action and making it run is low. At a certain point there is diminishing returns for particularly complex actions, e.g. you call out to awk instead of implementing awk in a shell language all else being equal.
jdc 7y ago
Python and Lisp use way more parens (which many find unergonomic) and don't have pipes. Let me qualify that - parens are cumbersome with the usual tools and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That being said, there is a Python shell called xonsh, but that uses kind of a Python - bash hybrid AFAIK.
- shakna 7y ago
  > and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That isn't quite true. While, yes, a pipe operator would probably be something new that you add to the language, Scheme which Racket originally comes from, actually comes with first-class pipe support. Scheme makes use of a concept called ports, quite extensively. You can turn files, sockets, string, etc. into ports. Ports are most definitely idiomatic Scheme. Racket extends ports with this: (make-pipe [limit input-name output-name]) Which works to transfer data from one port to another in basically the same way the pipe operator works at the shell level. To make it work the way people are more used to, you might also combine it with the infix library. So if you wanted to create a pipe operator for some reason, everything you need is already there, it'd just by syntax sugar.
  - jdc 7y ago
    Heh neat. I guess this is what scsh does.
jolmg 7y ago
> When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Well, first: > possibly better forms of "glue" -- say Python That's not a better form of glue. First, consider that executables of all languages are standarized to work with text. They get text arguments, text input through stdin, text output through stdout and stderr, text input and output through other files whose paths are given as arguments, etc. Shell languages are good glue because they work only with text. With the exception of control structures like "if", and "while", all statements in a shell languages are based on text. The command name is just text, the arguments are text. Syntax facilitates the manipulation of their standard inputs and outputs. Command substitution allows one to glue the output of one command as arguments to another. That's what makes shell languages a good glue. Python, on the other hand, is not as good at gluing these executables written in a myriad of languages because it doesn't put enough importance on this universal interface between languages that is text, nor does it sufficiently facilitate the execution of other executables. Another thing to consider is that shell languages are meant primarily to be used interactively. Things like putting commas between arguments and parenthesis around them would be a nuisance in interactive use. Having to quote every argument would also be a nuisance. In a not-primarily-interactive language like python, one uses variables more than literals, so it's easier to write variables than quoted strings. In a primarily-interactive language like bash, one typically uses literals more than variables, so it's the variables in bash that require the extra syntax (the sigil, '$'), while literal strings do not (most of the time). > with a uniform syntax I'm not sure what a more "uniform" syntax is, but I get your drift that there are warts in typical shell languages. I would also prefer that `if` statements didn't end in `fi` or `case` statements end in `esac`, but I think languages like bash and zsh offer to much to forgo them for such little details. There are more to these languages than these warts that makes them worth it. You might be interested in the fish shell project. It's a newish shell language with a simplified ("uniform"?) syntax. Personally, I don't use it because it lacks too many features, but it's there if you like it. > Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? That's shell languages. If you meant something like python, you can. Just do `usermod -s $(which python) $USER` to set python to be your shell. I wouldn't last long on it as my shell for the reasons I mentioned, but there's no-one not allowing us to if we wanted.
dan-robertson 7y ago
I guess shell scripting and the standard commands are convenient interfaces to a lot more functionality. The interactive use (and pipes) tend to make the linear processing super terse and easy (but lots of other things become annoying). In a type-safe language, it’s typically impossible to specify a function which can do as many things as (eg) sort can, and which doesn’t require specifying a huge amount of default arguments on every invocation. Therefore you end up with a limited function, a huge annoying function, or many small functions. In any case, changing from one kind of sort to another (eg how to sort things, what to sort, how to order them, whether to keep unique elements, whether to be stable, whether to take multiple inputs, whether to assume the inputs are sorted (so just merge), and so on) becomes difficult whereas with a tweak-in-many-ways command like sort it is easy. In a language like CL with optional arguments and no types, this problem can be less hard but whereas sort will likely work fone for a large dataset, your favourite library function may not. (In particular it almost definitely won’t use on-disk storage when needed or support parallelism. Partly note that storing objects from your favourite language to disk is probably hard but if you only work on lines of text it’s easy). The other big difference is the data types. The data type of shell programs is basically a sequence of lines, each of which is sometimes broken up into one or more records by some other separator. In many languages the sequence operations are all about getting the nth element, or extending them, or changing elements, or deleting them, or splitting the sequence up into other sequences. And these operations typically identify the elements by the position. In shell scripts, many of these operations are unavailable or not used. One basically only iterates forwards, processing each element, and one almost never thinks about the position of the element in the sequence. (c)split exists but isn’t commonly used. Lots of work in a typical language is converting one data type to another, or extracting bits of it, or doing data type-specific things. In shell these don’t really exist (eg you don’t sort as dates, you write your dates like 2019-09-28 and sort them like strings), and so data is simple, and because extraction is flexible (typically a field number or byte positions) lots of the bureaucracy of changing data types is omitted. These things all often lead to shell scripts being unreliable. But also resilient, flexible and “sufficient”. By “sufficient” I mean that they can do things well enough that the cost for a long term, thorough, or “proper” solution isn’t justified.
- wyatwerpy 7y ago
mistahenry 7y ago
Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow. At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use
- umvi 7y ago
  > I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow. That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows. [1] https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk
  - zwkrt 7y ago
    I guarantee that more people know awk syntax than pawk!
    - deleted 7y ago
      [deleted]
- Too 7y ago
  > It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.
inimino 7y ago
This is true. Some reasons why sh hasn't been replaced by LISP (for example) as the default Unix shell is because: - it was there first - grep awk etc are heavily optimized and faster than their equivalents in most languages - the primary job of a shell is to run other programs, not to be the program. Scripting languages don't generally let you run a program just by typing the name and hitting enter. All that being said, you can set your login shell to be whatever you want in /etc/passwd, and you're free to login to your REPL of choice instead of sh if that's what you really want.
- muro 7y ago
  Isn't lisp older than shell? :)
  - inimino 7y ago
    Yes. I mean it was there first in the Unix environment, so there's tremendous historical inertia that would be almost impossible to overcome.
agumonkey 7y ago
I think that's the mindset of 'the unix environment'. They see shell + fs as a repl.
- Annatar 7y ago
  Because that's exactly what it was designed to be and does it remarkably well.

mordechai9000 7y ago

Awk is part of BusyBox, thus it's present by default on a wide range of embedded systems/systems with limited resources. Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.

Annatar 7y ago
There is no inconsistency if one programs in straight Bourne shell from 1978, not using any additional features, and frankly, I've been programming in it for 25+ years and never needed "bashisms". ksh is also extremely consistent and portable. One of the main hidden strengths of AWK is that there is one and only one way how to do something in AWK syntax, so from that point, as long as one doesn't use GNU AWK-isms, it's programming luxury and paradise. Programs written in non-GNU AWK run everywhere (one would have to go out of one's way to use Thompson specific features, for example). As far as I'm concerned, it's a matter of understanding UNIX, not GNU/Linux, but UNIX, so that it is clear what is what and what specifically to avoid (like any GNU features). People who grew up on GNU/Linux often struggle with understanding what's what because they have a huge knowledge gap. The fact that GNU/Linux was their first UNIXoid OS is actually highly detrimental now.

snagglegaggle 7y ago

I suspect shell languages see widespread use because they are optimized for interactivity. The barrier between thinking of an action and making it run is low. At a certain point there is diminishing returns for particularly complex actions, e.g. you call out to awk instead of implementing awk in a shell language all else being equal.

jdc 7y ago

Python and Lisp use way more parens (which many find unergonomic) and don't have pipes. Let me qualify that - parens are cumbersome with the usual tools and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That being said, there is a Python shell called xonsh, but that uses kind of a Python - bash hybrid AFAIK.

shakna 7y ago
> and even if you add pipes to your Python/Lisp project the ecosystem doesn't follow the idiom. That isn't quite true. While, yes, a pipe operator would probably be something new that you add to the language, Scheme which Racket originally comes from, actually comes with first-class pipe support. Scheme makes use of a concept called ports, quite extensively. You can turn files, sockets, string, etc. into ports. Ports are most definitely idiomatic Scheme. Racket extends ports with this: (make-pipe [limit input-name output-name]) Which works to transfer data from one port to another in basically the same way the pipe operator works at the shell level. To make it work the way people are more used to, you might also combine it with the infix library. So if you wanted to create a pipe operator for some reason, everything you need is already there, it'd just by syntax sugar.
- jdc 7y ago
  Heh neat. I guess this is what scsh does.

jolmg 7y ago

> When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Well, first: > possibly better forms of "glue" -- say Python That's not a better form of glue. First, consider that executables of all languages are standarized to work with text. They get text arguments, text input through stdin, text output through stdout and stderr, text input and output through other files whose paths are given as arguments, etc. Shell languages are good glue because they work only with text. With the exception of control structures like "if", and "while", all statements in a shell languages are based on text. The command name is just text, the arguments are text. Syntax facilitates the manipulation of their standard inputs and outputs. Command substitution allows one to glue the output of one command as arguments to another. That's what makes shell languages a good glue. Python, on the other hand, is not as good at gluing these executables written in a myriad of languages because it doesn't put enough importance on this universal interface between languages that is text, nor does it sufficiently facilitate the execution of other executables. Another thing to consider is that shell languages are meant primarily to be used interactively. Things like putting commas between arguments and parenthesis around them would be a nuisance in interactive use. Having to quote every argument would also be a nuisance. In a not-primarily-interactive language like python, one uses variables more than literals, so it's easier to write variables than quoted strings. In a primarily-interactive language like bash, one typically uses literals more than variables, so it's the variables in bash that require the extra syntax (the sigil, '$'), while literal strings do not (most of the time). > with a uniform syntax I'm not sure what a more "uniform" syntax is, but I get your drift that there are warts in typical shell languages. I would also prefer that `if` statements didn't end in `fi` or `case` statements end in `esac`, but I think languages like bash and zsh offer to much to forgo them for such little details. There are more to these languages than these warts that makes them worth it. You might be interested in the fish shell project. It's a newish shell language with a simplified ("uniform"?) syntax. Personally, I don't use it because it lacks too many features, but it's there if you like it. > Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? That's shell languages. If you meant something like python, you can. Just do `usermod -s $(which python) $USER` to set python to be your shell. I wouldn't last long on it as my shell for the reasons I mentioned, but there's no-one not allowing us to if we wanted.

dan-robertson 7y ago

I guess shell scripting and the standard commands are convenient interfaces to a lot more functionality. The interactive use (and pipes) tend to make the linear processing super terse and easy (but lots of other things become annoying). In a type-safe language, it’s typically impossible to specify a function which can do as many things as (eg) sort can, and which doesn’t require specifying a huge amount of default arguments on every invocation. Therefore you end up with a limited function, a huge annoying function, or many small functions. In any case, changing from one kind of sort to another (eg how to sort things, what to sort, how to order them, whether to keep unique elements, whether to be stable, whether to take multiple inputs, whether to assume the inputs are sorted (so just merge), and so on) becomes difficult whereas with a tweak-in-many-ways command like sort it is easy. In a language like CL with optional arguments and no types, this problem can be less hard but whereas sort will likely work fone for a large dataset, your favourite library function may not. (In particular it almost definitely won’t use on-disk storage when needed or support parallelism. Partly note that storing objects from your favourite language to disk is probably hard but if you only work on lines of text it’s easy). The other big difference is the data types. The data type of shell programs is basically a sequence of lines, each of which is sometimes broken up into one or more records by some other separator. In many languages the sequence operations are all about getting the nth element, or extending them, or changing elements, or deleting them, or splitting the sequence up into other sequences. And these operations typically identify the elements by the position. In shell scripts, many of these operations are unavailable or not used. One basically only iterates forwards, processing each element, and one almost never thinks about the position of the element in the sequence. (c)split exists but isn’t commonly used. Lots of work in a typical language is converting one data type to another, or extracting bits of it, or doing data type-specific things. In shell these don’t really exist (eg you don’t sort as dates, you write your dates like 2019-09-28 and sort them like strings), and so data is simple, and because extraction is flexible (typically a field number or byte positions) lots of the bureaucracy of changing data types is omitted. These things all often lead to shell scripts being unreliable. But also resilient, flexible and “sufficient”. By “sufficient” I mean that they can do things well enough that the cost for a long term, thorough, or “proper” solution isn’t justified.

wyatwerpy 7y ago

mistahenry 7y ago

Command line tools are very terse, heavily optimized (ie fast), interactive, and easily chainable. I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. The other day my boss asked me to find out if there was any second where we processed more than 100 transactions a second on our server in the last 90 days. I needed to go through 100s of GB of data which mawk managed in no time and in a relatively simple one liner. I imagine my idiomatic python script would still be running Often times, analysis is an evolving thing. As you uncover one answer, new questions arise. It’s far easier imo to tweak my command line pipelines than to modify program flow. At the end of the day, general purpose programming languages are jacks of all trades and masters of none. They’re inherently held back by their flexibility when compared to DSLs. Text processing is so easy in awk b/c it’s a text processing domain specific language. If you really think that we should use on language for everything, you’re ignoring the value that specialization can provide you, whether in speed, power, or ease of use

umvi 7y ago
> I would never want to write a python program for quick analysis of a log file when I can grep/sed/awk/sort/uniq my way to a solution. Yeah but I would wager very few people actually know awk. I would also wager 90+% of awk programs are pasted directly from StackOverflow. That's why I like the idea of pawk[1] - it's like awk, except with python syntax instead of some esoteric syntax that nobody knows. [1] https://github.com/alecthomas/pawk https://github.com/alecthomas/pawk
- zwkrt 7y ago
  I guarantee that more people know awk syntax than pawk!
  - deleted 7y ago
    [deleted]
Too 7y ago
> It’s far easier imo to tweak my command line pipelines than to modify program flow. Why? Creating a throwaway file in the IDE and attaching the debugger to it is just two keyboard shortcuts away. Then i can just change one line, hit F5 and get the output instantly. Contrast with the command line where i have to be careful not to hit the up-button and get all my modifications reverted, and can't use fundamental features like selecting text using shift+left/right. When writing slightly longer shell-pipelines i even tend to first write them in the IDE and then paste them into the shell to avoid this! With a program i can also set breakpoints halfway through the script to see what has happened halfway through a pipeline.

inimino 7y ago

This is true. Some reasons why sh hasn't been replaced by LISP (for example) as the default Unix shell is because: - it was there first - grep awk etc are heavily optimized and faster than their equivalents in most languages - the primary job of a shell is to run other programs, not to be the program. Scripting languages don't generally let you run a program just by typing the name and hitting enter. All that being said, you can set your login shell to be whatever you want in /etc/passwd, and you're free to login to your REPL of choice instead of sh if that's what you really want.

muro 7y ago
Isn't lisp older than shell? :)
- inimino 7y ago
  Yes. I mean it was there first in the Unix environment, so there's tremendous historical inertia that would be almost impossible to overcome.

agumonkey 7y ago

I think that's the mindset of 'the unix environment'. They see shell + fs as a repl.

Annatar 7y ago
Because that's exactly what it was designed to be and does it remarkably well.