7 ms·

If this wants to sell itself as a shell scripting language, it should very quickly advertise what is it that makes it superior to say, bash for typical shell sc

If this wants to sell itself as a shell scripting language, it should very quickly advertise what is it that makes it superior to say, bash for typical shell scripting tasks. Shell scripts with bash are painful to the point that if I find myself writing more than around 10 lines of shell, I tend to stop and switch to Perl instead. But Perl hasn't been too popular lately and isn't ideal either, so I'm very much up for something better. Here's some features I want from a shell scripting language: * 100% reliable argument passing. That is, when I run `system("git", "clone", $url);` in Perl, I know with exact precision what arguments Git is going to get, and that no matter what weirdness $url contains, it'll be passed down as a single argument. Heck, make that mandatory. * 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. * No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. * Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B". * Good error handling and reporting * Easy capture of stdout and stderr, at the same time. Either together or individually, as needed. * Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available, while being able to report progress, record each failure, and abort the entire thing if needed. * Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". * Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. That's the kind of thing I care about for shell scripting. A better syntax is nice, but actually getting stuff done without having to work around gotchas and issues is what I'm looking for.

danmur 5y ago
Oh my god, your first point. I had to pass arbitrary arguments through a couple layers of shell and I couldn't do it. Gave up and rewrote it all in Python, even though it's annoying in other ways, and I'm decent at shell scripting. It's a nightmare.
jiggawatts 5y ago
Most of those features are in PowerShell, especially version 7. It uses strict ('reliable') argument passing, strong typing, etc.. It has a bazillion independent streams such as "Debug", "Verbose", "Warning", "Error", "Progress", "Output", "Information", etc... It has "foreach -parallel", which is a fun way to make the CPU fan spin up. > "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. $lines = dir -Recurse -Exclude *.git, *.vim | Get-Content Not exactly rocket science in most shells? In most shells you can make this a modular function that takes in a list of file names, but PowerShell's "dir" doesn't output file names. It outputs file objects, so then it is just one step to do a filter on something like extension (or length, or whatever): dir | ? Length -lt 1MB | ? Extension -in @( '.cs', '.rs', '.cpp' ) In PowerShell you can create functions that integrate with the object-oriented pipeline. You can pipe full objects into a function, and objects out. Object attributes can be mapped automatically to function arguments too. You can have begin/process/end blocks so you can do things like open a DB connection, pipe in data, then close the connection neatly without needing to do this all manually each time. Literally something as trivial as this: Get-Stuff | Out-DatabaseTable -TableName 'stuff_log' -Server '...' Seriously. Give it a go. It works on Linux too, and it's more unix than unix.
- dale_glass 5y ago
  PowerShell is indeed excellent, but there's nothing wrong with it having a bit of competition. Some of it is also less than ideal, but rather less than bash. Issues in powershell I can think of: * It leaks environment variables. Set an env var, and it propagates to the shell from which it was called from! * It can't comfortably import environment from batch files. Working with vcvars is an annoyance. You'd think somebody at Microsoft would have made built in support for that one. * Argument passing in Windows is a bloody horror. PowerShell accepts commands encoded in base64 because it sucks so much! * For some reason you can't quote the command itself. You can do: git checkout "blah" But for some reason can't do: "C:\Program Files\Git\bin\git.exe" checkout "blah". Granted, that can be worked around with & or Start-Process, but what a pointless annoyance. That said, I've not really tried it on Linux yet, got to give it a go.
paskozdilar 5y ago
Bash is definitely painful to write - I write shell scripts a lot and I enjoy it (in a masochistic kind of way), and I wholeheartedly agree with most of the criticisms you've written. However, the greatest strength of Bash is its ubiquity - it is available on almost every modern Unix-like environment - and when combined with standard POSIX tools it can provide a-little-less-painful environment for writing reliable shell scripts. In order to prove it, I will do my best to provide some kind of solution to each of the problems you have mentioned. Note that they are in no way perfect solutions, and all of them are just hacks that work around Bash's inherent clumsiness. --- > 100% reliable argument passing Wrapping a variable in double-quotes does this: var="1 2 3 4" for x in "$var"; do echo $x; done For double-parsing (e.g. SSH commands) you can wrap the entire expression in single-quotes to maintain the original string until the second shell: ssh user@host 'var="1 2 3 4"; for x in "$var"; do echo $x; done' > 100% reliable file iteration The "find" command does exactly this, and it works for whatever weird characters in the filename you have. find ./directory/ -type f -exec SOME_COMMAND {} \; > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. Do not evaluate arguments directly in shell, use xargs to feed the arguments as standard input. E.g.: find ./directory/ -type f | xargs SOME_COMMAND > Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B". dirname /foo/bar basename /foo/bar realpath ./symlink/to/foo/bar basename /foo/bar | sed 's/.*\.//' realpath --relative-to ./dirA/fileA ./dirB/fileB > Good error handling and reporting This is a tough one. Good error handling/reporting in Bash is extremely hard and requires discipline, but it's possible. To exit immediately as soon as a command exits with non-zero exitcode: set -e However, this does not play nice with temporary resources, so I personally prefer hand-written error checks: SOME_COMMAND || { echo "Some Command has failed" && exit 1; } Error reporting depends on a scenario, but I generally have a pattern of doing checks like: if [ "$var" != "expected" ] then echo "Error: var different from expected" && exit 1 fi And it works fine most of the time. > Easy capture of stdout and stderr, at the same time. Either together or individually, as needed. To capture both stdout and stderr in a same stream: SOME_COMMAND |& SOME_FILTER Capturing stdout and stderr in different streams is tough. The easiest way I can think of is to use a pipe and job control mechanisms: mkfifo pipe SOME_COMMAND 2>pipe | FILTER_1 & < pipe FILTER_2 & wait rm pipe * Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available, while being able to report progress, record each failure, and abort the entire thing if needed. While bash is truly terrible when it comes to concurrency, I find GNU Parallel to be pretty satisfying for most concurrent shell-scripting: cat FILE.txt | parallel --timeout=30 SOME_COMMAND {} \|\| echo "Some Command failed with argument: {}" For POSIX purity, xargs also can be used with "-L 1" argument that parses a single line per command iteration. For parallelism, there is also a "--max-procs" argument. * Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". I find that adding "set -x" to the top of your shell, which prints each command with "+" prefix as it is expanded and executed, very useful for error tracking. * Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. There are helpers out there for most common situations - it's just that they are implemented as CLI tools, and not officially part of the shell. And in the scenario where you can't expect the availability of those unofficial tools, you can always write your own small library of commonly used Bash functions, and just copy-paste them into your script whenever you need them. It's ugly, but still possible.
darrenf 5y ago
In my (biased) opinion Perl comes close to ideal with judicious use of Path::Tiny and Capture::Tiny.
hnlmorg 5y ago
Shameless plug, but the problems you've described are exactly the problems I was looking to solve with my own shell (https://murex.rocks https://murex.rocks / https://github.com/lmorg/murex https://github.com/lmorg/murex) > 100% reliable argument passing. That is, when I run `system("git", "clone", $url);` in Perl, I know with exact precision what arguments Git is going to get, and that no matter what weirdness $url contains, it'll be passed down as a single argument. Heck, make that mandatory. Variables are parsed as tokens so they're passed to the parameters whole, regardless of whether they contain white space or not. So you can still use the Bash terseness of parameters (eg `git clone $url`) but $url works in the same way as `system("git", "clone", $url);` in Perl. > 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. Murex is inspired by Perl in that $ is a scalar and @ is an array. So if you use `f +f` builtin (to return a list of files), it returns as a JSON array. From there you can use @ to expand that array with each value being a new parameter, eg rm -v @{ f +f } # delete all files or use $ to pass the entire array as a JSON string. Eg echo ${ f +f } # you could just run `f +f` without `echo` to get the list. # This is just a contrived example of passing the array as a string. Additionally there are lots of tools that are natively aware of arrays and will operate on them. Much like how `sed`, `grep`, `sort` et al treat documents as lists, murex will support lists as JSON arrays. Or arrays in YAML, TOML, S-expressions, etc. So you can have code that looks like this: f +f | foreach file { echo $file } > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. This is a kernel limit. I don't think there is any way to overcome this without using iteration instead (and suffering the performance impact form that). > Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B". There are a number of ways to query files and paths in murex: - f: This returns files based on meta data. So will pull files, or directories, or symlinks, etc depending on the flags you pass. eg `+f` would include files. `+d` would be include directories. or `-s` would exclude symlinks. - g: Globbing. Basically `*` and `?`. This can run as a function rather than being auto-expanded. eg `g *.txt` or `rm -v @{ g *.txt }` - rx: Like globbing but using regexp. eg `rx '\.txt$'` or `rm -v @{ rx '\.txt$' }` - this example looks terrible but using rx does sometimes come in handy if you have more complex patterns than a standard glob could support. eg `rm -v @{rx '\.(txt|rtf|md|doc|docx)$'}` The interactive shell also have an fzf like integration built in. So you can hit ctrl+f and then type a regexp pattern to filter the results. This means if you need to navigate through complex source tree (for example) to a specific file (eg ./src/modules/example/main.c) you could just type `vi ^fex.*main` and you'd automatically filter to that result. Or even just type `main` and only see the files in that tree with `main` in their name. > Good error handling and reporting / Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". A lot of work here. - `try` / `trypipe` blocks supported - `if` and `while` blocks check the exit code. So you can do if { which foobar} else { err "foobar does not exist!" } # 'then' and 'else' are optional keywords for readability in scripting. So a one liner could read: # !if {which foobar} {err foobar does not exist!} - built in support for unit tests - built in support for watches (IDE feature where you can watch the state of a variable) - unset variables error by default - empty arrays will error by default when passed as parameters in the next release (hopefully coming this week) - STDERR is highlighted red by default (can be disabled if that's not to your tastes) so you can clearly see any errors if they're muddled inside STDOUT. - non zero exit numbers automatically raise an error. All errors return a line and cell number so you can find the exact source of the error in any scripts Plus lots of other stuff that's referenced in the documents > Easy capture of stdout and stderr, at the same time. Either together or individually, as needed. Murex handles named pipes a little differently, they're passed as parameters inside triangle brackets, <>. STDERR is referenced with a proceeding exclamation mark. eg a normal command will appear internally as: command1 <out> <!err> parameter1 parameter2 etc | command2 <out> <!err> parameter1 parameter2 etc (You don't need to specify <out> nor <!err> for normal operation). So if you want to send STDERR off somewhere for later processing you could create a new named pipe. eg pipe example # creates a new pipe called "example" command1 <!example> parameter1 parameter2 | command2 <!null> parameter1 parameter2 This would say "capture the STDERR of the first command1 and send it to a new pipe, but dump the STDERR of command2". You can then later query the named pipe: <example> | grep "error message" > Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. This is where the typed pipelines of murex come into their own. It's like using `jq` but builtin the shell itself and works transparently with multiple different document types. There's also an extensive library of builtins for common problems, eg `jsplit` will read STDIN and output an array split based on a regexp pattern. So your example would be: cat file | jsplit \n > That's the kind of thing I care about for shell scripting. A better syntax is nice, but actually getting stuff done without having to work around gotchas and issues is what I'm looking for. I completely agree. I expect this shell I've created to be pretty niche and not to everyone's tastes. But I'd written it because I wanted to be a more productive sysadmin ~10 years ago and since I've moved into DevOps I've found it invaluable. It's been my primary shell for around 5 years and every time I run into a situation where I'm like "I wish I could do this easily" I just add it. The fact that it's a typed pipeline makes it really easy to add context aware features, like SQL query support against CSV files.
inkyoto 5y ago
> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters. ------ > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents: command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»). How the length limit works is explained at length (please pardon the pun) here: https://www.in-ulm.de/~mascheck/various/argmax/ https://www.in-ulm.de/~mascheck/various/argmax/ ------ > Excellent path parsing. […] finding the file extension […] File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1). Moreover, any valid character can be used as a separator and its interpretation is either left out (almost always), or the interpretation is left up to the semantically aware app. One is free to use a comma or the Javanese wasana pada as part of the extension; that is, «file,exec» and «file꧅ ꦆ ꧅exec» are both valid and both have the extension of «exec» – as long as the file system supports the appropriate character set. This is also why «basename», to work correctly, requires the separator as part of the imaginary «extension», i.e. «basename myfile,exec ,exec» will always give «myfile» as the result. DOS style extensions are a made up convention that neither the kernel, nor the shell, nor file processing utilities enforce as they are file extension unaware. It is better to think of UNIX file names as being made up a prefix and an optional, arbitrary length suffix (but not an extension). ------ > Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available With respect to «every core available». You almost certainly don't want that and should overprovision the number of running processes compared to the number of cores on your system. UNIX has supported multiprocessor systems for a very long time. Despite massive improvements in the hardware performance, disks and networks are still the slowest moving (or still) parts. They were even slower when UNIX was in its relative infancy when the CPU time was also very expensive. Therefore, the CPU time had to utilised efficiently whilst waiting for a disk to return a string of bytes. File processing tasks spend their time between: 1) waiting for I/O (in the blocked state) and 2) actually processing (the running state). Since the disk is still very slow compared to the speed of a modern CPU, the UNIX process scheduler blocks the process until I/O completes and checks the process run queue in the kernel to see if there is another process ready to compute something (i.e. in the running state). This inherent interleaving of «blocked for I/O» and «running» process states can be used to an advantage depending on a few factors. The less is size of the unit of data an app processes and the larger is the total size of the input (i.e. the input and output files), the more time the app spends in the «blocked for I/O» state, and most of the CPU time is simply wasted unless there is something else to do. But if we know such specifics (the size of the unit of work and the size of the input), we can overprovision the number of processes thereby utilising the CPU compute time more efficiently whilst the disk controller is transferring bytes into the memory via the direct memory access (I am oversimplifying a few bits here). This is the reason why «make -j12» will compile almost always faster than «make -j8» on a 8 CPU core system on projects with a large number of small(-er) files – because of the I/O overhead. Whilst there is no universal formula, 1.5x process overprovisioning factor is a decent starting point. For simpler daily file processing tasks GNU parallel is good enough to spare oneself of headaches of such computations, though. Therefore, the «We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available […]» does not make sense in the context of UNIX shell languages and the process scheduling in UNIX. In fact, you will underutilise your 128 CPU cores, sometimes pretty heavily, unless you correctly account for the I/O factor. Only if the app/process is aware of how to efficiently parallelise its own workload (because it knows its workload better than anyone else), then and only then it does need to know how many cores there are available at its disposal. No scripting language / shell can solve this problem. ------ > Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". The excellent error reporting has existed in UNIX since day 1. It is called the process status code; anything that is not a zero status code indicates an error. The semantic interpretation of each speific numeric status code, however, is entirely decoupled from the thing that might fail and is documented in the man page. The status code of 3 in «mv» and in «git» will mean two completely different things, therefore the specific status code processing is localised to the process invocation point in shell scripts. Most of the time, though, I personally don't want my shell script to explode with an error message from a random failed command unless it is something of extreme importance to me; checking for the process exit code and acting upon it accordingly is sufficient and is good enough. Whether such an approach is a good thing or not is a matter of a debate. Global lists of errors and/or error messages require an official register of both diligently maintained and updates of which to be centrally coordinated, which I don't think could work with the open source. Other operating systems have attempted to mandate error codes with complex structures and a well defined (and sometimes written by a professional technical writer!) error message. Yet, they have had limited success. Yes, OS/400 running on an AS/400 could inspect a failed process' error code and automatically dispatch a message to an IBM service centre to order a spare part for a specific failing piece of hardware whose SKU would have been deduced from a specic part of the error code without requiring the human intervention, but that is somewhat of an extreme and extravagant example and is certainly not mainstream.* EDIT: «make -j12» vs «make -j8» explanation.

danmur 5y ago

Oh my god, your first point. I had to pass arbitrary arguments through a couple layers of shell and I couldn't do it. Gave up and rewrote it all in Python, even though it's annoying in other ways, and I'm decent at shell scripting. It's a nightmare.

jiggawatts 5y ago

Most of those features are in PowerShell, especially version 7. It uses strict ('reliable') argument passing, strong typing, etc.. It has a bazillion independent streams such as "Debug", "Verbose", "Warning", "Error", "Progress", "Output", "Information", etc... It has "foreach -parallel", which is a fun way to make the CPU fan spin up. > "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. $lines = dir -Recurse -Exclude *.git, *.vim | Get-Content Not exactly rocket science in most shells? In most shells you can make this a modular function that takes in a list of file names, but PowerShell's "dir" doesn't output file names. It outputs file objects, so then it is just one step to do a filter on something like extension (or length, or whatever): dir | ? Length -lt 1MB | ? Extension -in @( '.cs', '.rs', '.cpp' ) In PowerShell you can create functions that integrate with the object-oriented pipeline. You can pipe full objects into a function, and objects out. Object attributes can be mapped automatically to function arguments too. You can have begin/process/end blocks so you can do things like open a DB connection, pipe in data, then close the connection neatly without needing to do this all manually each time. Literally something as trivial as this: Get-Stuff | Out-DatabaseTable -TableName 'stuff_log' -Server '...' Seriously. Give it a go. It works on Linux too, and it's more unix than unix.

dale_glass 5y ago
PowerShell is indeed excellent, but there's nothing wrong with it having a bit of competition. Some of it is also less than ideal, but rather less than bash. Issues in powershell I can think of: * It leaks environment variables. Set an env var, and it propagates to the shell from which it was called from! * It can't comfortably import environment from batch files. Working with vcvars is an annoyance. You'd think somebody at Microsoft would have made built in support for that one. * Argument passing in Windows is a bloody horror. PowerShell accepts commands encoded in base64 because it sucks so much! * For some reason you can't quote the command itself. You can do: git checkout "blah" But for some reason can't do: "C:\Program Files\Git\bin\git.exe" checkout "blah". Granted, that can be worked around with & or Start-Process, but what a pointless annoyance. That said, I've not really tried it on Linux yet, got to give it a go.

paskozdilar 5y ago

Bash is definitely painful to write - I write shell scripts a lot and I enjoy it (in a masochistic kind of way), and I wholeheartedly agree with most of the criticisms you've written. However, the greatest strength of Bash is its ubiquity - it is available on almost every modern Unix-like environment - and when combined with standard POSIX tools it can provide a-little-less-painful environment for writing reliable shell scripts. In order to prove it, I will do my best to provide some kind of solution to each of the problems you have mentioned. Note that they are in no way perfect solutions, and all of them are just hacks that work around Bash's inherent clumsiness. --- > 100% reliable argument passing Wrapping a variable in double-quotes does this: var="1 2 3 4" for x in "$var"; do echo $x; done For double-parsing (e.g. SSH commands) you can wrap the entire expression in single-quotes to maintain the original string until the second shell: ssh user@host 'var="1 2 3 4"; for x in "$var"; do echo $x; done' > 100% reliable file iteration The "find" command does exactly this, and it works for whatever weird characters in the filename you have. find ./directory/ -type f -exec SOME_COMMAND {} \; > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. Do not evaluate arguments directly in shell, use xargs to feed the arguments as standard input. E.g.: find ./directory/ -type f | xargs SOME_COMMAND > Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B". dirname /foo/bar basename /foo/bar realpath ./symlink/to/foo/bar basename /foo/bar | sed 's/.*\.//' realpath --relative-to ./dirA/fileA ./dirB/fileB > Good error handling and reporting This is a tough one. Good error handling/reporting in Bash is extremely hard and requires discipline, but it's possible. To exit immediately as soon as a command exits with non-zero exitcode: set -e However, this does not play nice with temporary resources, so I personally prefer hand-written error checks: SOME_COMMAND || { echo "Some Command has failed" && exit 1; } Error reporting depends on a scenario, but I generally have a pattern of doing checks like: if [ "$var" != "expected" ] then echo "Error: var different from expected" && exit 1 fi And it works fine most of the time. > Easy capture of stdout and stderr, at the same time. Either together or individually, as needed. To capture both stdout and stderr in a same stream: SOME_COMMAND |& SOME_FILTER Capturing stdout and stderr in different streams is tough. The easiest way I can think of is to use a pipe and job control mechanisms: mkfifo pipe SOME_COMMAND 2>pipe | FILTER_1 & < pipe FILTER_2 & wait rm pipe * Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available, while being able to report progress, record each failure, and abort the entire thing if needed. While bash is truly terrible when it comes to concurrency, I find GNU Parallel to be pretty satisfying for most concurrent shell-scripting: cat FILE.txt | parallel --timeout=30 SOME_COMMAND {} \|\| echo "Some Command failed with argument: {}" For POSIX purity, xargs also can be used with "-L 1" argument that parses a single line per command iteration. For parallelism, there is also a "--max-procs" argument. * Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". I find that adding "set -x" to the top of your shell, which prints each command with "+" prefix as it is expanded and executed, very useful for error tracking. * Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. There are helpers out there for most common situations - it's just that they are implemented as CLI tools, and not officially part of the shell. And in the scenario where you can't expect the availability of those unofficial tools, you can always write your own small library of commonly used Bash functions, and just copy-paste them into your script whenever you need them. It's ugly, but still possible.

darrenf 5y ago

In my (biased) opinion Perl comes close to ideal with judicious use of Path::Tiny and Capture::Tiny.

hnlmorg 5y ago

Shameless plug, but the problems you've described are exactly the problems I was looking to solve with my own shell (https://murex.rocks https://murex.rocks / https://github.com/lmorg/murex https://github.com/lmorg/murex) > 100% reliable argument passing. That is, when I run `system("git", "clone", $url);` in Perl, I know with exact precision what arguments Git is going to get, and that no matter what weirdness $url contains, it'll be passed down as a single argument. Heck, make that mandatory. Variables are parsed as tokens so they're passed to the parameters whole, regardless of whether they contain white space or not. So you can still use the Bash terseness of parameters (eg `git clone $url`) but $url works in the same way as `system("git", "clone", $url);` in Perl. > 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. Murex is inspired by Perl in that $ is a scalar and @ is an array. So if you use `f +f` builtin (to return a list of files), it returns as a JSON array. From there you can use @ to expand that array with each value being a new parameter, eg rm -v @{ f +f } # delete all files or use $ to pass the entire array as a JSON string. Eg echo ${ f +f } # you could just run `f +f` without `echo` to get the list. # This is just a contrived example of passing the array as a string. Additionally there are lots of tools that are natively aware of arrays and will operate on them. Much like how `sed`, `grep`, `sort` et al treat documents as lists, murex will support lists as JSON arrays. Or arrays in YAML, TOML, S-expressions, etc. So you can have code that looks like this: f +f | foreach file { echo $file } > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. This is a kernel limit. I don't think there is any way to overcome this without using iteration instead (and suffering the performance impact form that). > Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B". There are a number of ways to query files and paths in murex: - f: This returns files based on meta data. So will pull files, or directories, or symlinks, etc depending on the flags you pass. eg `+f` would include files. `+d` would be include directories. or `-s` would exclude symlinks. - g: Globbing. Basically `*` and `?`. This can run as a function rather than being auto-expanded. eg `g *.txt` or `rm -v @{ g *.txt }` - rx: Like globbing but using regexp. eg `rx '\.txt$'` or `rm -v @{ rx '\.txt$' }` - this example looks terrible but using rx does sometimes come in handy if you have more complex patterns than a standard glob could support. eg `rm -v @{rx '\.(txt|rtf|md|doc|docx)$'}` The interactive shell also have an fzf like integration built in. So you can hit ctrl+f and then type a regexp pattern to filter the results. This means if you need to navigate through complex source tree (for example) to a specific file (eg ./src/modules/example/main.c) you could just type `vi ^fex.*main` and you'd automatically filter to that result. Or even just type `main` and only see the files in that tree with `main` in their name. > Good error handling and reporting / Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". A lot of work here. - `try` / `trypipe` blocks supported - `if` and `while` blocks check the exit code. So you can do if { which foobar} else { err "foobar does not exist!" } # 'then' and 'else' are optional keywords for readability in scripting. So a one liner could read: # !if {which foobar} {err foobar does not exist!} - built in support for unit tests - built in support for watches (IDE feature where you can watch the state of a variable) - unset variables error by default - empty arrays will error by default when passed as parameters in the next release (hopefully coming this week) - STDERR is highlighted red by default (can be disabled if that's not to your tastes) so you can clearly see any errors if they're muddled inside STDOUT. - non zero exit numbers automatically raise an error. All errors return a line and cell number so you can find the exact source of the error in any scripts Plus lots of other stuff that's referenced in the documents > Easy capture of stdout and stderr, at the same time. Either together or individually, as needed. Murex handles named pipes a little differently, they're passed as parameters inside triangle brackets, <>. STDERR is referenced with a proceeding exclamation mark. eg a normal command will appear internally as: command1 <out> <!err> parameter1 parameter2 etc | command2 <out> <!err> parameter1 parameter2 etc (You don't need to specify <out> nor <!err> for normal operation). So if you want to send STDERR off somewhere for later processing you could create a new named pipe. eg pipe example # creates a new pipe called "example" command1 <!example> parameter1 parameter2 | command2 <!null> parameter1 parameter2 This would say "capture the STDERR of the first command1 and send it to a new pipe, but dump the STDERR of command2". You can then later query the named pipe: <example> | grep "error message" > Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable. This is where the typed pipelines of murex come into their own. It's like using `jq` but builtin the shell itself and works transparently with multiple different document types. There's also an extensive library of builtins for common problems, eg `jsplit` will read STDIN and output an array split based on a regexp pattern. So your example would be: cat file | jsplit \n > That's the kind of thing I care about for shell scripting. A better syntax is nice, but actually getting stuff done without having to work around gotchas and issues is what I'm looking for. I completely agree. I expect this shell I've created to be pretty niche and not to everyone's tastes. But I'd written it because I wanted to be a more productive sysadmin ~10 years ago and since I've moved into DevOps I've found it invaluable. It's been my primary shell for around 5 years and every time I run into a situation where I'm like "I wish I could do this easily" I just add it. The fact that it's a typed pipeline makes it really easy to add context aware features, like SQL query support against CSV files.

inkyoto 5y ago

> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters. In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters. ------ > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents: command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»). How the length limit works is explained at length (please pardon the pun) here: https://www.in-ulm.de/~mascheck/various/argmax/ https://www.in-ulm.de/~mascheck/various/argmax/ ------ > Excellent path parsing. […] finding the file extension […] File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1). Moreover, any valid character can be used as a separator and its interpretation is either left out (almost always), or the interpretation is left up to the semantically aware app. One is free to use a comma or the Javanese wasana pada as part of the extension; that is, «file,exec» and «file꧅ ꦆ ꧅exec» are both valid and both have the extension of «exec» – as long as the file system supports the appropriate character set. This is also why «basename», to work correctly, requires the separator as part of the imaginary «extension», i.e. «basename myfile,exec ,exec» will always give «myfile» as the result. DOS style extensions are a made up convention that neither the kernel, nor the shell, nor file processing utilities enforce as they are file extension unaware. It is better to think of UNIX file names as being made up a prefix and an optional, arbitrary length suffix (but not an extension). ------ > Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available With respect to «every core available». You almost certainly don't want that and should overprovision the number of running processes compared to the number of cores on your system. UNIX has supported multiprocessor systems for a very long time. Despite massive improvements in the hardware performance, disks and networks are still the slowest moving (or still) parts. They were even slower when UNIX was in its relative infancy when the CPU time was also very expensive. Therefore, the CPU time had to utilised efficiently whilst waiting for a disk to return a string of bytes. File processing tasks spend their time between: 1) waiting for I/O (in the blocked state) and 2) actually processing (the running state). Since the disk is still very slow compared to the speed of a modern CPU, the UNIX process scheduler blocks the process until I/O completes and checks the process run queue in the kernel to see if there is another process ready to compute something (i.e. in the running state). This inherent interleaving of «blocked for I/O» and «running» process states can be used to an advantage depending on a few factors. The less is size of the unit of data an app processes and the larger is the total size of the input (i.e. the input and output files), the more time the app spends in the «blocked for I/O» state, and most of the CPU time is simply wasted unless there is something else to do. But if we know such specifics (the size of the unit of work and the size of the input), we can overprovision the number of processes thereby utilising the CPU compute time more efficiently whilst the disk controller is transferring bytes into the memory via the direct memory access (I am oversimplifying a few bits here). This is the reason why «make -j12» will compile almost always faster than «make -j8» on a 8 CPU core system on projects with a large number of small(-er) files – because of the I/O overhead. Whilst there is no universal formula, 1.5x process overprovisioning factor is a decent starting point. For simpler daily file processing tasks GNU parallel is good enough to spare oneself of headaches of such computations, though. Therefore, the «We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available […]» does not make sense in the context of UNIX shell languages and the process scheduling in UNIX. In fact, you will underutilise your 128 CPU cores, sometimes pretty heavily, unless you correctly account for the I/O factor. Only if the app/process is aware of how to efficiently parallelise its own workload (because it knows its workload better than anyone else), then and only then it does need to know how many cores there are available at its disposal. No scripting language / shell can solve this problem. ------ > Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere". The excellent error reporting has existed in UNIX since day 1. It is called the process status code; anything that is not a zero status code indicates an error. The semantic interpretation of each speific numeric status code, however, is entirely decoupled from the thing that might fail and is documented in the man page. The status code of 3 in «mv» and in «git» will mean two completely different things, therefore the specific status code processing is localised to the process invocation point in shell scripts. Most of the time, though, I personally don't want my shell script to explode with an error message from a random failed command unless it is something of extreme importance to me; checking for the process exit code and acting upon it accordingly is sufficient and is good enough. Whether such an approach is a good thing or not is a matter of a debate. Global lists of errors and/or error messages require an official register of both diligently maintained and updates of which to be centrally coordinated, which I don't think could work with the open source. Other operating systems have attempted to mandate error codes with complex structures and a well defined (and sometimes written by a professional technical writer!) error message. Yet, they have had limited success. Yes, OS/400 running on an AS/400 could inspect a failed process' error code and automatically dispatch a message to an IBM service centre to order a spare part for a specific failing piece of hardware whose SKU would have been deduced from a specic part of the error code without requiring the human intervention, but that is somewhat of an extreme and extravagant example and is certainly not mainstream.* EDIT: «make -j12» vs «make -j8» explanation.