5 ms·

We had some problem, roughly, "run a tool on all YAML files in the directory" or something. The first attempt that led to a production outage: (It is late, bas

We had some problem, roughly, "run a tool on all YAML files in the directory" or something. The first attempt that led to a production outage: (It is late, bash examples are approximate. Don't sweat the syntax.) the_tool *.yaml There weren't any YAML files, so argv[1] was literally "*.yaml", which then wasn't found, leading to errors, etc. shopt -s nullglob the_tool *.yaml (Don't even get me started on how there's shopt -s and set -o and WTF.) There weren't any YAML files, so argc was == 0, which the program considered an error. Mea culpea, perhaps. AN_ARRAY=(*.yaml) if [[ < length of the array is not 0 > ]]; then the_tool "${AN_ARRAY[@]}" fi This failed, b/c bash can't represent an empty array! (And we run -u, as one must, if one wants reasonable behavior.) So finally the contorted, AN_ARRAY=(*.yaml) if [[ "${AN_ARRAY+x}" = x && < length is not 0 > ]]; then the_tool "${AN_ARRAY[@]}" fi To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering. Use it on the command line in your day to day, fine; but if it's getting automated, jump to at least Python.

Izkata 5y ago
My preferred way under those conditions would be more along the lines of "see if a safe command fails, do the desired command if not", like: if ls *.yaml > /dev/null; then the_tool *.yaml fi
- xorcist 5y ago
  That only looks simple because we don't know what problem was being solved. Is it really right to not process any files just because one of them just disappeared? In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
  - Izkata 5y ago
    > Is it really right to not process any files just because one of them just disappeared? No idea what you mean here, the glob can expand to one or many. > In general, situations like this suffer from race conditions and time-of-check/time-of-use problems. I mean, that's a problem in every version presented so far, one that I don't think can be fixed externally without something extra like hardlinking to a new directory, because "the_tool" apparently can't handle missing files and they can go missing when "the_tool" starts up. That's why I didn't even bother.
yjftsjthsd-h 5y ago
On the contrary, I am 100% open to suggestions that BASH sucks. (Personally, I would have picked the inadequacies of `set -e` as my favorite ... unfortunate aspect.) My objection was very narrow: this specific article does a poor job at saying why shell sucks, and a worse job still at presenting Python as a superior alternative. That's not to say that good reasons to use something else don't exist, and it's not even to say that Python can't be that alternative. (Your sibling comments suggesting xonsh and https://amoffat.github.io/sh/ https://amoffat.github.io/sh/ paint a fairly compelling way to actually move to python without losing the advantages of shell, for instance.)
Arnavion 5y ago
find . -maxdepth 1 -name '*.yaml' -print0 | xargs -r0 the_tool >To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering. If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad.
- raffraffraff 5y ago
  Also, how much code would it take to do the same thing in python?
  - kleene_op 5y ago
    Probably something along the line of Popen(["the_tool"] + list(Path().glob("*.yml")))
    - deleted 5y ago
      [deleted]
    - mannykannot 5y ago
      This is reasonable in general, though in this case, I think this will replicate the problem of the second Bash version, as the problem there is with the_tool requiring at least one argument. While dethanatos's point about Bash's default behavior with a non-matching glob is on target, the remainder of the example demonstrates a problem in the_tool, which will need to be worked around regardless of how you invoke it.
- rascul 5y ago
  > If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad. find has -exec, xargs isn't necessary.
theteapot 5y ago
Didn't know "Fractal of Bad Design" even existed. I like aspects of PHP. Sorry.
theteapot 5y ago
What happens if someone runs `the_tool this-file-does-not-exist.yaml` or the yaml file is removed by some other means before you get a chance to process it? Does your world also come tumbling down in these cases? If yes, please stop blaming Bash for it.
zaphirplane 5y ago
This isn’t you are holding it wrong It is in the “do as I want not as I say” category
1vuio0pswjnm7 5y ago
Personally, I never use bash for scripting. I use the system's scripting shell, Debian's Dash or NetBSD Ash. Rhetorical question: If Bash was great for shell scripting, why doesn't the OS project choose it for scripting. When I compile the OS kernel and userland, the project (NetBSD) uses shell scripts not Python. Not suggesting the shell is necessarily better but I am suggesting it works and people understand it. There are many ways to run a command on every .yaml file in a directory. Personally I like using xtrace. Something like: echo cd the_directory > 1.sh echo "test -f *.yaml" >> 1.sh ls -1 *.yaml|sed 's/^/the_tool /' >> 1.sh chmod +x 1.sh less 1.sh sh -ex 1.sh As mostly a day-to-day shell user who automates some things, I do not use Python. Never had an outage.
DyslexicAtheist 5y ago
> To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering. is this a cloud AMD64 opinion that ignores the 2 trillion devices which Gartner predicts will never run python? I recently finished a ~600 lines ash script and the kids at work which just graduated complained that *"Y u no python?" For them it was difficult to understand because apparently I avoided cat/grep/dirname/basename/etc and used shell syntax wherever possible to avoid shelling out. (the thing I did would only run when system reaches a certain load) Everyone forgot that the target hardware has only busybox (ash) and was massively resource constrained. My only options were to implement what I was doing as an eBPF, in C, or shell (ash). The shell script took 4 days (and was refactored/optimized several times within that period), the C (or eBPF) program would have taken ~10 days and the eBPF which is the best solution is unmaintainable once I leave the company. from a performance perspective I take a bash script any time. Also python programs over time have a risk of eventually pulling in a dozen dependencies.
- rhn_mk1 5y ago
  Judging by the reaction of the kids at work, what you did is already unmaintainable. Plus, if you are running shell out of necessity on a constrained system, that doesn't mean shell deserves to be a programming language where there's actual choice.
  - DyslexicAtheist 5y ago
    > what you did is already unmaintainable. how so? there is no python or perl or ruby etc on the target. what would have been a maintainable option other than /bin/sh ? C/C++ ? > where there's actual choice. again what choice? :) as I said there is no other solution than what I've given. the system literally doesn't have anything else regardless what "I think" it should support.
  - psd1 5y ago
    Anything is unmaintainable if the maintainers don't have the foundational skills.
xorcist 5y ago
This tool of yours can not cope with files not existing, and causes production outages? Yet somehow you manage to blame bash for this? The outcome would not have been different if systemd or python started your tool. In fact, handling that types of errors is what bash is suitable for. "If file is readable, process it" is not an unusual construct. Please do take care however to avoid races if this directory is a spool directory that can be populated at any time, in which case it is useful that rename() is atomic, sometimes an additional working directory is required. Again, this is how the operating system is designed and nothing the shell can gloss over. I have seen many problems from previously unknown race conditions when concurrency is increased. Scripts have to be kept composable and trivial to understand in order to guard against that. To be honest, I would argue to remove any script that looks like the last example above. It is not clear what side effects of that bizarre array construct are useful to the system. Keep scripts as simple and readable as possible, a trivial truth perhaps, applicable to every language, but one that needs to be reiterated until the end of our profession.
sfoley 5y ago
> This failed, b/c bash can't represent an empty array! It absolutely can, I have no idea where you’re getting this idea.

Izkata 5y ago

My preferred way under those conditions would be more along the lines of "see if a safe command fails, do the desired command if not", like: if ls *.yaml > /dev/null; then the_tool *.yaml fi

xorcist 5y ago
That only looks simple because we don't know what problem was being solved. Is it really right to not process any files just because one of them just disappeared? In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
- Izkata 5y ago
  > Is it really right to not process any files just because one of them just disappeared? No idea what you mean here, the glob can expand to one or many. > In general, situations like this suffer from race conditions and time-of-check/time-of-use problems. I mean, that's a problem in every version presented so far, one that I don't think can be fixed externally without something extra like hardlinking to a new directory, because "the_tool" apparently can't handle missing files and they can go missing when "the_tool" starts up. That's why I didn't even bother.

yjftsjthsd-h 5y ago

On the contrary, I am 100% open to suggestions that BASH sucks. (Personally, I would have picked the inadequacies of `set -e` as my favorite ... unfortunate aspect.) My objection was very narrow: this specific article does a poor job at saying why shell sucks, and a worse job still at presenting Python as a superior alternative. That's not to say that good reasons to use something else don't exist, and it's not even to say that Python can't be that alternative. (Your sibling comments suggesting xonsh and https://amoffat.github.io/sh/ https://amoffat.github.io/sh/ paint a fairly compelling way to actually move to python without losing the advantages of shell, for instance.)

Arnavion 5y ago

find . -maxdepth 1 -name '*.yaml' -print0 | xargs -r0 the_tool >To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering. If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad.

raffraffraff 5y ago
Also, how much code would it take to do the same thing in python?
- kleene_op 5y ago
  Probably something along the line of Popen(["the_tool"] + list(Path().glob("*.yml")))
  - deleted 5y ago
    [deleted]
  - mannykannot 5y ago
    This is reasonable in general, though in this case, I think this will replicate the problem of the second Bash version, as the problem there is with the_tool requiring at least one argument. While dethanatos's point about Bash's default behavior with a non-matching glob is on target, the remainder of the example demonstrates a problem in the_tool, which will need to be worked around regardless of how you invoke it.
rascul 5y ago
> If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad. find has -exec, xargs isn't necessary.

theteapot 5y ago

Didn't know "Fractal of Bad Design" even existed. I like aspects of PHP. Sorry.

theteapot 5y ago

What happens if someone runs `the_tool this-file-does-not-exist.yaml` or the yaml file is removed by some other means before you get a chance to process it? Does your world also come tumbling down in these cases? If yes, please stop blaming Bash for it.

zaphirplane 5y ago

This isn’t you are holding it wrong It is in the “do as I want not as I say” category

1vuio0pswjnm7 5y ago

Personally, I never use bash for scripting. I use the system's scripting shell, Debian's Dash or NetBSD Ash. Rhetorical question: If Bash was great for shell scripting, why doesn't the OS project choose it for scripting. When I compile the OS kernel and userland, the project (NetBSD) uses shell scripts not Python. Not suggesting the shell is necessarily better but I am suggesting it works and people understand it. There are many ways to run a command on every .yaml file in a directory. Personally I like using xtrace. Something like: echo cd the_directory > 1.sh echo "test -f *.yaml" >> 1.sh ls -1 *.yaml|sed 's/^/the_tool /' >> 1.sh chmod +x 1.sh less 1.sh sh -ex 1.sh As mostly a day-to-day shell user who automates some things, I do not use Python. Never had an outage.

DyslexicAtheist 5y ago

> To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering. is this a cloud AMD64 opinion that ignores the 2 trillion devices which Gartner predicts will never run python? I recently finished a ~600 lines ash script and the kids at work which just graduated complained that *"Y u no python?" For them it was difficult to understand because apparently I avoided cat/grep/dirname/basename/etc and used shell syntax wherever possible to avoid shelling out. (the thing I did would only run when system reaches a certain load) Everyone forgot that the target hardware has only busybox (ash) and was massively resource constrained. My only options were to implement what I was doing as an eBPF, in C, or shell (ash). The shell script took 4 days (and was refactored/optimized several times within that period), the C (or eBPF) program would have taken ~10 days and the eBPF which is the best solution is unmaintainable once I leave the company. from a performance perspective I take a bash script any time. Also python programs over time have a risk of eventually pulling in a dozen dependencies.

rhn_mk1 5y ago
Judging by the reaction of the kids at work, what you did is already unmaintainable. Plus, if you are running shell out of necessity on a constrained system, that doesn't mean shell deserves to be a programming language where there's actual choice.
- DyslexicAtheist 5y ago
  > what you did is already unmaintainable. how so? there is no python or perl or ruby etc on the target. what would have been a maintainable option other than /bin/sh ? C/C++ ? > where there's actual choice. again what choice? :) as I said there is no other solution than what I've given. the system literally doesn't have anything else regardless what "I think" it should support.
- psd1 5y ago
  Anything is unmaintainable if the maintainers don't have the foundational skills.

xorcist 5y ago

This tool of yours can not cope with files not existing, and causes production outages? Yet somehow you manage to blame bash for this? The outcome would not have been different if systemd or python started your tool. In fact, handling that types of errors is what bash is suitable for. "If file is readable, process it" is not an unusual construct. Please do take care however to avoid races if this directory is a spool directory that can be populated at any time, in which case it is useful that rename() is atomic, sometimes an additional working directory is required. Again, this is how the operating system is designed and nothing the shell can gloss over. I have seen many problems from previously unknown race conditions when concurrency is increased. Scripts have to be kept composable and trivial to understand in order to guard against that. To be honest, I would argue to remove any script that looks like the last example above. It is not clear what side effects of that bizarre array construct are useful to the system. Keep scripts as simple and readable as possible, a trivial truth perhaps, applicable to every language, but one that needs to be reiterated until the end of our profession.

sfoley 5y ago

> This failed, b/c bash can't represent an empty array! It absolutely can, I have no idea where you’re getting this idea.