4 ms·

Unless you can carefully code, it's not worth pretending a script to be idempotent. The example checks for /mnt/dev in a file but it doesn't check whether the

Unless you can carefully code, it's not worth pretending a script to be idempotent. The example checks for /mnt/dev in a file but it doesn't check whether the string is in a comment or if it's part of /mnt/develop, and pretending you've achieved idempotency is dangerous unless you've gone through the effort of checking every corner case. It's easier and safer to pretend it's not idempotent in the first place.

VWWHFSfQ 5y ago
This whole thing is also loaded with TOCTOU problems. It's almost always best just to try to do what you want to do and check for failure.
- zymhan 5y ago
  I had to look up TOCTOU https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use Very interesting!
  - User23 5y ago
    That link also describes the Catholic schoolboy algorithm: ask forgiveness not permission.
- pas 5y ago
  Orchestration/install/setup scripts are usually run while nothing else is running. Bash is not ideal for this. But it's not ideal for almost everything. Still making things idempotent-ish is better than bailing on every recoverable error and making the admin do the cleanup manually so the script can finally do what it supposed to do. (Of course a few years ago immutable infrastructure was the rage, because it means if the script runs once, pack it up as a VM and you're done. And that's how the docker sausage images are made.)
dzhiurgis 5y ago
Is there a tool that treats txt files as a Set? Been looking for one a while a go and it feels like it’s something sound be built in.
- SavantIdiot 5y ago
  In bash? How would you even implement a set in bash without just doing linear greps? Or did someone add sets to bash 20 years ago and I never got the memo?
  - adrianmonk 5y ago
    You could use the 'look' command, which does a binary search. It's basically meant to look up spellings in /usr/share/dict/words, but it can work on any file. It will match any line that your pattern is a prefix of, so you'd have to add logic to eliminate longer matches. But if you had some huge file to search and you wanted to do it from a shell script, that would be one way. Caveat: although it's fairly standard, 'look' might not be installed on every system. Also, you have to be sure to maintain your file in sorted order. So no adding things by appending to the end, and checking if something is in the set is much quicker at the expense of adding things being much slower.
- sa46 5y ago
  You can use a combination of `comm` and `uniq` to implement set intersection and union. https://ss64.com/bash/comm.html https://ss64.com/bash/comm.html > Return the unique lines in the file words.txt that don't exist in countries.txt comm -23 <(sort words.txt | uniq) <(sort countries.txt | uniq) >Return the lines that are in both words.txt and countries.txt: comm -12 <(sort words.txt | uniq) <(sort countries.txt | uniq)
- kmstout 5y ago
  I wrote these years ago. They're damn handy. It's true that they're not implemented in Bash (that would be nuts), but having them on hand lets me do much more on the command line than would otherwise be possible. ~/bin/union =========== #! /usr/bin/awk -f !acc[$0]++ ~/bin/intersection ================== #! /usr/bin/awk -f !buf[$0]++ {acc[$0] += 1} ENDFILE { delete buf; files++ } END { for (k in acc) if (acc[k] == files) print k } ~/bin/set-diff ============== #! /usr/bin/awk -f ! filenum { acc[$0] = 1 } filenum { delete acc[$0] } ENDFILE { filenum++ } END { for (k in acc) print k }
SkittyDog 5y ago
I agree with you. Bash is a wonderful tool, but it's not very well designed for sophisticated control flow. Gotta know when to say when, and switch to something else.
- gigatexal 5y ago
  Do-one-thing-and-only-one-thing-well bash scripts coupled with a Makefile gives one “sophisticated control flow”.
User23 5y ago
When you’re paying six figures careful coding isn’t too much to ask.
AceJohnny2 5y ago
> Unless you can carefully code, it's not worth pretending a script to be idempotent. Just because you can't reach perfection doesn't mean it's useless to strive towards it. Idempotency is a very useful property that more shell script writers should be made aware of, even if they only learn basic things like "mkdir -p" or "rf -f SOMEFILE"

VWWHFSfQ 5y ago

This whole thing is also loaded with TOCTOU problems. It's almost always best just to try to do what you want to do and check for failure.

zymhan 5y ago
I had to look up TOCTOU https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use Very interesting!
- User23 5y ago
  That link also describes the Catholic schoolboy algorithm: ask forgiveness not permission.
pas 5y ago
Orchestration/install/setup scripts are usually run while nothing else is running. Bash is not ideal for this. But it's not ideal for almost everything. Still making things idempotent-ish is better than bailing on every recoverable error and making the admin do the cleanup manually so the script can finally do what it supposed to do. (Of course a few years ago immutable infrastructure was the rage, because it means if the script runs once, pack it up as a VM and you're done. And that's how the docker sausage images are made.)

dzhiurgis 5y ago

Is there a tool that treats txt files as a Set? Been looking for one a while a go and it feels like it’s something sound be built in.

SavantIdiot 5y ago
In bash? How would you even implement a set in bash without just doing linear greps? Or did someone add sets to bash 20 years ago and I never got the memo?
- adrianmonk 5y ago
  You could use the 'look' command, which does a binary search. It's basically meant to look up spellings in /usr/share/dict/words, but it can work on any file. It will match any line that your pattern is a prefix of, so you'd have to add logic to eliminate longer matches. But if you had some huge file to search and you wanted to do it from a shell script, that would be one way. Caveat: although it's fairly standard, 'look' might not be installed on every system. Also, you have to be sure to maintain your file in sorted order. So no adding things by appending to the end, and checking if something is in the set is much quicker at the expense of adding things being much slower.
sa46 5y ago
You can use a combination of `comm` and `uniq` to implement set intersection and union. https://ss64.com/bash/comm.html https://ss64.com/bash/comm.html > Return the unique lines in the file words.txt that don't exist in countries.txt comm -23 <(sort words.txt | uniq) <(sort countries.txt | uniq) >Return the lines that are in both words.txt and countries.txt: comm -12 <(sort words.txt | uniq) <(sort countries.txt | uniq)
kmstout 5y ago
I wrote these years ago. They're damn handy. It's true that they're not implemented in Bash (that would be nuts), but having them on hand lets me do much more on the command line than would otherwise be possible. ~/bin/union =========== #! /usr/bin/awk -f !acc[$0]++ ~/bin/intersection ================== #! /usr/bin/awk -f !buf[$0]++ {acc[$0] += 1} ENDFILE { delete buf; files++ } END { for (k in acc) if (acc[k] == files) print k } ~/bin/set-diff ============== #! /usr/bin/awk -f ! filenum { acc[$0] = 1 } filenum { delete acc[$0] } ENDFILE { filenum++ } END { for (k in acc) print k }

SkittyDog 5y ago

I agree with you. Bash is a wonderful tool, but it's not very well designed for sophisticated control flow. Gotta know when to say when, and switch to something else.

gigatexal 5y ago
Do-one-thing-and-only-one-thing-well bash scripts coupled with a Makefile gives one “sophisticated control flow”.

User23 5y ago

When you’re paying six figures careful coding isn’t too much to ask.

AceJohnny2 5y ago

> Unless you can carefully code, it's not worth pretending a script to be idempotent. Just because you can't reach perfection doesn't mean it's useless to strive towards it. Idempotency is a very useful property that more shell script writers should be made aware of, even if they only learn basic things like "mkdir -p" or "rf -f SOMEFILE"