3 ms·

> - Do not 'set -o pipefail' by default. It will only waste your time when your scripts start failing because some command in the middle of a pipe returned non-

by nikau 4y ago

> - Do not 'set -o pipefail' by default. It will only waste your time when your scripts start failing because some command in the middle of a pipe returned non-zero while still outputting the result you wanted, and you spend 3 hours adding debugging to figure out where the error was coming from and write some extra error handling code. Just check if the result of the pipe looks accurate. Same result, fewer failures. Exactly how do you "check if the result of the pipe looks accurate" without having to get a human to check the log file? The whole point of automation is to get rid of human interaction even if it requires some upfront work to implement error handling where appropriate...

0xbadcafebee 4y ago
Well it's context-dependent, but there are many different strategies. Most well designed command-line programs output to STDERR if they have an error, and STDOUT if they don't. So the simplest possible method is just to see if STDOUT was 0-length or not. If it's not 0-length, then assume the command completed successfully. grep can help validate the output. Depending on the command, it may resist partial failures; for example, many commands that output a JSON blob will wait until they have all their data before outputting the blob. If the script you're writing depends on always working 100%, then you can do a lot more work to check for errors. First you would enable set +e, because if you exit on error with pipefail, you can't even check what the error was from PIPE_STATUS. Instead you can inspect every pipe result and tell your user what part failed, and decide if you should continue, print only potentially partial results, retry the operation, or fail immediately. Most programmers don't handle error conditions properly and just immediately fail on any error, which itself causes errors which needs a human to inspect a log. Worst case, the shell script without pipefail just keeps working, when with pipefail it would have failed a lot.

by nikau 4y ago

0xbadcafebee 4y ago

Well it's context-dependent, but there are many different strategies. Most well designed command-line programs output to STDERR if they have an error, and STDOUT if they don't. So the simplest possible method is just to see if STDOUT was 0-length or not. If it's not 0-length, then assume the command completed successfully. grep can help validate the output. Depending on the command, it may resist partial failures; for example, many commands that output a JSON blob will wait until they have all their data before outputting the blob. If the script you're writing depends on always working 100%, then you can do a lot more work to check for errors. First you would enable set +e, because if you exit on error with pipefail, you can't even check what the error was from PIPE_STATUS. Instead you can inspect every pipe result and tell your user what part failed, and decide if you should continue, print only potentially partial results, retry the operation, or fail immediately. Most programmers don't handle error conditions properly and just immediately fail on any error, which itself causes errors which needs a human to inspect a log. Worst case, the shell script without pipefail just keeps working, when with pipefail it would have failed a lot.