5 ms·

Bash retry function with exponential backoff

121 points by geocrasher 4y ago

sillysaurusx 4y ago
I was today years old when I learned that bash can do simple arithmetic with $(($i + 1)). Thanks!
- mmh0000 4y ago
  If you just want to add/subtract 1, in bash you can do just '((i++))', i.e.: [mmh@x670]$ i=0 [mmh@x670]$ ((i++)); echo $i 1 [mmh@x670]$ ((i++)); echo $i 2 Beware that the above is very -bashy- and purists will rip your head off for using it.
  - sillysaurusx 4y ago
    Whoa. No dollar signs at all? I’m at Lowes — does ((i+1)) work? Or is it just for incrementing / decrementing (which is still very useful!) EDIT: thinking this over, does this work for variable substitution too? E.g. ((i)) being equivalent to $i. Not that you’d necessarily want to…
    - mmh0000 4y ago
      To add arbitrary numbers, you'd have to do an assignment, like this: [mmh@x670]$ ((i=i+2)); echo $i 6 [mmh@x670]$ ((i=i+2)); echo $i 8
    - Arnavion 4y ago
      `(( ))` is an arithmetic evaluation block. Its content has to be an arithmetic expression. Arithmetic expressions don't require `$` before simple variable names and some more complex expressions like array indexing. `(( i + 1 ))` will evaluate the result of adding one to `$i` and then throw away the result. It doesn't do anything useful (other than having a different exit code depending on whether the expression evaluated to 0 or not). `$(( ... ))` evaluates the expression and then returns its value. ie `i=$(( i + 1 ))` will increment `$i`, just like `(( i++ ))` did.
  - mmh0000 4y ago
    Also, the wooledge wiki is an amazing resource for Bash/shell scripting. To learn more magic see: https://mywiki.wooledge.org/ArithmeticExpression https://mywiki.wooledge.org/ArithmeticExpression
  - cmckn 4y ago
    There is a gotcha here if you use: set -o errexit Because *any bash arithmetic expression that evaluates to zero returns the exit code of 1*. So this works: #!/usr/bin/env bash set -o errexit i=0 ((++i)) echo $i # 1 will be printed to stdout But this doesn't: #!/usr/bin/env bash set -o errexit i=0 ((i++)) # terminates here with $? of 1 echo $i The relevant doc is for `let`: let arg [arg ...] Each arg is an arithmetic expression to be evaluated (see ARITHMETIC EVALUATION above). If the last arg evaluates to 0, let returns 1; 0 is returned otherwise. https://www.gnu.org/software/bash/manual/bash.html#index-let https://www.gnu.org/software/bash/manual/bash.html#index-let
    - chubot 4y ago
      Yeah it's best to just use POSIX shell arithmetic, and not be clever, and not the other 2 ways bash has to do it! THat is: i=$((i + 1)) The (( )) construct, without the $, is different, and it's also not POSIX. It has the gotchas around the exit code, which a plain assignment doesn't have.
  - fab23 4y ago
    It does also matter where the plus signs are: $ i=0 $ echo $((++i)) 1 $ echo $((i++)) 1 $ echo $((++i)) 3 $ ((++i)) does add 1 before the command is run, and ((i++)) does add 1 after the command is run.
- jwilk 4y ago
  Or, more simply, $((i + 1)). And it's not only bash; it's feature of POSIX shell.
  - shrewm 4y ago
    Or as they did in ancient times $[i+1]
    - jwilk 4y ago
      > The old format $[expression] is deprecated and will be removed in upcoming versions of bash.
- jcynix 4y ago
  Or with two brackets instead of four parens: $[i+4] And, btw, res=$(command) instead of those pesky backticks res=`command` ,-)
- bdowling 4y ago
  You can also use the builtin command `let`: let count=0 until "$@"; do let wait=2**count sleep $wait let count=count+1 done
- yx827ha 4y ago
  Don't forget about: declare -i i i+=1
ghayes 4y ago
`until "$@"; do` is pretty crafty.
- jandrese 4y ago
  Crafty or crufty?
- krackers 4y ago
  Could you elaborate? What does it do?
  - ghayes 4y ago
    bash’s ‘if’ command will executes its arguments and enters the clause if the exit code is non-zero. Usually we use “[“ which is the test built-in to accomplish this (like “if [ $n -gt 5 ]”), which works the same way. But here the script uses “$@“, which is the rest of args “splat” in bash. The script passes those args to the “if” command which will thus execute “$@“ and enter the inner clause if that command returns non-zero. This is exactly what we want, since that inner clause is the retry logic. Just fun and weird bashism in the wild. (Sorry if there’s minor errors here, on a phone and going off of memory)
gorgoiler 4y ago
This won’t work if you retry another function and are relying on set -e to detect errors: set -e retry() { until "$@" do sleep 1 done } f() { false echo oh no } retry f # oh no set -e is ignored when a function executes in the context of a predicate. Shell scripts are great for when you just need to run stuff. If something bad happens you notice it and run another command to ameliorate the error. This is how most interactive shell sessions pan out. As soon as you need to handle errors programmatically I heartily suggest that you switch to a language with fewer quirks than sh.
- inferiorhuman 4y ago
  That quote from Jurassic Park comes to mind. As soon as you need to handle errors programmatically I heartily suggest that you switch to a language with fewer quirks than sh. Basically that. `set -e` is evil.
  - 411111111111111 4y ago
    No, it's not. Nor did the parent indicate such. set -euo pipefail should be the default so that the script exits on at least some unexpected state/command results. If you want to do more then that with various fallbacks etc... then use another language.
    - inferiorhuman 4y ago
      The problem with set -e is that you end up failing in inconsistent and non-intuitive ways. e.g. https://stackoverflow.com/questions/4072984/how-do-i-get-the-effect-and-usefulness-of-set-e-inside-a-shell-function https://stackoverflow.com/questions/4072984/how-do-i-get-the... https://serverfault.com/questions/143445/what-does-set-e-do-and-why-might-it-be-considered-dangerous https://serverfault.com/questions/143445/what-does-set-e-do-... pipefail is worse because tools like grep return non-zero for things that aren't inherently errors. Error handling in sh-like shells is archaic enough that if you're reaching for it you should strongly consider reaching for a different language.
operator-name 4y ago
Pure exponential back off can result in very long delays. It is often useful to truncate the sleep time, and for most contentious jobs also some jitter.
- emmelaich 4y ago
  Yeah a plateau and repeat limit are useful. Also fibonacci could be considered.
- geocrasher 4y ago
  This function is called with an argument that limits the number of retries, so unlimited exponential backoff isn't an issue. Jitter is an interesting idea.
  - operator-name 4y ago
    Truncation/capping the sleep time is useful even under a finite number of retries?
dspillett 4y ago
I would add some random jitter to the timing of the backoff. If the fail is due to two such processes getting themselves into some sort of deadlock and they both fail at the same time, they might retry at the same time if at the same part of the retry cycle and fail again for the same reason.
geocrasher 4y ago
I posted this because it worked perfectly for what I needed it for: a watchdog script on a cron job that issuis a PING to a process that occasionally times out. I'm sure it has a million flaws. Everything in Bash does, depending on who you ask. I'm certain there are a thousand reasons it won't work for somebody's application. It worked for mine.
- pkkm 4y ago
  Changed a bit in case of wrong or weird input. (The quotes on the right side of assignments aren't necessary but I find it to be a good practice to use quotes everywhere.) #!/usr/bin/env bash function retry { local retries="$1" shift case "$retries" in ""|*[!0-9]*) echo "retry: First argument must be a number, got '$retries'" 1>&2 return 1 ;; esac local count=0 until "$@"; do local exit="$?" local wait="$((2 ** count))" count="$((count + 1))" if [ "$count" -lt "$retries" ]; then echo "Retry $count/$retries exited $exit, retrying in $wait seconds..." 1>&2 sleep "$wait" else echo "Retry $count/$retries exited $exit, no more retries left." 1>&2 return "$exit" fi done return 0 } retry "$@"
cryptonector 4y ago
You need some randomness in how long you'll wait.
majkinetor 4y ago
Here is similar in pwsh, it retries script block while exception is thrown, until no exception or timeout: https://github.com/majkinetor/posh/blob/master/MM_Sugar/Wait-Action.ps1 https://github.com/majkinetor/posh/blob/master/MM_Sugar/Wait...
pixelbeat__ 4y ago
This technique is useful for running things that normally run quickly, but sometimes need more time. The technique is used extensively in the GNU coreutils test suite, as detailed in the "performance" section at https://www.pixelbeat.org/docs/coreutils-testing.html https://www.pixelbeat.org/docs/coreutils-testing.html
askvictor 4y ago
In this day and age I'm surprise there isn't something like this built into the standard library of every language. Or at least Python.
- BiteCode_dev 4y ago
  Inded. At least there is a very good decorator in pypy: https://pypi.org/project/retry/ https://pypi.org/project/retry/ But having it in the stdlib would make a ton of sense.

121 points by geocrasher 4y ago

sillysaurusx 4y ago

I was today years old when I learned that bash can do simple arithmetic with $(($i + 1)). Thanks!

mmh0000 4y ago
If you just want to add/subtract 1, in bash you can do just '((i++))', i.e.: [mmh@x670]$ i=0 [mmh@x670]$ ((i++)); echo $i 1 [mmh@x670]$ ((i++)); echo $i 2 Beware that the above is very -bashy- and purists will rip your head off for using it.
- sillysaurusx 4y ago
  Whoa. No dollar signs at all? I’m at Lowes — does ((i+1)) work? Or is it just for incrementing / decrementing (which is still very useful!) EDIT: thinking this over, does this work for variable substitution too? E.g. ((i)) being equivalent to $i. Not that you’d necessarily want to…
  - mmh0000 4y ago
    To add arbitrary numbers, you'd have to do an assignment, like this: [mmh@x670]$ ((i=i+2)); echo $i 6 [mmh@x670]$ ((i=i+2)); echo $i 8
  - Arnavion 4y ago
    `(( ))` is an arithmetic evaluation block. Its content has to be an arithmetic expression. Arithmetic expressions don't require `$` before simple variable names and some more complex expressions like array indexing. `(( i + 1 ))` will evaluate the result of adding one to `$i` and then throw away the result. It doesn't do anything useful (other than having a different exit code depending on whether the expression evaluated to 0 or not). `$(( ... ))` evaluates the expression and then returns its value. ie `i=$(( i + 1 ))` will increment `$i`, just like `(( i++ ))` did.
- mmh0000 4y ago
  Also, the wooledge wiki is an amazing resource for Bash/shell scripting. To learn more magic see: https://mywiki.wooledge.org/ArithmeticExpression https://mywiki.wooledge.org/ArithmeticExpression
- cmckn 4y ago
  There is a gotcha here if you use: set -o errexit Because *any bash arithmetic expression that evaluates to zero returns the exit code of 1*. So this works: #!/usr/bin/env bash set -o errexit i=0 ((++i)) echo $i # 1 will be printed to stdout But this doesn't: #!/usr/bin/env bash set -o errexit i=0 ((i++)) # terminates here with $? of 1 echo $i The relevant doc is for `let`: let arg [arg ...] Each arg is an arithmetic expression to be evaluated (see ARITHMETIC EVALUATION above). If the last arg evaluates to 0, let returns 1; 0 is returned otherwise. https://www.gnu.org/software/bash/manual/bash.html#index-let https://www.gnu.org/software/bash/manual/bash.html#index-let
  - chubot 4y ago
    Yeah it's best to just use POSIX shell arithmetic, and not be clever, and not the other 2 ways bash has to do it! THat is: i=$((i + 1)) The (( )) construct, without the $, is different, and it's also not POSIX. It has the gotchas around the exit code, which a plain assignment doesn't have.
- fab23 4y ago
  It does also matter where the plus signs are: $ i=0 $ echo $((++i)) 1 $ echo $((i++)) 1 $ echo $((++i)) 3 $ ((++i)) does add 1 before the command is run, and ((i++)) does add 1 after the command is run.
jwilk 4y ago
Or, more simply, $((i + 1)). And it's not only bash; it's feature of POSIX shell.
- shrewm 4y ago
  Or as they did in ancient times $[i+1]
  - jwilk 4y ago
    > The old format $[expression] is deprecated and will be removed in upcoming versions of bash.
jcynix 4y ago
Or with two brackets instead of four parens: $[i+4] And, btw, res=$(command) instead of those pesky backticks res=`command` ,-)
bdowling 4y ago
You can also use the builtin command `let`: let count=0 until "$@"; do let wait=2**count sleep $wait let count=count+1 done
yx827ha 4y ago
Don't forget about: declare -i i i+=1

ghayes 4y ago

`until "$@"; do` is pretty crafty.

jandrese 4y ago
Crafty or crufty?
krackers 4y ago
Could you elaborate? What does it do?
- ghayes 4y ago
  bash’s ‘if’ command will executes its arguments and enters the clause if the exit code is non-zero. Usually we use “[“ which is the test built-in to accomplish this (like “if [ $n -gt 5 ]”), which works the same way. But here the script uses “$@“, which is the rest of args “splat” in bash. The script passes those args to the “if” command which will thus execute “$@“ and enter the inner clause if that command returns non-zero. This is exactly what we want, since that inner clause is the retry logic. Just fun and weird bashism in the wild. (Sorry if there’s minor errors here, on a phone and going off of memory)

gorgoiler 4y ago

This won’t work if you retry another function and are relying on set -e to detect errors: set -e retry() { until "$@" do sleep 1 done } f() { false echo oh no } retry f # oh no set -e is ignored when a function executes in the context of a predicate. Shell scripts are great for when you just need to run stuff. If something bad happens you notice it and run another command to ameliorate the error. This is how most interactive shell sessions pan out. As soon as you need to handle errors programmatically I heartily suggest that you switch to a language with fewer quirks than sh.

inferiorhuman 4y ago
That quote from Jurassic Park comes to mind. As soon as you need to handle errors programmatically I heartily suggest that you switch to a language with fewer quirks than sh. Basically that. `set -e` is evil.
- 411111111111111 4y ago
  No, it's not. Nor did the parent indicate such. set -euo pipefail should be the default so that the script exits on at least some unexpected state/command results. If you want to do more then that with various fallbacks etc... then use another language.
  - inferiorhuman 4y ago
    The problem with set -e is that you end up failing in inconsistent and non-intuitive ways. e.g. https://stackoverflow.com/questions/4072984/how-do-i-get-the-effect-and-usefulness-of-set-e-inside-a-shell-function https://stackoverflow.com/questions/4072984/how-do-i-get-the... https://serverfault.com/questions/143445/what-does-set-e-do-and-why-might-it-be-considered-dangerous https://serverfault.com/questions/143445/what-does-set-e-do-... pipefail is worse because tools like grep return non-zero for things that aren't inherently errors. Error handling in sh-like shells is archaic enough that if you're reaching for it you should strongly consider reaching for a different language.

operator-name 4y ago

Pure exponential back off can result in very long delays. It is often useful to truncate the sleep time, and for most contentious jobs also some jitter.

emmelaich 4y ago
Yeah a plateau and repeat limit are useful. Also fibonacci could be considered.
geocrasher 4y ago
This function is called with an argument that limits the number of retries, so unlimited exponential backoff isn't an issue. Jitter is an interesting idea.
- operator-name 4y ago
  Truncation/capping the sleep time is useful even under a finite number of retries?

dspillett 4y ago

I would add some random jitter to the timing of the backoff. If the fail is due to two such processes getting themselves into some sort of deadlock and they both fail at the same time, they might retry at the same time if at the same part of the retry cycle and fail again for the same reason.

geocrasher 4y ago

I posted this because it worked perfectly for what I needed it for: a watchdog script on a cron job that issuis a PING to a process that occasionally times out. I'm sure it has a million flaws. Everything in Bash does, depending on who you ask. I'm certain there are a thousand reasons it won't work for somebody's application. It worked for mine.

pkkm 4y ago
Changed a bit in case of wrong or weird input. (The quotes on the right side of assignments aren't necessary but I find it to be a good practice to use quotes everywhere.) #!/usr/bin/env bash function retry { local retries="$1" shift case "$retries" in ""|*[!0-9]*) echo "retry: First argument must be a number, got '$retries'" 1>&2 return 1 ;; esac local count=0 until "$@"; do local exit="$?" local wait="$((2 ** count))" count="$((count + 1))" if [ "$count" -lt "$retries" ]; then echo "Retry $count/$retries exited $exit, retrying in $wait seconds..." 1>&2 sleep "$wait" else echo "Retry $count/$retries exited $exit, no more retries left." 1>&2 return "$exit" fi done return 0 } retry "$@"

cryptonector 4y ago

You need some randomness in how long you'll wait.

majkinetor 4y ago

Here is similar in pwsh, it retries script block while exception is thrown, until no exception or timeout: https://github.com/majkinetor/posh/blob/master/MM_Sugar/Wait-Action.ps1 https://github.com/majkinetor/posh/blob/master/MM_Sugar/Wait...

pixelbeat__ 4y ago

This technique is useful for running things that normally run quickly, but sometimes need more time. The technique is used extensively in the GNU coreutils test suite, as detailed in the "performance" section at https://www.pixelbeat.org/docs/coreutils-testing.html https://www.pixelbeat.org/docs/coreutils-testing.html

askvictor 4y ago

In this day and age I'm surprise there isn't something like this built into the standard library of every language. Or at least Python.

BiteCode_dev 4y ago
Inded. At least there is a very good decorator in pypy: https://pypi.org/project/retry/ https://pypi.org/project/retry/ But having it in the stdlib would make a ton of sense.