5 ms·

Did a devops elective last semester; I was surprised to hear that sleeping to wait for the system to come up was "common in the industry". 30 seconds was the ma

Did a devops elective last semester; I was surprised to hear that sleeping to wait for the system to come up was "common in the industry". 30 seconds was the magic number, which was somehow always either mostly a waste or not enough when I tried to use it. For similar reasons, we similarly can't keep making sleep-sort faster by scaling the sleeps down.

drdaeman 3y ago
Sleeping on error is a valid pattern when combined with retries. Consider if that would be a long-starting external process that sets up the listener (or even a remote server spinning up). I.e. something we have no control over and that has no ability to signal us it's ready. In such case, sleeping a little then trying to connect (and repeating) would be simple, effective and economical solution.
- mindcrime 3y ago
  Sleeping on error is a valid pattern when combined with retries. Agreed. Just note that it can sometimes (often?) be a good idea to add a small stochastic element to the sleep time, and/or do something similar to the exponential back-off[1] approach that Ethernet (among other things) uses. [1]: https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff
  - tylerhou 3y ago
    You ideally want both exponential back off plus some random delay (to prevent thundering herd).
  - chrsig 3y ago
    I've never personaly seen any benefit from exponential backoff... constant backoff + jitters + short timeouts has always worked well for me. If anything, people underestimating the ramp up of exponential back off(or not capping it) has left me waiting much longer than necessary
    - justcool393 3y ago
      the amount of times something (Discord is the first example that comes to mind) starts waiting 30 seconds between retries because I wasn't connected to the network has been super absurd.
- hayley-patton 3y ago
  That was "more sophisticated" and I was given a sample script without a loop. (I got Terraform to wait in a retry loop, which was apparently too complex.)
  - xena 3y ago
    Yeah for the record I'm not talking about delay loops, I'm talking about hardcoded sleeps in test code that cause random failures when unspoken invariants change. DevOps is a unique blend of hellfire that is almost unfair to compare to everything else in the industry. I'm glad I don't work in DevOps anymore.
bheadmaster 3y ago
> we similarly can't keep making sleep-sort faster by scaling the sleeps down. Technically, we can, at least to a certain limit - as long as the total time needed to schedule the sleeps is less than the shortest sleep duration, sleepsort will still work. Though we're just offloading the sorting to the scheduler, which has to implement the sorting algorithm anyway...
FireBeyond 3y ago
I ended up writing a function that would sleep and retry according to the Fibonacci sequence up to a given total elapsed time, so you could set it to a max timeout of 60s, for example, but if the system came up in 8 seconds, you'd be waiting not much longer than that (11 seconds, technically).
- wiml 3y ago
  I do this too occasionally — Fibonacci is easy to implement, and it's an exponential backoff (with a lower exponent than 2, which is also easy to implement ;) )
  - nmca 3y ago
    I found the following line in a codebase once, which made me laugh: ``` interval = interval*1.6 # fiboroughly ```
bandrami 3y ago
This bleeds into the Init Wars (sorrows, prayers), but there was this idea for a while that every service was going to be directly queryable by the run control system so that there would never be a need for magic sleep numbers. The result fell rather short of the ideal (particularly on the teardown side of things) but it did at least help admins get a firmer handle on exactly where we were (and still are, for the most part) using magic waits of "long enough".
- murphyslaw 3y ago
  Way back I worked on a project to used virtualized routers inside labs for a bunch of use cases. The idea was to remove the need for building after building filled with equipment that just sat there idling 99% of the time. The biggest headache we had was to figure out when everything was up. There simply was no way to detect this programmatically, so the only way would have been to alter the code to give some kind of signal on the console like "System is up." similar to a linux box showing the login prompt. So I resorted to wait loops too. We ended up canceling the project.
  - bandrami 3y ago
    There have been several developments in the past decade or so that center on the philosophy of "make it the client's problem" (hello, Wayland) which has simultaneously led to much more robust and maintainable centralized systems with much more fragmented and unpredictable peripheral clients. I'm still not sure whether I regret this or not.
paranoidrobot 3y ago
I've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so. I've been endlessly frustrated by how prevalent "just add a <x> second sleep" is in the Kubernetes world. Examples: If you're using IRSA to provide pods an AWS IAM Role, it can take some time on first-run for the role to become usable. So your app has to just wait until it can successfully talk to AWS. If you're using the AWS Elastic Load Balancer controller, you need to add sleeps during pre-stop for a pod. The controller has to see the shutdown event, then start target deregistration on the ELB Target Group. That's a minimum of 30 seconds, during which you can be getting new connections. Service meshes on Kubernetes add some fun on both startup AND shutdown. Linkerd, for instance, will only inject it's linkerd-proxy container into your pod after startup. So it's entirely possible for your main application container to start, begin talking to the world and then a moment later all your connections get reset when Linkerd-proxy takes over the pod's networking configuration. You can fix this by injecting into all of your containers a linkerd-await utility that replaces the container entrypoint. It will wait until linkerd starts before starting your regular entrypoint. It's basically a more advanced version of sleep. Shutdown is just as fun - linkerd-proxy will immediately terminate, dropping all network connectivity for all the containers in your pod. There's a fix for this, too, which involves the above linkerd-await and another option.
- mikepurvis 3y ago
  With systemd having standardized so much stuff on sd_notify for a basic liveness check, it's frustrating that container setups can't piggy-back more on that.
  - arianvanp 3y ago
    Systemd's socket activation mechanism also got born from the "Sleep is not a synchronisation primitive; talking to a socket is!" and the container world just ignored it whilst it's a great fit! Actually it's a generalisation of what OP is doing here. Move the listener _out_ of the program and you never have to know if a service is up or not. Connecting to a service just blocks until it's up.
- jabradoodle 3y ago
Aloha 3y ago
I think the sleep N > check for availability paradigm is pretty useful. The alternatives have a ton more complexity for not much benefit.

drdaeman 3y ago

Sleeping on error is a valid pattern when combined with retries. Consider if that would be a long-starting external process that sets up the listener (or even a remote server spinning up). I.e. something we have no control over and that has no ability to signal us it's ready. In such case, sleeping a little then trying to connect (and repeating) would be simple, effective and economical solution.

mindcrime 3y ago
Sleeping on error is a valid pattern when combined with retries. Agreed. Just note that it can sometimes (often?) be a good idea to add a small stochastic element to the sleep time, and/or do something similar to the exponential back-off[1] approach that Ethernet (among other things) uses. [1]: https://en.wikipedia.org/wiki/Exponential_backoff https://en.wikipedia.org/wiki/Exponential_backoff
- tylerhou 3y ago
  You ideally want both exponential back off plus some random delay (to prevent thundering herd).
- chrsig 3y ago
  I've never personaly seen any benefit from exponential backoff... constant backoff + jitters + short timeouts has always worked well for me. If anything, people underestimating the ramp up of exponential back off(or not capping it) has left me waiting much longer than necessary
  - justcool393 3y ago
    the amount of times something (Discord is the first example that comes to mind) starts waiting 30 seconds between retries because I wasn't connected to the network has been super absurd.
hayley-patton 3y ago
That was "more sophisticated" and I was given a sample script without a loop. (I got Terraform to wait in a retry loop, which was apparently too complex.)
- xena 3y ago
  Yeah for the record I'm not talking about delay loops, I'm talking about hardcoded sleeps in test code that cause random failures when unspoken invariants change. DevOps is a unique blend of hellfire that is almost unfair to compare to everything else in the industry. I'm glad I don't work in DevOps anymore.

bheadmaster 3y ago

> we similarly can't keep making sleep-sort faster by scaling the sleeps down. Technically, we can, at least to a certain limit - as long as the total time needed to schedule the sleeps is less than the shortest sleep duration, sleepsort will still work. Though we're just offloading the sorting to the scheduler, which has to implement the sorting algorithm anyway...

FireBeyond 3y ago

I ended up writing a function that would sleep and retry according to the Fibonacci sequence up to a given total elapsed time, so you could set it to a max timeout of 60s, for example, but if the system came up in 8 seconds, you'd be waiting not much longer than that (11 seconds, technically).

wiml 3y ago
I do this too occasionally — Fibonacci is easy to implement, and it's an exponential backoff (with a lower exponent than 2, which is also easy to implement ;) )
- nmca 3y ago
  I found the following line in a codebase once, which made me laugh: ``` interval = interval*1.6 # fiboroughly ```

bandrami 3y ago

This bleeds into the Init Wars (sorrows, prayers), but there was this idea for a while that every service was going to be directly queryable by the run control system so that there would never be a need for magic sleep numbers. The result fell rather short of the ideal (particularly on the teardown side of things) but it did at least help admins get a firmer handle on exactly where we were (and still are, for the most part) using magic waits of "long enough".

murphyslaw 3y ago
Way back I worked on a project to used virtualized routers inside labs for a bunch of use cases. The idea was to remove the need for building after building filled with equipment that just sat there idling 99% of the time. The biggest headache we had was to figure out when everything was up. There simply was no way to detect this programmatically, so the only way would have been to alter the code to give some kind of signal on the console like "System is up." similar to a linux box showing the login prompt. So I resorted to wait loops too. We ended up canceling the project.
- bandrami 3y ago
  There have been several developments in the past decade or so that center on the philosophy of "make it the client's problem" (hello, Wayland) which has simultaneously led to much more robust and maintainable centralized systems with much more fragmented and unpredictable peripheral clients. I'm still not sure whether I regret this or not.

paranoidrobot 3y ago

I've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so. I've been endlessly frustrated by how prevalent "just add a <x> second sleep" is in the Kubernetes world. Examples: If you're using IRSA to provide pods an AWS IAM Role, it can take some time on first-run for the role to become usable. So your app has to just wait until it can successfully talk to AWS. If you're using the AWS Elastic Load Balancer controller, you need to add sleeps during pre-stop for a pod. The controller has to see the shutdown event, then start target deregistration on the ELB Target Group. That's a minimum of 30 seconds, during which you can be getting new connections. Service meshes on Kubernetes add some fun on both startup AND shutdown. Linkerd, for instance, will only inject it's linkerd-proxy container into your pod after startup. So it's entirely possible for your main application container to start, begin talking to the world and then a moment later all your connections get reset when Linkerd-proxy takes over the pod's networking configuration. You can fix this by injecting into all of your containers a linkerd-await utility that replaces the container entrypoint. It will wait until linkerd starts before starting your regular entrypoint. It's basically a more advanced version of sleep. Shutdown is just as fun - linkerd-proxy will immediately terminate, dropping all network connectivity for all the containers in your pod. There's a fix for this, too, which involves the above linkerd-await and another option.

mikepurvis 3y ago
With systemd having standardized so much stuff on sd_notify for a basic liveness check, it's frustrating that container setups can't piggy-back more on that.
- arianvanp 3y ago
  Systemd's socket activation mechanism also got born from the "Sleep is not a synchronisation primitive; talking to a socket is!" and the container world just ignored it whilst it's a great fit! Actually it's a generalisation of what OP is doing here. Move the listener _out_ of the program and you never have to know if a service is up or not. Connecting to a service just blocks until it's up.
jabradoodle 3y ago

Aloha 3y ago

I think the sleep N > check for availability paradigm is pretty useful. The alternatives have a ton more complexity for not much benefit.