4 ms·
I've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so. I've been endlessly frustrated by how prevalent "just
by paranoidrobot 3y ago
I've been doing "devops" stuff for a long time, but only got into Kubernetes in the last 18 months or so.
I've been endlessly frustrated by how prevalent "just add a <x> second sleep" is in the Kubernetes world.
Examples:
If you're using IRSA to provide pods an AWS IAM Role, it can take some time on first-run for the role to become usable. So your app has to just wait until it can successfully talk to AWS.
If you're using the AWS Elastic Load Balancer controller, you need to add sleeps during pre-stop for a pod. The controller has to see the shutdown event, then start target deregistration on the ELB Target Group. That's a minimum of 30 seconds, during which you can be getting new connections.
Service meshes on Kubernetes add some fun on both startup AND shutdown.
Linkerd, for instance, will only inject it's linkerd-proxy container into your pod after startup. So it's entirely possible for your main application container to start, begin talking to the world and then a moment later all your connections get reset when Linkerd-proxy takes over the pod's networking configuration.
You can fix this by injecting into all of your containers a linkerd-await utility that replaces the container entrypoint. It will wait until linkerd starts before starting your regular entrypoint. It's basically a more advanced version of sleep.
Shutdown is just as fun - linkerd-proxy will immediately terminate, dropping all network connectivity for all the containers in your pod. There's a fix for this, too, which involves the above linkerd-await and another option.
- mikepurvis 3y agoWith systemd having standardized so much stuff on sd_notify for a basic liveness check, it's frustrating that container setups can't piggy-back more on that.
- arianvanp 3y agoSystemd's socket activation mechanism also got born from the "Sleep is not a synchronisation primitive; talking to a socket is!" and the container world just ignored it whilst it's a great fit! Actually it's a generalisation of what OP is doing here. Move the listener _out_ of the program and you never have to know if a service is up or not. Connecting to a service just blocks until it's up.
- jabradoodle 3y ago> It's basically a more advanced version of sleep. Is then the waiting for the availability/existence of anything a "more advanced version of sleep"? Is a readiness probe a sleep? I don't follow at all.
- GauntletWizard 3y agoPolling relies on sleep. Setting the polling time matters a great deal, because you'll almost always take at least one poll interval to discover that something has come up, and on average every polled step in your startup process adds (poll time/2) to your total start-up time. If you have a bunch of steps that require polling, that adds up fast. The problem is often relying too much on cold-starts; these polls amortize well in a real system, where you can afford (and want to afford!)