3 ms·

> A lot of this seems like the fault of the ALB, is it? I definitely think the ALB Controller should be taking a more active hand in termination of pods that a

> A lot of this seems like the fault of the ALB, is it? I definitely think the ALB Controller should be taking a more active hand in termination of pods that are targets of an ALB. But the ALB Controller is exhibiting the same symptom I keep running into throughout Kubernetes. The amount of "X is a problem because the pod dies too quickly before Y has a chance to clean up/whatever, so we add a preStop sleep of 30 seconds" in the Kubernetes world is truly frustrating.

lucasyvas 2y ago
If you are referring the 30 seconds to kill time, that would be holding it wrong. As long as your process is PID 1, you can rig up your own process exit handlers, which completely resolves the problem. Many people don’t run the main process in the container as PID 1, so this “problem” remains. If it’s not feasible to remove something like a shell process from being the first thing that runs, exec will allow replacing the shell process with the application process.
- relistan 2y ago
  Yeah, exactly. We just catch the TERM, clean up, and then shut down. But the rest of the top post in the thread is right on.
  - remram 2y ago
    If I understand correctly, because ALB does its own health checks, you need to catch TERM, wait 30s while returning non-ready for ALB to have time to notice, then clean up and shut down.
- paranoidrobot 2y ago
  > If you are referring the 30 seconds to kill time, that would be holding it wrong. As long as your process is PID 1, you can rig up your own process exit handlers, which completely resolves the problem. Maybe I am holding it wrong. I'd love not to have to do this work. But I don't see how being PID 1 or not helps (and yes, for most workloads it is PID 1) The ALB controller is the one that would need to deregister a target from the target group, and it won't until the pod is gone. So we have to force it by having the app do the functional equivalent with the readiness check.
kunley 2y ago
Kubernetes was written by people who have developer, not ops, background and is full of things like this. The fact that it became a standard is a disaster
- gchamonlive 2y ago
  Maybe, or maybe orchestration and load balancing is hard. I think it's too simplistic to dismiss k8s development because the devs weren't ops. I don't know of a tool that does a significantly better job at this without having other drawbacks and gotchas, and even if it did it doesn't void the value k8s brings. I have my own set of gripes with software production engineering in general and specially with k8s, having seen first hand how much effort big corps have to put just to manage a cluster, but it's disrespectful to qualify this whole endeavour as disastrous.
  - kunley 2y ago
    Guys who wrote it are ok, they put a lot of effort and that's fine. If I understand things correctly, they were also compensated well. But the effort based on some wrong assumptions makes a product that is flawed. Lot of people then are forced to use it because there is no alternative, or the alternatives are easily dismissed - behavior based in turn on a certain propaganda and marketing. And that part is a disaster. This is not personal, btw.

lucasyvas 2y ago

If you are referring the 30 seconds to kill time, that would be holding it wrong. As long as your process is PID 1, you can rig up your own process exit handlers, which completely resolves the problem. Many people don’t run the main process in the container as PID 1, so this “problem” remains. If it’s not feasible to remove something like a shell process from being the first thing that runs, exec will allow replacing the shell process with the application process.

relistan 2y ago
Yeah, exactly. We just catch the TERM, clean up, and then shut down. But the rest of the top post in the thread is right on.
- remram 2y ago
  If I understand correctly, because ALB does its own health checks, you need to catch TERM, wait 30s while returning non-ready for ALB to have time to notice, then clean up and shut down.
paranoidrobot 2y ago
> If you are referring the 30 seconds to kill time, that would be holding it wrong. As long as your process is PID 1, you can rig up your own process exit handlers, which completely resolves the problem. Maybe I am holding it wrong. I'd love not to have to do this work. But I don't see how being PID 1 or not helps (and yes, for most workloads it is PID 1) The ALB controller is the one that would need to deregister a target from the target group, and it won't until the pod is gone. So we have to force it by having the app do the functional equivalent with the readiness check.

kunley 2y ago

Kubernetes was written by people who have developer, not ops, background and is full of things like this. The fact that it became a standard is a disaster

gchamonlive 2y ago
Maybe, or maybe orchestration and load balancing is hard. I think it's too simplistic to dismiss k8s development because the devs weren't ops. I don't know of a tool that does a significantly better job at this without having other drawbacks and gotchas, and even if it did it doesn't void the value k8s brings. I have my own set of gripes with software production engineering in general and specially with k8s, having seen first hand how much effort big corps have to put just to manage a cluster, but it's disrespectful to qualify this whole endeavour as disastrous.
- kunley 2y ago
  Guys who wrote it are ok, they put a lot of effort and that's fine. If I understand things correctly, they were also compensated well. But the effort based on some wrong assumptions makes a product that is flawed. Lot of people then are forced to use it because there is no alternative, or the alternatives are easily dismissed - behavior based in turn on a certain propaganda and marketing. And that part is a disaster. This is not personal, btw.