17 ms·

We reduced 502 errors by caring about PID 1 in Kubernetes

116 points by TangerineDream 4y ago

sp332 4y ago
Wouldn't it make more sense to remove the pod from the Endpoint first, before sending it shutdown signals? Or are the containers dying unexpectedly on a regular basis for some reason?
- morelisp 4y ago
  Removing the endpoint won't close the existing open connections. The container has to do that on shutdown. The signal is the, uh, signal to start that process. Also, > We had a bit of tunnel vision and kept focusing/blaming that we aren't removing the Pod from the Endpoint quickly enough.
bovermyer 4y ago
I love this writeup. It's a great article about the troubleshooting process, and enlightening as to certain behaviors of Kubernetes. It also, though, makes me think about whether technology has progressed by adopting Kubernetes, or if we've just made things more complicated.
- davedx 4y ago
  Things are definitely more complicated. At GitLab's scale though, it's probably worth the trade-off.
- 0xbadcafebee 4y ago
  The end-user capabilities are easier, but not more advanced. Much of the design is problematic, it definitely overcomplicates most things, and it lacks important modern-day functionality. Kubernetes is essentially Jenkins w/ JCasC & Docker plugins, and a hundred extra abstractions and APIs. It's a mutable state mess with a proprietary backend, master-slave dynamics, lacks real multitenancy, doesn't use immutable version-controlled artifacts or objects, has a flawed security model, and adds unnecessary abstraction that requires massive add-on integration to make a working system. But, like Jenkins, because it "does everything" and is free and self-hosted, people are hooked on it and can't stop using.
  - wernerb 4y ago
    Interested in the proprietary backend part. What is proprietary?
    - 0xbadcafebee 4y ago
      "Proprietary" was bad wording, technically other people can use etcd... I mean to say inflexible & incompatible. Someone suggested supporting Consul 8 years ago (nearly functionally equivalent to Etcd), was requested for years by like a thousand people, never happened. Somebody even submitted a PR I think, but it was quashed. I/O should have been abstracted at the very beginning as loosely-coupled pluggable layers.
      - pm90 4y ago
        Generally when you make claims like this, giving proof (eg PR for contributing consul support that was quashed) makes the claim stronger. I looked it up and this seems to be the PR https://github.com/kubernetes/kubernetes/pull/31622 https://github.com/kubernetes/kubernetes/pull/31622. Reading through the discussion, it doesn’t look like it was quashed; the project owners had a reasoned debate about the pros/cons of what would happen if such a support was included at the time and its a totally reasonable argument to me.
- pm90 4y ago
  Kubernetes should be thought of as the missing layer of abstraction between computers and production grade services. Replicating the capabilities it provides in another system that offers such capabilities wouldn’t involve less complexity. In fact, I would argue that the way the authors reasoned through the issues was made possible because they were able to operate with these abstractions. Imagine doing this but with eg AWS NLBs. This team might have discovered similar quirks in that setup. But now their knowledge and what they could share would be restricted to AWS NLBs. Since they’re working with K8s, the lessons they learned are a lot more widely applicable. Another thing to consider (as pointed out by a sibling) is that an average user of k8s may not likely have to worry about minimizing 5xxs during application termination. But if they did want to, now they can.
solatic 4y ago
Great write-up, but, why not just use dumb-init? https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init
- morelisp 4y ago
  Running dumb-init in a shell would not have helped. Running dumb-init (or tini or whatever, there's lots of these) when your application can handle SIGTERM itself is unnecessary. These are tools for when you don't control an executable in the stack and it doesn't do the right thing itself.
  - richardwhiuk 4y ago
    You won't get a corefile if you are PID 1, which is a pretty good reason to use tini.
    - morelisp 4y ago
      While I suppose this is true, I have never needed to examine a core dump from a container, I don't believe our Kubernetes deployment can collect them at the moment (or at least I would need to ask our ops team where to put them / how to get them once dumped), and to a first approximation zero of the services I run containerized produce helpful core dumps in the first place. It's not even out of the question for us to have more RAM used by one process than free disk space. (I do have a lot of experience with helpful corefiles, it's just been totally orthogonal to my experience with "at-scale" backend deployment. Different languages, different dev/ops relations, different attitudes towards abends in general.)
  - pm90 4y ago
    > when your application can handle SIGTERM itself is unnecessary. Thats a big “when”. Many applications don’t. e.g. they spawn child processes which they forget about
    - morelisp 4y ago
      Container inits don't magically get you clean shutdown if your application doesn't handle SIGTERM, it just gets you, let's say, fast shutdown. (And "fast", if you're not going to shutdown cleanly, might be much worse!) If you don't want connection issues you'll still need to write a SIGTERM handler, and then you don't need the container init anymore. And "many" is kind of a cop-out word; yes, I'm sure by absolute number it's a lot but: - If you're in e.g. Go, you spawn thousands of child green threads but it's trivial to route a SIGTERMable context into all of them and wait for only the relevant ones. - If you're in e.g. Python, you'll be running in an application server that handles it and you're probably not spawning many threads yourself because it's usually pointless. - If you're any piece of software written to run outside of containers, that's also what non-container environments have used for decades to handle smooth upgrades. - etc. for other common cases in the end, "spawning threads" is no excuse for not handling clean listener shutdown and seems to be more about how "Unix-like" the application is and not threading - e.g. old Java programs tend to be worst in my experience. I've used tini et al mostly when putting entire legacy environments (e.g. nginx+PHP+memcache+mysqld) into a single container with a hacky shell script around them because we needed them off bare metal yesterday but there's no dev resources to properly adapt the application. There's definitely "many" of those in the world, but it doesn't speak to picking tooling to support that by default.
hericium 4y ago
> As an SRE at GitLab, you get to touch a lot of services that you didn't build yourself and interact with system dependencies that you might have not touched before. There's always detective work to do! This doesn't sound good to me and translates to "our staff may not know what they're doing". Lack of or outdated documentation, lack of time to read it?
- roflc0ptic 4y ago
  Where have you worked where there were enough resources to effectively document all of your backend code? My job is currently mostly detective work, and it’s neither a function of incompetence nor laziness, it’s just a function of needing to prioritize business objectives over sacrifice-able things like documentation.
  - cranekam 4y ago
    Documentation also just doesn't cover the kind of issue the GitLab post is talking about. Sure, good docs on what talks to what and so on will help someone learn the lay of the land but it's not going to pinpoint the cause of a spike of 502 errors. Working that kind of thing out requires looking at the problem with a fresh set of eyes to see what's really happening, not what should be happening according to the docs. Some of the best troubleshooters I know figure things out by just looking at what's going on. What's this server talking to? Why is this backend slower? What else is happening on this host? Which network links? etc. None of this is in a manual.
    - hericium 4y ago
      > Documentation also just doesn't cover the kind of issue the GitLab post is talking about. Sure, good docs on what talks to what and so on will help someone learn the lay of the land but it's not going to pinpoint the cause of a spike of 502 errors. And that's perfectly fine. This post was published so GL folks can say "we figured out something tricky/interesting and our scale is what made it more tricky". I love figuring out "Frankenstein" type of bugs, errors and problems myself. What I dislike figuring out is something that shouldn't require any figuring out at all. I read GitLab or Cloudflare's posts with pleasure. These doesn't seem to be written solely as technical recruitment "we do cool stuff" posts.
AndyMcConachie 4y ago
> GitLab Workhorse is a Go service in front of our Ruby on Rails monolith and it's deployed as a sidecar inside of the webservice pod, which runs Ruby on Rails using the Puma web server. It's always amazing to me how complicated web stuff has gotten these days. I started out in the LAMP days over 20 years ago. I'm not this complexity is necessarily bad. I don't know the details and I don't know how to make a web app scalable today. But damn if this doesn't sound like a pile of hot garbage waiting to fail to my ears.
- klysm 4y ago
  I think ruby is mostly to blame. Proxies are almost always useful though and rarely contain that much complexity.
  - davedx 4y ago
    This was nothing to do with ruby. It was to do with how docker, k8s and shells interact when trying to shut down processes.
- marcosdumay 4y ago
  Compare it with Gogs, that is an executable that you run. (Yes, it doesn't do CI/CD, but I don't think avoiding a CI/CD server is worth the extra complexity.) GitLab (the software) is a child of the microservices culture. That said, that architecture is probably a very good fit for GitLab (the company). It's just not very good for other people to self-host it.
  - anderspitman 4y ago
    This is one of the reasons I'm focused on helping self-hosting take off. It reduces the complexity of the software by an order of magnitude, while retaining much of the scalability. One problem is it's still too hard. Hosting your own Gitea/gogs shouldn't be any more difficult or less secure than installing an app on your phone. This is where my work is currently. Another problem is things like repo stars. You can federate, but then every instance has to deal with spam. Not sure what a great solution to this is.
kevingadd 4y ago
Feels really strange to adjust the liveness checks in a way that potentially causes healthy instances to get thrown out (while still having a delay due to the polling interval) instead of just deterministically making it dead when you start shutting it down.
- morelisp 4y ago
  Isn't that why it's step 2 of 3? It sounds like with the SIGTERM issue fixed it could be dialed back to its previous value. But it also sounds like they triaged reasonably, reducing the incidence to buy more time to debug the deeper cause.
- totony 4y ago
  A pod can be unready but still live, which is what should happen if its a temporary failure.
jeffbee 4y ago
The client - "Gitlab Pages" in this article - should be retrying 502s on a different socket anyway.
- morelisp 4y ago
  Nice in theory, but in practice the client might be a public web browser, or even something internal making a non-idempotent request.
  - nrmitchi 4y ago
    It could be, but in this instance it was not. It was an internal application that Gitlab had control over making an internal idempotent request. The underlying root cause should be identified so that this same issue can be corrected when retries aren't a solution, but if this was causing Gitlab Pages to burn through it's SLO (which is stated in the article), the Pages team could have (and should have) stopped the leak with a retry which the root cause was being determined.
    - morelisp 4y ago
      But an immediate retry would've ended up going over the same socket and getting the same response. Yes, "do some backoff", etc., but then we're in much murkier waters where Pages probably also has a latency SLI and if you're talking to a connection that might not let you know it's going to reset for up to 30 seconds, that's probably not going to fly either.
      - jeffbee 4y ago
        Why would a retry go over the same channel?
        morelisp 4y ago
        Why would you close a TCP connection just because you got an HTTP error?
mc4ndr3 4y ago
Never fond of CMD. Always prefer ENTRYPOINT.
pm90 4y ago
> This was exactly what we were doing! we weren't using CMD in exec form, but in shell form. Changing this confirmed that gitlab-workhorse is now PID 1, and also receives the termination signal after testing it locally Do shells not forward signals correctly to their child processes? This seems surprising to me.
- js2 4y ago
  It depends, and it's complicated. Here's bash's behavior: https://www.gnu.org/software/bash/manual/html_node/Signals.html https://www.gnu.org/software/bash/manual/html_node/Signals.h... I'm confused about this myself because I thought bash ignores SIGTERM only when it's interactive: https://www.gnu.org/software/bash/manual/html_node/Interactive-Shell-Behavior.html https://www.gnu.org/software/bash/manual/html_node/Interacti... In this case bash is not interactive, so I'm not sure why SIGTERM is ignored, but FWIW, the docker documentation explicitly says that it will: > The shell form prevents any CMD or run command line arguments from being used, but has the disadvantage that your ENTRYPOINT will be started as a subcommand of /bin/sh -c, which does not pass signals. This means that the executable will not be the container’s PID 1 - and will not receive Unix signals - so your executable will not receive a SIGTERM from docker stop <container>. https://docs.docker.com/engine/reference/builder/#cmd https://docs.docker.com/engine/reference/builder/#cmd The bash documentation for -c says nothing about different signal handling. POSIX mode doesn't address this behavior either.
deleted 4y ago
[deleted]
aeyes 4y ago
> Pods are removed from the Service/Endpoint in async They list this in the Takeaways but I don't see that they took any action to work around this problem. SIGTERM to the pod and updating the service/endpoint race and you will absolutely see traffic getting sent to terminated processes because of it. The only "solution" I know is to put a sleep in the preStop hook to delay the SIGTERM so that the pod can be removed from the service. Cloud native!
- jayd16 4y ago
  Seems like they were not seeing the race, and instead only saw that open connections would stay open if not explicitly closed during the graceful shutdown. Once the graceful shutdown was properly executed, it closed any open connections to that pod and stopped the 502s they were seeing. Sounds like either the race wasn't happening or they didn't see/care about it.
deleted 4y ago
[deleted]
0xbadcafebee 4y ago
Anybody know of any Kubernetes replacements in development? I would be happy to work on such a project.
- mdaniel 4y ago
  I believe Nomad is hailed as the replacement: https://github.com/hashicorp/nomad#readme https://github.com/hashicorp/nomad#readme I would caution one against "javascript framework syndrome:" bah, this framework is too complicated, I just want something simpler ... ok, it doesn't work right in all circumstances, I'll just add this one feature(, ... ok, maybe just this one other feature)+ ... bah, this framework is too complicated!
  - 0xbadcafebee 4y ago
    I like Nomad, but it's just a scheduler. There's a happier balance to be found with a complete distributed system that is loosely coupled, simple, and uses modern syseng best practices. I started writing something in Bash (https://github.com/peterwwillis/shell-distributed-system/blob/main/DESIGN.md https://github.com/peterwwillis/shell-distributed-system/blo...) just to show people it doesn't have to be complicated, but I got bored.
    - klysm 4y ago
      > complete distributed system that is loosely coupled, simple, and uses modern syseng best practices > in Bash Yeah I wouldn’t touch that with a 10ft pole. No way you are going to do something complex without introducing hideous errors because bash is a horrible language for doing anything complicated.
      - 0xbadcafebee 4y ago
        Using a complex programming language makes it easier to do complex things. But what if you don't want to do something complex? Turns out that simple things become complex with a complex programming language. The best way to get to a simple solution is to use simple tools in a simple way. I probably made the mistake of making it too complicated already. I'll go back at some point and strip most of it out.
justin_oaks 4y ago
It is poor design for a Dockerfile to have two forms of CMD where each is only distinguished from the other by argument formatting. For people new to Dockerfiles, it's too easy to be oblivious to the fact that the two forms do something differently. For people who've used Dockerfiles a lot, it's too easy to miss the formatting difference, especially when the two different CMD forms can successfully run what you want it to run. CMD should have been split into two separate statement types, either completely separate types (e.g. SHELLCMD vs RAWCMD) or two explicit subtypes of CMD (e.g. CMD SHELL vs CMD RAW). Then you couldn't read each of the CMD variants without seeing the difference. New users would see the difference and think "Why is there a difference? I should look it up." Old users would more easily notice when the shell CMD is used when the raw CMD should be used, and vice versa.
- wernerb 4y ago
  Kubernetes lets you override the dockerfile with "command" and "args". Always try to be explicit especially in nonstandard/exceptional application images. A shared base image with known proven startup knowledge that is used throughout all applications is easier to manage and more transparent. In more pluriform environments just submit and ensure the configuration of your pod/deployment is as explicit we possible.
uberduper 4y ago
This is pretty common knowledge if you have any experience working with kubernetes. Some less common knowledge: Do your pre-startup config with a shell script and then `exec` your process. If you need some sort of process manager, use supervisord and configure stdout/stderr capture and forwarding. Supervisord will do anything you need from init in a container. If you're running a jvm in the container, set the dns cache ttl in the java security config. Sometimes it's very useful to include a preStop hook that will end your process and then sleep for N seconds so that your process unbinds its listening socket and then for a period of time the pod will respond to incoming SYNs with a RST. So your upstream app doesn't sit there waiting on a timeout because for various reasons it still made new connection attempts despite the pod being removed from the service endpoints or w/e. Fix your resolver configs. ndots. timeout. retry. Remove all the hosts search domains by setting `--resolv-conf=/dev/null` in your kubelet options. Then set dnsConfig.options in your pod spec. While you're at it, set `--allowed-unsafe-sysctls 'net.ipv4.tcp.keepalive*'` so you can configure your pods with sane values. Update pod spec with securityContext.sysctls to set the per pod values. Update the kubelet before applying that pod spec because lol if you don't... (your pods do not inherit these values from the host! So if you think you've set them, you're probably wrong.)
- azinman2 4y ago
  Why aren’t these all just defaults?
  - robertlagrant 4y ago
    The DNS I suppose obeys the principle of least surprise. If I want to call the Twilio API I may be surprised at it not resolving on DNS by default.
- GauntletWizard 4y ago
  Writing good Lameducking is the single best trick for preserving SLO with Kubernetes. It's not that hard, but it is tricky, and testing it is the hardest part.
  - 4y ago
kristjansson 4y ago
The differences between shell and exec form of CMD and ENTRYPOINT are subtle and percinous, and the difference in syntax isn’t nearly commensurate with the change in behavior. For example, apparently the child processes of the shell form dont inherit environment variables with dots in the name (yeah, I know…). That was a fun one to puzzle out.

Wouldn't it make more sense to remove the pod from the Endpoint first, before sending it shutdown signals? Or are the containers dying unexpectedly on a regular basis for some reason?

morelisp 4y ago
Removing the endpoint won't close the existing open connections. The container has to do that on shutdown. The signal is the, uh, signal to start that process. Also, > We had a bit of tunnel vision and kept focusing/blaming that we aren't removing the Pod from the Endpoint quickly enough.

bovermyer 4y ago

I love this writeup. It's a great article about the troubleshooting process, and enlightening as to certain behaviors of Kubernetes. It also, though, makes me think about whether technology has progressed by adopting Kubernetes, or if we've just made things more complicated.

davedx 4y ago
Things are definitely more complicated. At GitLab's scale though, it's probably worth the trade-off.
0xbadcafebee 4y ago
The end-user capabilities are easier, but not more advanced. Much of the design is problematic, it definitely overcomplicates most things, and it lacks important modern-day functionality. Kubernetes is essentially Jenkins w/ JCasC & Docker plugins, and a hundred extra abstractions and APIs. It's a mutable state mess with a proprietary backend, master-slave dynamics, lacks real multitenancy, doesn't use immutable version-controlled artifacts or objects, has a flawed security model, and adds unnecessary abstraction that requires massive add-on integration to make a working system. But, like Jenkins, because it "does everything" and is free and self-hosted, people are hooked on it and can't stop using.
- wernerb 4y ago
  Interested in the proprietary backend part. What is proprietary?
  - 0xbadcafebee 4y ago
    "Proprietary" was bad wording, technically other people can use etcd... I mean to say inflexible & incompatible. Someone suggested supporting Consul 8 years ago (nearly functionally equivalent to Etcd), was requested for years by like a thousand people, never happened. Somebody even submitted a PR I think, but it was quashed. I/O should have been abstracted at the very beginning as loosely-coupled pluggable layers.
    - pm90 4y ago
      Generally when you make claims like this, giving proof (eg PR for contributing consul support that was quashed) makes the claim stronger. I looked it up and this seems to be the PR https://github.com/kubernetes/kubernetes/pull/31622 https://github.com/kubernetes/kubernetes/pull/31622. Reading through the discussion, it doesn’t look like it was quashed; the project owners had a reasoned debate about the pros/cons of what would happen if such a support was included at the time and its a totally reasonable argument to me.
pm90 4y ago
Kubernetes should be thought of as the missing layer of abstraction between computers and production grade services. Replicating the capabilities it provides in another system that offers such capabilities wouldn’t involve less complexity. In fact, I would argue that the way the authors reasoned through the issues was made possible because they were able to operate with these abstractions. Imagine doing this but with eg AWS NLBs. This team might have discovered similar quirks in that setup. But now their knowledge and what they could share would be restricted to AWS NLBs. Since they’re working with K8s, the lessons they learned are a lot more widely applicable. Another thing to consider (as pointed out by a sibling) is that an average user of k8s may not likely have to worry about minimizing 5xxs during application termination. But if they did want to, now they can.

solatic 4y ago

Great write-up, but, why not just use dumb-init? https://github.com/Yelp/dumb-init https://github.com/Yelp/dumb-init

morelisp 4y ago
Running dumb-init in a shell would not have helped. Running dumb-init (or tini or whatever, there's lots of these) when your application can handle SIGTERM itself is unnecessary. These are tools for when you don't control an executable in the stack and it doesn't do the right thing itself.
- richardwhiuk 4y ago
  You won't get a corefile if you are PID 1, which is a pretty good reason to use tini.
  - morelisp 4y ago
    While I suppose this is true, I have never needed to examine a core dump from a container, I don't believe our Kubernetes deployment can collect them at the moment (or at least I would need to ask our ops team where to put them / how to get them once dumped), and to a first approximation zero of the services I run containerized produce helpful core dumps in the first place. It's not even out of the question for us to have more RAM used by one process than free disk space. (I do have a lot of experience with helpful corefiles, it's just been totally orthogonal to my experience with "at-scale" backend deployment. Different languages, different dev/ops relations, different attitudes towards abends in general.)
- pm90 4y ago
  > when your application can handle SIGTERM itself is unnecessary. Thats a big “when”. Many applications don’t. e.g. they spawn child processes which they forget about
  - morelisp 4y ago
    Container inits don't magically get you clean shutdown if your application doesn't handle SIGTERM, it just gets you, let's say, fast shutdown. (And "fast", if you're not going to shutdown cleanly, might be much worse!) If you don't want connection issues you'll still need to write a SIGTERM handler, and then you don't need the container init anymore. And "many" is kind of a cop-out word; yes, I'm sure by absolute number it's a lot but: - If you're in e.g. Go, you spawn thousands of child green threads but it's trivial to route a SIGTERMable context into all of them and wait for only the relevant ones. - If you're in e.g. Python, you'll be running in an application server that handles it and you're probably not spawning many threads yourself because it's usually pointless. - If you're any piece of software written to run outside of containers, that's also what non-container environments have used for decades to handle smooth upgrades. - etc. for other common cases in the end, "spawning threads" is no excuse for not handling clean listener shutdown and seems to be more about how "Unix-like" the application is and not threading - e.g. old Java programs tend to be worst in my experience. I've used tini et al mostly when putting entire legacy environments (e.g. nginx+PHP+memcache+mysqld) into a single container with a hacky shell script around them because we needed them off bare metal yesterday but there's no dev resources to properly adapt the application. There's definitely "many" of those in the world, but it doesn't speak to picking tooling to support that by default.

hericium 4y ago

> As an SRE at GitLab, you get to touch a lot of services that you didn't build yourself and interact with system dependencies that you might have not touched before. There's always detective work to do! This doesn't sound good to me and translates to "our staff may not know what they're doing". Lack of or outdated documentation, lack of time to read it?

roflc0ptic 4y ago
Where have you worked where there were enough resources to effectively document all of your backend code? My job is currently mostly detective work, and it’s neither a function of incompetence nor laziness, it’s just a function of needing to prioritize business objectives over sacrifice-able things like documentation.
- cranekam 4y ago
  Documentation also just doesn't cover the kind of issue the GitLab post is talking about. Sure, good docs on what talks to what and so on will help someone learn the lay of the land but it's not going to pinpoint the cause of a spike of 502 errors. Working that kind of thing out requires looking at the problem with a fresh set of eyes to see what's really happening, not what should be happening according to the docs. Some of the best troubleshooters I know figure things out by just looking at what's going on. What's this server talking to? Why is this backend slower? What else is happening on this host? Which network links? etc. None of this is in a manual.
  - hericium 4y ago
    > Documentation also just doesn't cover the kind of issue the GitLab post is talking about. Sure, good docs on what talks to what and so on will help someone learn the lay of the land but it's not going to pinpoint the cause of a spike of 502 errors. And that's perfectly fine. This post was published so GL folks can say "we figured out something tricky/interesting and our scale is what made it more tricky". I love figuring out "Frankenstein" type of bugs, errors and problems myself. What I dislike figuring out is something that shouldn't require any figuring out at all. I read GitLab or Cloudflare's posts with pleasure. These doesn't seem to be written solely as technical recruitment "we do cool stuff" posts.

AndyMcConachie 4y ago

> GitLab Workhorse is a Go service in front of our Ruby on Rails monolith and it's deployed as a sidecar inside of the webservice pod, which runs Ruby on Rails using the Puma web server. It's always amazing to me how complicated web stuff has gotten these days. I started out in the LAMP days over 20 years ago. I'm not this complexity is necessarily bad. I don't know the details and I don't know how to make a web app scalable today. But damn if this doesn't sound like a pile of hot garbage waiting to fail to my ears.

klysm 4y ago
I think ruby is mostly to blame. Proxies are almost always useful though and rarely contain that much complexity.
- davedx 4y ago
  This was nothing to do with ruby. It was to do with how docker, k8s and shells interact when trying to shut down processes.
marcosdumay 4y ago
Compare it with Gogs, that is an executable that you run. (Yes, it doesn't do CI/CD, but I don't think avoiding a CI/CD server is worth the extra complexity.) GitLab (the software) is a child of the microservices culture. That said, that architecture is probably a very good fit for GitLab (the company). It's just not very good for other people to self-host it.
- anderspitman 4y ago
  This is one of the reasons I'm focused on helping self-hosting take off. It reduces the complexity of the software by an order of magnitude, while retaining much of the scalability. One problem is it's still too hard. Hosting your own Gitea/gogs shouldn't be any more difficult or less secure than installing an app on your phone. This is where my work is currently. Another problem is things like repo stars. You can federate, but then every instance has to deal with spam. Not sure what a great solution to this is.

kevingadd 4y ago

Feels really strange to adjust the liveness checks in a way that potentially causes healthy instances to get thrown out (while still having a delay due to the polling interval) instead of just deterministically making it dead when you start shutting it down.

morelisp 4y ago
Isn't that why it's step 2 of 3? It sounds like with the SIGTERM issue fixed it could be dialed back to its previous value. But it also sounds like they triaged reasonably, reducing the incidence to buy more time to debug the deeper cause.
totony 4y ago
A pod can be unready but still live, which is what should happen if its a temporary failure.

jeffbee 4y ago

The client - "Gitlab Pages" in this article - should be retrying 502s on a different socket anyway.

morelisp 4y ago
Nice in theory, but in practice the client might be a public web browser, or even something internal making a non-idempotent request.
- nrmitchi 4y ago
  It could be, but in this instance it was not. It was an internal application that Gitlab had control over making an internal idempotent request. The underlying root cause should be identified so that this same issue can be corrected when retries aren't a solution, but if this was causing Gitlab Pages to burn through it's SLO (which is stated in the article), the Pages team could have (and should have) stopped the leak with a retry which the root cause was being determined.
  - morelisp 4y ago
    But an immediate retry would've ended up going over the same socket and getting the same response. Yes, "do some backoff", etc., but then we're in much murkier waters where Pages probably also has a latency SLI and if you're talking to a connection that might not let you know it's going to reset for up to 30 seconds, that's probably not going to fly either.
    - jeffbee 4y ago
      Why would a retry go over the same channel?
      - morelisp 4y ago
        Why would you close a TCP connection just because you got an HTTP error?

mc4ndr3 4y ago

Never fond of CMD. Always prefer ENTRYPOINT.

pm90 4y ago

> This was exactly what we were doing! we weren't using CMD in exec form, but in shell form. Changing this confirmed that gitlab-workhorse is now PID 1, and also receives the termination signal after testing it locally Do shells not forward signals correctly to their child processes? This seems surprising to me.

js2 4y ago
It depends, and it's complicated. Here's bash's behavior: https://www.gnu.org/software/bash/manual/html_node/Signals.html https://www.gnu.org/software/bash/manual/html_node/Signals.h... I'm confused about this myself because I thought bash ignores SIGTERM only when it's interactive: https://www.gnu.org/software/bash/manual/html_node/Interactive-Shell-Behavior.html https://www.gnu.org/software/bash/manual/html_node/Interacti... In this case bash is not interactive, so I'm not sure why SIGTERM is ignored, but FWIW, the docker documentation explicitly says that it will: > The shell form prevents any CMD or run command line arguments from being used, but has the disadvantage that your ENTRYPOINT will be started as a subcommand of /bin/sh -c, which does not pass signals. This means that the executable will not be the container’s PID 1 - and will not receive Unix signals - so your executable will not receive a SIGTERM from docker stop <container>. https://docs.docker.com/engine/reference/builder/#cmd https://docs.docker.com/engine/reference/builder/#cmd The bash documentation for -c says nothing about different signal handling. POSIX mode doesn't address this behavior either.

deleted 4y ago

[deleted]

aeyes 4y ago

> Pods are removed from the Service/Endpoint in async They list this in the Takeaways but I don't see that they took any action to work around this problem. SIGTERM to the pod and updating the service/endpoint race and you will absolutely see traffic getting sent to terminated processes because of it. The only "solution" I know is to put a sleep in the preStop hook to delay the SIGTERM so that the pod can be removed from the service. Cloud native!

jayd16 4y ago
Seems like they were not seeing the race, and instead only saw that open connections would stay open if not explicitly closed during the graceful shutdown. Once the graceful shutdown was properly executed, it closed any open connections to that pod and stopped the 502s they were seeing. Sounds like either the race wasn't happening or they didn't see/care about it.

deleted 4y ago

[deleted]

0xbadcafebee 4y ago

Anybody know of any Kubernetes replacements in development? I would be happy to work on such a project.

mdaniel 4y ago
I believe Nomad is hailed as the replacement: https://github.com/hashicorp/nomad#readme https://github.com/hashicorp/nomad#readme I would caution one against "javascript framework syndrome:" bah, this framework is too complicated, I just want something simpler ... ok, it doesn't work right in all circumstances, I'll just add this one feature(, ... ok, maybe just this one other feature)+ ... bah, this framework is too complicated!
- 0xbadcafebee 4y ago
  I like Nomad, but it's just a scheduler. There's a happier balance to be found with a complete distributed system that is loosely coupled, simple, and uses modern syseng best practices. I started writing something in Bash (https://github.com/peterwwillis/shell-distributed-system/blob/main/DESIGN.md https://github.com/peterwwillis/shell-distributed-system/blo...) just to show people it doesn't have to be complicated, but I got bored.
  - klysm 4y ago
    > complete distributed system that is loosely coupled, simple, and uses modern syseng best practices > in Bash Yeah I wouldn’t touch that with a 10ft pole. No way you are going to do something complex without introducing hideous errors because bash is a horrible language for doing anything complicated.
    - 0xbadcafebee 4y ago
      Using a complex programming language makes it easier to do complex things. But what if you don't want to do something complex? Turns out that simple things become complex with a complex programming language. The best way to get to a simple solution is to use simple tools in a simple way. I probably made the mistake of making it too complicated already. I'll go back at some point and strip most of it out.

justin_oaks 4y ago

It is poor design for a Dockerfile to have two forms of CMD where each is only distinguished from the other by argument formatting. For people new to Dockerfiles, it's too easy to be oblivious to the fact that the two forms do something differently. For people who've used Dockerfiles a lot, it's too easy to miss the formatting difference, especially when the two different CMD forms can successfully run what you want it to run. CMD should have been split into two separate statement types, either completely separate types (e.g. SHELLCMD vs RAWCMD) or two explicit subtypes of CMD (e.g. CMD SHELL vs CMD RAW). Then you couldn't read each of the CMD variants without seeing the difference. New users would see the difference and think "Why is there a difference? I should look it up." Old users would more easily notice when the shell CMD is used when the raw CMD should be used, and vice versa.

wernerb 4y ago
Kubernetes lets you override the dockerfile with "command" and "args". Always try to be explicit especially in nonstandard/exceptional application images. A shared base image with known proven startup knowledge that is used throughout all applications is easier to manage and more transparent. In more pluriform environments just submit and ensure the configuration of your pod/deployment is as explicit we possible.

uberduper 4y ago

This is pretty common knowledge if you have any experience working with kubernetes. Some less common knowledge: Do your pre-startup config with a shell script and then `exec` your process. If you need some sort of process manager, use supervisord and configure stdout/stderr capture and forwarding. Supervisord will do anything you need from init in a container. If you're running a jvm in the container, set the dns cache ttl in the java security config. Sometimes it's very useful to include a preStop hook that will end your process and then sleep for N seconds so that your process unbinds its listening socket and then for a period of time the pod will respond to incoming SYNs with a RST. So your upstream app doesn't sit there waiting on a timeout because for various reasons it still made new connection attempts despite the pod being removed from the service endpoints or w/e. Fix your resolver configs. ndots. timeout. retry. Remove all the hosts search domains by setting `--resolv-conf=/dev/null` in your kubelet options. Then set dnsConfig.options in your pod spec. While you're at it, set `--allowed-unsafe-sysctls 'net.ipv4.tcp.keepalive*'` so you can configure your pods with sane values. Update pod spec with securityContext.sysctls to set the per pod values. Update the kubelet before applying that pod spec because lol if you don't... (your pods do not inherit these values from the host! So if you think you've set them, you're probably wrong.)

azinman2 4y ago
Why aren’t these all just defaults?
- robertlagrant 4y ago
  The DNS I suppose obeys the principle of least surprise. If I want to call the Twilio API I may be surprised at it not resolving on DNS by default.
GauntletWizard 4y ago
Writing good Lameducking is the single best trick for preserving SLO with Kubernetes. It's not that hard, but it is tricky, and testing it is the hardest part.
- 4y ago

kristjansson 4y ago

The differences between shell and exec form of CMD and ENTRYPOINT are subtle and percinous, and the difference in syntax isn’t nearly commensurate with the change in behavior. For example, apparently the child processes of the shell form dont inherit environment variables with dots in the name (yeah, I know…). That was a fun one to puzzle out.