4 ms·

I love this writeup. It's a great article about the troubleshooting process, and enlightening as to certain behaviors of Kubernetes. It also, though, makes me

I love this writeup. It's a great article about the troubleshooting process, and enlightening as to certain behaviors of Kubernetes. It also, though, makes me think about whether technology has progressed by adopting Kubernetes, or if we've just made things more complicated.

davedx 4y ago
Things are definitely more complicated. At GitLab's scale though, it's probably worth the trade-off.
0xbadcafebee 4y ago
The end-user capabilities are easier, but not more advanced. Much of the design is problematic, it definitely overcomplicates most things, and it lacks important modern-day functionality. Kubernetes is essentially Jenkins w/ JCasC & Docker plugins, and a hundred extra abstractions and APIs. It's a mutable state mess with a proprietary backend, master-slave dynamics, lacks real multitenancy, doesn't use immutable version-controlled artifacts or objects, has a flawed security model, and adds unnecessary abstraction that requires massive add-on integration to make a working system. But, like Jenkins, because it "does everything" and is free and self-hosted, people are hooked on it and can't stop using.
- wernerb 4y ago
  Interested in the proprietary backend part. What is proprietary?
  - 0xbadcafebee 4y ago
    "Proprietary" was bad wording, technically other people can use etcd... I mean to say inflexible & incompatible. Someone suggested supporting Consul 8 years ago (nearly functionally equivalent to Etcd), was requested for years by like a thousand people, never happened. Somebody even submitted a PR I think, but it was quashed. I/O should have been abstracted at the very beginning as loosely-coupled pluggable layers.
    - pm90 4y ago
      Generally when you make claims like this, giving proof (eg PR for contributing consul support that was quashed) makes the claim stronger. I looked it up and this seems to be the PR https://github.com/kubernetes/kubernetes/pull/31622 https://github.com/kubernetes/kubernetes/pull/31622. Reading through the discussion, it doesn’t look like it was quashed; the project owners had a reasoned debate about the pros/cons of what would happen if such a support was included at the time and its a totally reasonable argument to me.
pm90 4y ago
Kubernetes should be thought of as the missing layer of abstraction between computers and production grade services. Replicating the capabilities it provides in another system that offers such capabilities wouldn’t involve less complexity. In fact, I would argue that the way the authors reasoned through the issues was made possible because they were able to operate with these abstractions. Imagine doing this but with eg AWS NLBs. This team might have discovered similar quirks in that setup. But now their knowledge and what they could share would be restricted to AWS NLBs. Since they’re working with K8s, the lessons they learned are a lot more widely applicable. Another thing to consider (as pointed out by a sibling) is that an average user of k8s may not likely have to worry about minimizing 5xxs during application termination. But if they did want to, now they can.

davedx 4y ago

Things are definitely more complicated. At GitLab's scale though, it's probably worth the trade-off.

0xbadcafebee 4y ago

The end-user capabilities are easier, but not more advanced. Much of the design is problematic, it definitely overcomplicates most things, and it lacks important modern-day functionality. Kubernetes is essentially Jenkins w/ JCasC & Docker plugins, and a hundred extra abstractions and APIs. It's a mutable state mess with a proprietary backend, master-slave dynamics, lacks real multitenancy, doesn't use immutable version-controlled artifacts or objects, has a flawed security model, and adds unnecessary abstraction that requires massive add-on integration to make a working system. But, like Jenkins, because it "does everything" and is free and self-hosted, people are hooked on it and can't stop using.

wernerb 4y ago
Interested in the proprietary backend part. What is proprietary?
- 0xbadcafebee 4y ago
  "Proprietary" was bad wording, technically other people can use etcd... I mean to say inflexible & incompatible. Someone suggested supporting Consul 8 years ago (nearly functionally equivalent to Etcd), was requested for years by like a thousand people, never happened. Somebody even submitted a PR I think, but it was quashed. I/O should have been abstracted at the very beginning as loosely-coupled pluggable layers.
  - pm90 4y ago
    Generally when you make claims like this, giving proof (eg PR for contributing consul support that was quashed) makes the claim stronger. I looked it up and this seems to be the PR https://github.com/kubernetes/kubernetes/pull/31622 https://github.com/kubernetes/kubernetes/pull/31622. Reading through the discussion, it doesn’t look like it was quashed; the project owners had a reasoned debate about the pros/cons of what would happen if such a support was included at the time and its a totally reasonable argument to me.

pm90 4y ago

Kubernetes should be thought of as the missing layer of abstraction between computers and production grade services. Replicating the capabilities it provides in another system that offers such capabilities wouldn’t involve less complexity. In fact, I would argue that the way the authors reasoned through the issues was made possible because they were able to operate with these abstractions. Imagine doing this but with eg AWS NLBs. This team might have discovered similar quirks in that setup. But now their knowledge and what they could share would be restricted to AWS NLBs. Since they’re working with K8s, the lessons they learned are a lot more widely applicable. Another thing to consider (as pointed out by a sibling) is that an average user of k8s may not likely have to worry about minimizing 5xxs during application termination. But if they did want to, now they can.