7 ms·

Using Kubernetes to rethink your system architecture and ease technical debt

64 points by kiyanwang 5y ago

develatio 5y ago
Kubernetes is a great piece of software and it definitely has it's advantages and use-cases. That said, this post strikes me on several fronts. The writer starts by saying that servers were treated as pets, which inherently means difficulty (as the author clears out). I assume that, at some point, their services got reworked to a shape where they could be deployed in servers treated as cattle. This is a work the author had to pay no matter if the final goal was Kubernetes, Autoscale Groups or any other solution. Then the author explains how they got to the conclusion that they need to containerise their service(s) in order to be able to use K8s, which would then, in turn, allow them to scale properly (adding on top the Go rewrite that is mentioned at some point in the post). The author also explains that Autoscale Groups were discarded because of their complexity. It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services. In my experience, once your service(s) are ready to be deployed in servers that can be managed as cattle, the benefits show up independently of the deployment solution that is used to deploy the service(s) (K8s, Autoscale Groups, etc...). Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"? If that were the case (which I believe it might have), the other thing to consider would be the difficulty of managing a K8s cluster vs the difficulty of managing an Autoscale Group. I've seen quite some horror stories about K8s[1] and I'd rather choose ASG over K8s. [1]: https://k8s.af https://k8s.af
- vp8989 5y ago
  If you are a Director of DevOps, what is easier to say to the CTO: "I need X million USD in engineering spend this year because we need to be on Kubernetes". "I need X million USD in engineering spend this year to fix half a decade of awful technical execution and decision making (that all happened under your watch, BTW)".
  - edoceo 5y ago
    > awful technical execution Leave that out. Many right decisions look stupid later. It will happen to you to. Time makes fools of us all.
    - deleted 5y ago
      [deleted]
    - milesvp 5y ago
      This. It’s always important to assume the decisions made before you were correct, or at least a good approximation to correct given the conditions or knowledge at the time. It’s sort of a generalization of chesterson’s fence. If you don’t know why a decision might have been smart at the time, you will likely make another mistake trying to correct it. It may not always be true but thinking this way can at least keep you honest.
    - leucineleprec0n 5y ago
      The question was one in rhetorical jest, no? Clearly the former statement is "correct"
  - yawaramin 5y ago
    Neither: "I need X million USD in engineering spend this year because we need to practice DevOps and SRE."
    - qw 5y ago
      "I need X million USD in engineering spend this year because it would improve my CV"
      - rualca 5y ago
        The author of this blog post states quite clearly that his company already used EKS elsewhere. If all he wanted to do was add a fancy line to his CV, why would he bother with something he already had? Also, it's silly to just assume that continuing to treat pampered EC2 instances that had to be manually deployed ande cared for is something that benefits a projectsl, specially if the company already shed those bad practices elsewhere.
- edoceo 5y ago
  Yea. I'm a VM type of admin. One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets. As if docker was the only path to that end. Never even bothered to talk about the existing infra that was managed like cattle - and that it was more a function of App-Architecture than Docker vs X. Its nice to love your tools. Don't start thinking they are the one true path.
  - tisFine 5y ago
    Agreed. Software is not hardware. Behavioral patterns for a user are not tethered to one ephemeral implementation. I ran a huge VMWare cluster in 3 data centers across the globe, not so different from an AWS/EKS pipeline. Yes, I had to treat the hardware like pets to an extent. But the developers had no problem pivoting between VMWare and what we ran in AWS because the software tooling we provided looked the same. Some folks believe memorizing YouTube videos = expertise.
  - ransom1538 5y ago
    "One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets." You can always point out that pods are ran on nodes. Nodes use the standard built in GCP Auto-scaler (instance groups + load balancer). So its a great wrapper! But, it's a wrapper. GCP also! has a way to use a docker image as your VM.
    - edoceo 5y ago
      Agree and understood. You sound reasonable. IME zealots trade reason for enthusiasm and they focus too much on the method, not the concept. This individual was shocked to learn I thought you could treat the iron-born servers as cattle too. (But, my first clusters were in the late 90s, so got to the cattle concept quite some time ago)
  - rualca 5y ago
    > Never even bothered to talk about the existing infra that was managed like cattle I don't see the point of your counterargument. For starters, there are plenty of managed containerized solutions that just run your containers without you ever having to bother with them. Secondly, it seems you got it entirely wrong: the point of pets vs cattle are the services, not the servers. You want to take your services down and put them up without surprises. Unless you're planning on rebooting your infrastructure just for shits and giggles, it makes no sense to talk about treating "servers" as pets, specially as you already get those as managed services just like you get VMs.
- rualca 5y ago
  > It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services. It seems you're missing the forest for the trees. They put in the work to refactorize their services so that they could be treated as cattle and also autoscale, and the author stated that after doing his homework he determined that for his company Kubernetes offered the most benefits. There isn't much to read from this. > Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"? The author explains that he considered using EC2 instances but after analising the tradeoffs the company decided to just jump to EKS. If you'd like to debate containers vs VMs then it's perfectly fine, but the author of this blog post just documented his learning path and pointed out what worked for him. Sometimes you need to call a shot, and the tradeoffs you need to consider are so minor and irrelevant that a coin flip will do.
toast0 5y ago
How many lines of code is k8s? Depending on it is adding those millions of lines to your tech debt. Unless you need so many of the features, mostly k8s seems to be a way to increase complexity and build resumes. Yes, you need to automate building instances and service discovery, but neither of those tasks needs millions of lines of code.
- orf 5y ago
  Totally agree. This is why I avoid using Linux because it adds approximately 30 million lines of technical debt to my service, and I definitely don’t need all of the kernel APIs it offers. So I build my own out of assembly. Now my REST api is free of technical debt!
MichaelMoser123 5y ago
Did the stackoverflow architecture change significantly from this series of posts? (seems to mentioning 2016-2019) [1] The article mentions a rewrite, but it wasn't very specific on the scope of the rewrite, and what kind of stack they are using now. I thought they were running their own farm with most of the action on IIS with ASP.net and MSSQL, and some elasticsearch for search. [1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
- jimminy 5y ago
  The article is a guest post from individuals that work at Pusher[1], about architectural decisions they made for their products and services. I also thought it would be about StackOverflow until they started mentioning all the Pusher related products. [1] https://pusher.com https://pusher.com
  - MichaelMoser123 5y ago
    Oh, thanks. Skipped that emtirely. (Too much HN for me)
- papito 5y ago
  No, Stackoverflow architecture is still a simple server farm with aggressively maintained code and a lite SQL ORM that lets them to highly optimize queries. One of the architects talked about it on their pod a few weeks ago. She mentioned explicitly how they don't jump on new hotness and keep it classic, maintainable, and highly performant without spending tens of millions of dollars on FAANG alumni who seem to think that every shop needs a Google infrastructure because THIS IS THE WAY.
commandlinefan 5y ago
> rewrites should be avoided unless they are truly necessary Which is always.
- Thiez 5y ago
  It's always necessary, or they should always be avoided?
  - commandlinefan 5y ago
    Always necessary.
Hackbraten 5y ago
> Kubernetes has great documentation so new starters can get up to speed more quickly even if they don’t have experience with it. I strongly disagree. What feels easy for one developer may be super hard for others. For me, the Kubernetes tutorial [1] and documentation [2] [3] feels borderline incomprehensible. It introduces several abstract concepts at once without telling me what they really mean. It raises more questions than it answers. For example: > Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit. What does that even mean? Is that “single unit” the entire application? Or rather one component, e. g. the backend? You usually don’t have more than one service per container, right? So what exactly is the cluster here? Why is there not a single example? > The Control Plane is responsible for managing the cluster. The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates. We haven’t even established what the cluster is and how it relates to an application but now we’re already talking about several applications? I feel entirely lost. Ok, let’s look at another introductory page [2]. It says things like: > The worker node(s) host the Pods that are the components of the application workload. But it won’t tell me what a Pod is. So I click on the Pods link [3]. It says: > A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific "logical host": it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host. I find that explanation outright hostile. Not a single example that a newbie can understand. Instead, a pile-up of more and more abstract concepts. I still have no clue how a Pod, a node, an application, its instances, its components and a cluster relates. I’m giving up here. This documentation is abysmal. How can the blog author possibly call this thing well-documented? [1]: https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ https://kubernetes.io/docs/tutorials/kubernetes-basics/creat... [2]: https://kubernetes.io/docs/concepts/overview/components/ https://kubernetes.io/docs/concepts/overview/components/ [3]: https://kubernetes.io/docs/concepts/workloads/pods/ https://kubernetes.io/docs/concepts/workloads/pods/
- 5y ago
LordGrey 5y ago
With apologies to Jamie Zawinski: > Some people, when confronted with a problem, think "I know, I'll use Kubernetes." Now they have two problems. Kubernetes can be a great solution to a set of problems, but it is not automatically that way. My company is in the midst of adding k8s support to a large legacy application and that work has turned into a deep dive into madness in some places. To be fair, we're making it hard on ourselves by supporting both k8s and the legacy environment, but the point stands. And Helm doesn't help.

64 points by kiyanwang 5y ago

Kubernetes is a great piece of software and it definitely has it's advantages and use-cases. That said, this post strikes me on several fronts. The writer starts by saying that servers were treated as pets, which inherently means difficulty (as the author clears out). I assume that, at some point, their services got reworked to a shape where they could be deployed in servers treated as cattle. This is a work the author had to pay no matter if the final goal was Kubernetes, Autoscale Groups or any other solution. Then the author explains how they got to the conclusion that they need to containerise their service(s) in order to be able to use K8s, which would then, in turn, allow them to scale properly (adding on top the Go rewrite that is mentioned at some point in the post). The author also explains that Autoscale Groups were discarded because of their complexity. It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services. In my experience, once your service(s) are ready to be deployed in servers that can be managed as cattle, the benefits show up independently of the deployment solution that is used to deploy the service(s) (K8s, Autoscale Groups, etc...). Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"? If that were the case (which I believe it might have), the other thing to consider would be the difficulty of managing a K8s cluster vs the difficulty of managing an Autoscale Group. I've seen quite some horror stories about K8s[1] and I'd rather choose ASG over K8s. [1]: https://k8s.af https://k8s.af

vp8989 5y ago
If you are a Director of DevOps, what is easier to say to the CTO: "I need X million USD in engineering spend this year because we need to be on Kubernetes". "I need X million USD in engineering spend this year to fix half a decade of awful technical execution and decision making (that all happened under your watch, BTW)".
- edoceo 5y ago
  > awful technical execution Leave that out. Many right decisions look stupid later. It will happen to you to. Time makes fools of us all.
  - deleted 5y ago
    [deleted]
  - milesvp 5y ago
    This. It’s always important to assume the decisions made before you were correct, or at least a good approximation to correct given the conditions or knowledge at the time. It’s sort of a generalization of chesterson’s fence. If you don’t know why a decision might have been smart at the time, you will likely make another mistake trying to correct it. It may not always be true but thinking this way can at least keep you honest.
  - leucineleprec0n 5y ago
    The question was one in rhetorical jest, no? Clearly the former statement is "correct"
- yawaramin 5y ago
  Neither: "I need X million USD in engineering spend this year because we need to practice DevOps and SRE."
  - qw 5y ago
    "I need X million USD in engineering spend this year because it would improve my CV"
    - rualca 5y ago
      The author of this blog post states quite clearly that his company already used EKS elsewhere. If all he wanted to do was add a fancy line to his CV, why would he bother with something he already had? Also, it's silly to just assume that continuing to treat pampered EC2 instances that had to be manually deployed ande cared for is something that benefits a projectsl, specially if the company already shed those bad practices elsewhere.
edoceo 5y ago
Yea. I'm a VM type of admin. One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets. As if docker was the only path to that end. Never even bothered to talk about the existing infra that was managed like cattle - and that it was more a function of App-Architecture than Docker vs X. Its nice to love your tools. Don't start thinking they are the one true path.
- tisFine 5y ago
  Agreed. Software is not hardware. Behavioral patterns for a user are not tethered to one ephemeral implementation. I ran a huge VMWare cluster in 3 data centers across the globe, not so different from an AWS/EKS pipeline. Yes, I had to treat the hardware like pets to an extent. But the developers had no problem pivoting between VMWare and what we ran in AWS because the software tooling we provided looked the same. Some folks believe memorizing YouTube videos = expertise.
- ransom1538 5y ago
  "One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets." You can always point out that pods are ran on nodes. Nodes use the standard built in GCP Auto-scaler (instance groups + load balancer). So its a great wrapper! But, it's a wrapper. GCP also! has a way to use a docker image as your VM.
  - edoceo 5y ago
    Agree and understood. You sound reasonable. IME zealots trade reason for enthusiasm and they focus too much on the method, not the concept. This individual was shocked to learn I thought you could treat the iron-born servers as cattle too. (But, my first clusters were in the late 90s, so got to the cattle concept quite some time ago)
- rualca 5y ago
  > Never even bothered to talk about the existing infra that was managed like cattle I don't see the point of your counterargument. For starters, there are plenty of managed containerized solutions that just run your containers without you ever having to bother with them. Secondly, it seems you got it entirely wrong: the point of pets vs cattle are the services, not the servers. You want to take your services down and put them up without surprises. Unless you're planning on rebooting your infrastructure just for shits and giggles, it makes no sense to talk about treating "servers" as pets, specially as you already get those as managed services just like you get VMs.
rualca 5y ago
> It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services. It seems you're missing the forest for the trees. They put in the work to refactorize their services so that they could be treated as cattle and also autoscale, and the author stated that after doing his homework he determined that for his company Kubernetes offered the most benefits. There isn't much to read from this. > Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"? The author explains that he considered using EC2 instances but after analising the tradeoffs the company decided to just jump to EKS. If you'd like to debate containers vs VMs then it's perfectly fine, but the author of this blog post just documented his learning path and pointed out what worked for him. Sometimes you need to call a shot, and the tradeoffs you need to consider are so minor and irrelevant that a coin flip will do.

toast0 5y ago

How many lines of code is k8s? Depending on it is adding those millions of lines to your tech debt. Unless you need so many of the features, mostly k8s seems to be a way to increase complexity and build resumes. Yes, you need to automate building instances and service discovery, but neither of those tasks needs millions of lines of code.

orf 5y ago
Totally agree. This is why I avoid using Linux because it adds approximately 30 million lines of technical debt to my service, and I definitely don’t need all of the kernel APIs it offers. So I build my own out of assembly. Now my REST api is free of technical debt!

MichaelMoser123 5y ago

Did the stackoverflow architecture change significantly from this series of posts? (seems to mentioning 2016-2019) [1] The article mentions a rewrite, but it wasn't very specific on the scope of the rewrite, and what kind of stack they are using now. I thought they were running their own farm with most of the action on IIS with ASP.net and MSSQL, and some elasticsearch for search. [1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

jimminy 5y ago
The article is a guest post from individuals that work at Pusher[1], about architectural decisions they made for their products and services. I also thought it would be about StackOverflow until they started mentioning all the Pusher related products. [1] https://pusher.com https://pusher.com
- MichaelMoser123 5y ago
  Oh, thanks. Skipped that emtirely. (Too much HN for me)
papito 5y ago
No, Stackoverflow architecture is still a simple server farm with aggressively maintained code and a lite SQL ORM that lets them to highly optimize queries. One of the architects talked about it on their pod a few weeks ago. She mentioned explicitly how they don't jump on new hotness and keep it classic, maintainable, and highly performant without spending tens of millions of dollars on FAANG alumni who seem to think that every shop needs a Google infrastructure because THIS IS THE WAY.

commandlinefan 5y ago

> rewrites should be avoided unless they are truly necessary Which is always.

Thiez 5y ago
It's always necessary, or they should always be avoided?
- commandlinefan 5y ago
  Always necessary.

Hackbraten 5y ago

> Kubernetes has great documentation so new starters can get up to speed more quickly even if they don’t have experience with it. I strongly disagree. What feels easy for one developer may be super hard for others. For me, the Kubernetes tutorial [1] and documentation [2] [3] feels borderline incomprehensible. It introduces several abstract concepts at once without telling me what they really mean. It raises more questions than it answers. For example: > Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit. What does that even mean? Is that “single unit” the entire application? Or rather one component, e. g. the backend? You usually don’t have more than one service per container, right? So what exactly is the cluster here? Why is there not a single example? > The Control Plane is responsible for managing the cluster. The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates. We haven’t even established what the cluster is and how it relates to an application but now we’re already talking about several applications? I feel entirely lost. Ok, let’s look at another introductory page [2]. It says things like: > The worker node(s) host the Pods that are the components of the application workload. But it won’t tell me what a Pod is. So I click on the Pods link [3]. It says: > A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific "logical host": it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host. I find that explanation outright hostile. Not a single example that a newbie can understand. Instead, a pile-up of more and more abstract concepts. I still have no clue how a Pod, a node, an application, its instances, its components and a cluster relates. I’m giving up here. This documentation is abysmal. How can the blog author possibly call this thing well-documented? [1]: https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ https://kubernetes.io/docs/tutorials/kubernetes-basics/creat... [2]: https://kubernetes.io/docs/concepts/overview/components/ https://kubernetes.io/docs/concepts/overview/components/ [3]: https://kubernetes.io/docs/concepts/workloads/pods/ https://kubernetes.io/docs/concepts/workloads/pods/

5y ago

LordGrey 5y ago

With apologies to Jamie Zawinski: > Some people, when confronted with a problem, think "I know, I'll use Kubernetes." Now they have two problems. Kubernetes can be a great solution to a set of problems, but it is not automatically that way. My company is in the midst of adding k8s support to a large legacy application and that work has turned into a deep dive into madness in some places. To be fair, we're making it hard on ourselves by supporting both k8s and the legacy environment, but the point stands. And Helm doesn't help.