14 ms·

How we migrated onto K8s in less than 12 months

298 points by ianvonseggern 2y ago

jb1991 2y ago
Can anyone advise what is the most common language used in enterprise settings for interfacing with K8s?
- JohnMakin 2y ago
  IME almost exclusively golang.
  - roshbhatia 2y ago
    ++, most controllers are written in go, but there's plenty of client libraries for other languages. A common pattern you'll see though is skipping writing any sort of code and instead using a higher level dsl-ish configuration usually via yaml, using tools like Kyverno.
  - angio 2y ago
    I'm seeing a lot of custom operators written in Rust nowadays. Obviously biased because I do a lot of rust myself so people I'm talking to also do rust.
- gadflyinyoureye 2y ago
  Depends on what you mean. Helm will control a lot. You can make the yaml file in any language. Also you can admin it from command line tools. So again any language but often zsh or bash.
- cortesoft 2y ago
  A lot of yaml
  - yen223 2y ago
    The fun kind of yaml that has a lot of {{ }} in them that breaks your syntax highlighter.
- mplewis 2y ago
  I have seen more Terraform than anything else.
- akdor1154 2y ago
  On the platform consumer side (app infra description) - well schema'd yaml, potentially orchestrated by helm ("templates to hellish extremes") or kustomize ("no templates, this is the hill we will die on"). On the platform integration/hook side (app code doing specialised platform-specific integration stuff, extensions to k8s itself), golang is the lingua franca but bindings for many languages are around and good.
- bithavoc 2y ago
  If you’re talking about connecting to Kubernetes and create resources programmatically, Pulumi allows you to interface with it from all the languages they support(js, ts, go, c#, python) including wrapping up Helm charts and inject secrets( my personal favorite). If you want to build your own Kubernetes Custom Resources and Controllers, Go lang works pretty well for that.
JohnMakin 2y ago
I like how this article clearly and articulately states the reasons it gains to benefit from Kubernetes. Many make the jump without knowing what they even stand to gain, or if they need to in the first place - the reasons given here are good.
- nailer 2y ago
  I was about to write the opposite - the logic is poor and circular - but multiple other commenters have already raised this: https://news.ycombinator.com/item?id=41194506 https://news.ycombinator.com/item?id=41194506 https://news.ycombinator.com/item?id=41194420 https://news.ycombinator.com/item?id=41194420
  - JohnMakin 2y ago
    I don't really see those rebuttals as all that valid. The reasons given in this article are completely valid, from my perspective of someone who's worked heavily with Kubernetes/ECS. Helm, for instance, is a great time saver for installing software. Often software will support nothing but helm. Ease of deployment is a good consideration. Their points on networking are absolutely spot on. The scaling considerations are spot on. Killing/isolating unhealthy containers is completely valid. I could go on a lot more, but I don't see a single point listed as invalid.
  - samcat116 2y ago
    They're quite specific in that they mention that teams would like to make use of existing helm charts for other software products. Telling them to build and maintain definitions for those services from scratch is added work in their mind.
dijksterhuis 2y ago
> When applied, Terraform code would spin up a template of what the service should look like by creating an ECS task set with zero instances. Then, the developer would need to deploy the service and clone this template task set [and do a bunch of manual things] > This meant that something as simple as adding an environment variable required writing and applying Terraform, then running a deploy This sounds less like a problem with ECS and more like an overcomplication in how they were using terraform + ECS to manage their deployments. I get the generating templates part for verification prior to live deploys. But this seems... dunno.
- wfleming 2y ago
  Very much agree. I have built infra on ECS with terraform at two companies now, and we have zero manual steps for actions like this, beyond “add the env var to a terraform file, merge it and let CI deploy”. The majority of config changes we would make are that process.
  - dijksterhuis 2y ago
    Yeah.... thinking about it a bit more i just don't see why they didn't set up their CI to deploy a short lived environment on a push to a feature branch. To me that seems like the simpler solution.
- roshbhatia 2y ago
  I'm with you here -- ECS deploys are pretty painless and uncomplicated, but I can picture a few scenarios where this ends up being necessary, for ex; if they have a lot of services deployed on ECS and it ends up bloating the size of the Terraform state. That'd slow down plans and applies significantly, which makes sharding the Terraform state by literally cloning the configuration based on a template a lot safer.
  - freedomben 2y ago
    > ECS deploys are pretty painless and uncomplicated Unfortunately in my experience, this is true until it isn't. Once it isn't true, it can quickly become a painful blackbox debugging exercise. If your org is big enough to have dedicated AWS support then they can often get help from engineers, but if you aren't then life can get really complicated. Still not a bad choice for most apps though, especially if it's just a run-of-the-mill HTTP-based app
Aeolun 2y ago
Honestly, I find the reasons they name for using Kubernetes flimsy as hell. “ECS doesn’t support helm charts!” No shit sherlock, that’s a thing literally built on Kubernetes. It’s like a government RFP that can only be fulfilled by a single client.
- Carrok 2y ago
  > We also encountered many smaller paper cuts, like attempting to gracefully terminate a single poorly behaving EC2 machine when running ECS on EC2. This is easy on Amazon’s Elastic Kubernetes Service (EKS), which allows you to simply cordon off the bad node and let the API server move the pods off to another machine while respecting their shutdown routines. I dunno, that seems like a very good reason to me.
  - watermelon0 2y ago
    I assume that ECS Fargate would solve this, because one misbehaving ECS task would not affect others, and stopping it should still respect the shutdown routines.
    - ko_pivot 2y ago
      Fargate is very expensive at scale. Great for small or bursty workloads, but when you’re at Figma scale, you almost always go EC2 for cost-effectiveness.
      - ihkasfjdkabnsk 2y ago
        this isn't really true. It was very expensive when it was first released but now it's pretty cost competitive with EC2, especially when you consider easier scale down/up.
      - Aeolun 2y ago
        I think when you are at Figma scale you should have learned to keep things simpler. At this point I don’t think the (slightly) lower costs of EC2 weigh up against the benefits of Fargate.
vouwfietsman 2y ago
Maybe its normal for a company this size, but I have a hard time following much of the decision making around these gigantic migrations or technology efforts because the decisions don't seem to come from any user or company need. There was a similar post from Figma earlier, I think around databases, that left me feeling the same. For instance: they want to go to k8s because they want to use etcd/helm, which they can't on ECS? Why do you want to use etcd/helm? Is it really this important? Is there really no other way to achieve the goals of the company than exactly like that? When a decision is founded on a desire of the user, its easy to validate that downstream decisions make sense. When a decision is founded on a technological desire, downstream decisions may make sense in the context of the technical desire, but do they make sense in the context of the user, still? Either I don't understand organizations of this scale, or it is fundamentally difficult for organizations of this scale to identify and reason about valuable work.
- WaxProlix 2y ago
  People move to K8s (specifically from ECS) so that they can use cloud provider agnostic tooling and products. I suspect a lot of larger company K8s migrations are fueled by a desire to be multicloud or hybrid on-prem, mitigate cost, availability, and lock-in risk.
  - timbotron 2y ago
    there's a pretty direct translation from ECS task definition to docker-compose file
  - zug_zug 2y ago
    I've heard all of these lip-service justifications before, but I've yet to see anybody actually publish data showing how they saved any money. Would love to be proven wrong by some hard data, but something tells me I won't be.
    - nailer 2y ago
      Likewise. I'm not sure Kubernetes famous complexity (and the resulting staff requirements) are worth it to preemptively avoid vendor lockin, and wouldn't be solved more efficiently by migrating to another cloud provider's native tools if the need arises.
tedunangst 2y ago
How long will it take to migrate off?
- codetrotter 2y ago
  It’s irreversible.
- hujun 2y ago
  depends on how much "k8s native" code you have, there are application designed to run on k8s which uses a lot of k8s api and also if you app already micro-serviced, it is also not straight forward to change it back
wrs 2y ago
A migration with the goal of improving the infrastructure foundation is great. However, I was surprised to see that one of the motivations was to allow teams to use Helm charts rather than converting to Terraform. I haven’t seen in practice the consistent ability to actually use random Helm charts unmodified, so by encouraging its use you end up with teams forking and modifying the charts. And Helm is such a horrendous tool, you don’t really want to be maintaining your own bespoke Helm charts. IMO you’re actually better off rewriting in Terraform so at least your local version is maintainable. Happy to hear counterexamples, though — maybe the “indent 4” insanity and multi-level string templating in Helm is gone nowadays?
- smellybigbelly 2y ago
  Our team also suffered from the problems you described of public helm charts. There is always something you need to customise to make things work on your own environment. Our approach has been to use the public helm chart as-is and do any customisation with `kustomize —enable-helm`.
- BobbyJo 2y ago
  Helm is quite often the default supported way of launching containerized third-party products. I have works at two separate startups whose 'on prem' product was offered this way.
  - freedomben 2y ago
    Indeed. I try hard to minimize the amount of Helm we use, but a significant amount of tools are only shipped as Helm charts. Fortunately I'm increasingly seeing people provide "raw k8s" yaml, but it's far from universal.
- cwiggs 2y ago
  Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not. I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places. Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.
andrewguy9 2y ago
I look forward to the blog post where they get off K8, in just 18 months.
surfingdino 2y ago
ECS makes sense when you are building and breaking stuff. K8s makes sense when you are mature (as an org).
xiwenc 2y ago
I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run their apps on VM’s?
- elktown 2y ago
  I think some are just pretty sick and tired of the explosion of needless complexity we've seen in the last decade or so in software, and rightly so. This is an industry-wide problem of deeply misaligned incentives (& some amount of ZIRP gold rush), not specific to this particular case - if this one is even a good example of this to begin with. Honestly, as it stands, I think we'd be seen as pretty useless craftsmen in any other field due to an unhealthy obsession of our tooling and meta-work - consistently throwing any kind of sensible resource usage out of the window in favor of just getting to work with certain tooling. It's some kind of a "Temporarily embarrassed FAANG engineer" situation.
  - cwiggs 2y ago
    I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA. In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it. I blame this logic on the low interest rates for a decade. I could be wrong.
    - loire280 2y ago
      This week we had a few minutes of downtime on an internal service because of a node rotation that triggered an alert. The responding engineer started to put together a plan to make the service HA (which would have tripled the cost to serve). I asked how frequently the service went down and how many people would be inconvenienced if it did. They didn't know, but when we checked the metrics it had single-digit minutes of downtime this year and fewer than a dozen daily users. We bumped the threshold on the alert to longer than it takes for a pod to be re-scheduled and resolved the ticket.
solatic 2y ago
I don't get the hate for Kubernetes in this thread. TFA is from Figma. You can talk all day long about how early startups just don't need the kind of management benefits that Kubernetes offers, but the article isn't written by someone working for a startup, it's written by a company that nearly got sold to Adobe for $20 billion. Y'all really don't think a company like Figma stands to benefit from the flexibility that Kubernetes offers?
- BobbyJo 2y ago
  Kubernetes isn't even that complicated, and first party support from cloud providers often means you're doing something in K8s inleu of doing it in a cloud specific way (like ingress vs cloud specific load balancer setups). At a certain scale, K8s is the simple option. I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.
  - JohnMakin 2y ago
    > I think much of the hate on HN comes from the "ruby on rails is all you need" crowd. Maybe - people seem really gungho about serverless solutions here too
    - dexwiz 2y ago
      The hype for serverless cooled after that article about Prime Video dropping lambda. No one wants a product that a company won’t dogfood. I realize Amazon probably uses lambda elsewhere, but it was still a bad look.
      - LordKeren 2y ago
        I think it was much more about one specific use case of lambda that was a bad fit for the prime video team’s need and not a rejection of lambda/serverless. TBH, it kind of reflected more poorly on the team than lambda as a product
      - cmckn 2y ago
breakingcups 2y ago
I feel so out of touch when I read a blog post which casually mentions 6 CNCF projects with kool names that I've never heard of, for gaining seemingly simple functionality. I'm really wondering if I'm aging out of professional software development.
- renewiltord 2y ago
  Nah, there’s lots of IC work. It just means that you’re unfamiliar with one approach to org scaling: abstracting over hardware, logging, retrying handled by platform team. It’s not the only approach so you may well be familiar with others.
twodave 2y ago
TL;DR because they already ran everything in containers. Having performed a migration where this wasn’t the case, the path from non-containerized to containerized is way more effort than going from containerized non-k8s to k8s.
_pdp_ 2y ago
In my own experience, AWS Fargate is easier, more secure and way more robust then running your K8S even with EKS.
- watermelon0 2y ago
  Do you mean ECS Fargate? Because you can use AWS Fargate with EKS, with some limitations.
  - _pdp_ 2y ago
    Yes, ECS Fargate.
ko_pivot 2y ago
I’m not surprised that the first reason they state for moving off of ECS was the lack of support for stateful services. The lack of integration between EBS and ECS has always felt really strange to me, considering that AWS already built all the logic to integrate EKS with EBS in a StatefulSet compliant way.
- datatrashfire 2y ago
  https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/ https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ec... This was actually added beginning of the year. Definitely was on my most wanted list for a while. You could technically use EFS, but that’s a very expensive way to run anything IO intensive.
  - ko_pivot 2y ago
    This adds support for ephemeral EBS volumes. When a task is created a volume gets created, and when the task is destroyed, for whatever reason, the volume is destroyed too. It has no concept of task identity. If the task needs to be moved to a new host, the volume is destroyed.
deleted 2y ago
[deleted]
julienmarie 2y ago
I personally love k8s. I run multiple small but complex custom e-commerce shops and handle all the tech on top of marketing, finance and customer service. I was running on dedicated servers before. My stack is quite complicated and deploys were a nightmare. In the end the dread of deploying was slowing down the little company. Learning and moving to k8s took me a month. I run around 25 different services ( front ends, product admins, logistics dashboards, delivery routes optimizers, orsm, ERP, recommendation engine, search, etc.... ). It forced me to clean my act and structure things in a repeatable way. Having all your cluster config in one place allows you to exactly know the state of every service, which version is running. It allowed me to do rolling deploys with no downtime. Yes it's complex. As programmers we are used to complex. An Nginx config file is complex as well. But the more you dive into it the more you understand the architecture if k8s and how it makes sense. It forces you to respect the twelve factors to the letter. And yes, HA is more than nice, especially when your income is directly linked to the availability and stability of your stack. And it's not that expensive. I lay around 400 usd a month in hosting.
- belter 2y ago
  > I run multiple small but complex custom e-commerce shops How do you handle the lack of multi tenancy in Kubernetes?
- maccard 2y ago
  Figma were running on ECS before, so they weren't just running dedicated servers. I'm a K8S believer, but it _is_ complicated. It solves hard problems. If you're multi-cloud, it's a no brainer. If you're doing complex infra that you want a 1:1 mapping of locally, it works great. But if you're less than 100 developers and are deploying containers to just AWS, I think you'd be insane to use EKS over ECS + Fargate in 2024.
  - epgui 2y ago
    I don’t know if it’s just me, but I really don’t see how kubernetes is more complex than ECS. Even for a one-man show.
xyst 2y ago
Of course there’s no mention of performance loss or gain after migration. I remember when microservices architecture was the latest hot trend that came off the presses. Small and big firms were racing to redesign/reimplement apps. But most forgot they weren’t Google/Netflix/Facebook. I remember end user experience ended up being _worse_ after the implementation. There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown. There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1). Company didn’t have an easy way to trace logs across different services (way before distributed tracing was a thing). Turns out only a specific condition would trigger the dependency loop (maybe, 1 in 100 requests?). Good times. Also, job safety.
- api 2y ago
  This is a very fad driven industry. One of the things you earn after being in it for a long time is intuition for spotting fads and gratuitous complexity traps.
  - makeitdouble 2y ago
    I think that aspect is indirectly covered, as one of the main motivation was to get on a popular platform that helps hiring. I agree on how it's technically a waste of time to pursue fads, but it's also a huge PITA to have a platform that good engineers actively try to avoid, as their careers would stagnate (even as they themselves know that it's half a fad)
    - sangnoir 2y ago
      I avoid working at organisations with NIH syndrome - if they are below a certain size (i.e. they lack a standing dev-eng team to support their homegrown K8s "equivalent"). Extra red flags if the said homegrown-system was developed by that guy[1] who's ostensibly a genius and has very strong opinions about his system. Give me k8s' YAML-hell any day instead, at least that bloat has transferable skills, amd I can actually Google common resolutions. 1. Has been at org for so long, management condones them flaunting the rules, like pushing straight to prod. Hates the "inefficiency" of open source platforms and purpose-built something "suitable for the company" by themselves, no documentation you have to ask them to fix issues because they don't accept code or suggestions from others. The DSL they developed is inconsistent and has no parser/linter.
jmdots 2y ago
Please just use it as a warehouse scale computer and don’t make mode groups into pets.
jokethrowaway 2y ago
In which universe migrating from docker containers in ECS to Kubernetes is an effort measured in years?
strivingtobe 2y ago
> At the time we did not auto-scale any of our containerized services and were spending a lot of unnecessary money to keep services provisioned such that they could always handle peak load, even on nights and weekends when our traffic is much lower. Huh? You've been running on AWS for how long and haven't been using auto scaling AT ALL? How was this not priority number one for the company to fix? You're just intentionally burning money at that point! > While there is some support for auto-scaling on ECS, the Kubernetes ecosystem has robust open source offerings such as Keda for auto-scaling. In addition to simple triggers like CPU utilization, Keda supports scaling on the length of an AWS Simple Queue Service (SQS) queue as well as any custom metrics from Datadog. ECS autoscaling is easy, and supports these things. Fair play if you just really wanted to use CNCF projects, but this just seems like you didn't really utilize your previous infrastructure very well.
kachapopopow 2y ago
It appears to me that people don't really understand kubernetes here. Kubernetes does not mean microservices, it does not mean containerization and isolation, hell, it doesn't even mean service discovery most of the time. The default smallest kubernetes installation provides you two things: kubelet (the scheduling agent) and kubeapi. What do these two allow you to do? KubeApi provides an API to interact with kubelet instances by telling them to do with manifests. That's all, that's all kubernetes is, just a dumb agent with some default bootstrap behavior that allows you to interact with a backend database. Now, let's get into kubernetes default extensions: - CoreDNS - linking service names to service addresses. - KubeProxy - routing traffic from host to services. - CNI(many options) - Networking between various service resources. After that, kubernetes is whatever you want it to be. It can be something that you can use to spawn few test databases. Deploy an entire production-certified clustered databases. A full distributed fs with automatic device discovery. Deploy backend services if you want to take advantage of service discovery, autoscaling and networking. Or it can be something as small as deploying monitoring (such as node-exporter) to every instance. And as a bonus, it allows you to do it from the comfort of your own local computer. This article says that figma migrated necessary services to kubernetes to improve developer experience and clearly said that things that don't need to be kubernetes aren't. For all we know they still run their services in raw instances and only use kubernetes for their storage and databases. And to add to all of that, kubernetes doesn't care where it runs, which is a great way to increase competition between cloud providers lowering the costs for all.
- tbrownaw 2y ago
  Is it very common to use it without containers?
  - otabdeveloper4 2y ago
    It is impossible.
    - lisnake 2y ago
      it's possible with virtlet or kubevirt
ec109685 2y ago
> At high-growth companies, resources are precious Yeah, at those low growth companies, you have unlimited resources /s
datadeft 2y ago
> Migrating onto Kubernetes can take years What a heck am I reading? For who? I am not sure why companies even bother with such migrations. Where is the business value? Where is the gain for the customer? Is this one of those "L'art pour l'art" project that Figma does it just because they can?
- xorcist 2y ago
  It solves the "we have recently been acquired and have a lot of resources that we must put to use" problem.
- kevstev 2y ago
  FWIW... I was pretty taken aback by this statement as well- and also the "brag" that they moved onto K8s in less than a year. At a very well established firm ~30 years old and with the baggage that came with it, we moved to K8s in far less time- though we made zero attempt to move everything to k8s, just stuff that could benefit from it. Our pitch was more or less- move to k8s and when we do the planned datacenter move at the end of the year, you don't have to do anything aside from a checkout. Otherwise you will have to redeploy your apps to new machines or VMs and deal with all the headache around that. Or you could just containerize now if you aren't already and we take care of the rest. Most migrated and were very happy with the results. There was plenty of services that were latency sensitive or in the HPC realm where it made no sense to force a migration though, and there was no attempt to force them to shoehorn in.
deleted 2y ago
[deleted]
rayrrr 2y ago
Just out of curiosity, is there any other modern system or service that anyone here can think of, where anyone in their right might would brag about migrating to it in less than a year?
- jjice 2y ago
  It's a hard question to answer. Not all systems are equal in size, scope, and impact. K8s as a system is often the core of your infra, meaning everything running will be impacted. That coupled with their team constraints in the article make it sound like a year isn't awful. One system I can think of off the top of my head is when Amazon moved away from Oracle to fully Amazon/OSS RDBMSs a while ago, but that was multi year I think. If they could have done it in less than a year, they'd definitely be bragging.
- therealdrag0 2y ago
  I’ve seen many migrations take over a year. It’s less about the technology and more about your tech debt, integration complexity, and resourcing.
05bmckay 2y ago
I don't think this is the flex they think it is...
ravedave5 2y ago
Completely left out of this post and most of the conversation is that being on K8 makes it much, much easier to go multi-cloud. K8 is k8.
syngrog66 2y ago
k8s and "12 months" -> my priors likely confirmed. ha
sjkoelle 2y ago
The title alone is a teensie bit hilarious
Ramiro 2y ago
I love reading these "reports from the field"; I always pick up a thing or two. Thanks for sharing @ianvonseggern!
jstrong 2y ago
enjoyed the article - provides a comprehensive tour of the benefits of deploying in a tmux session (seamless restarts via ctrl-c, up, enter). for super important stuff, a systemd unit is worth considering.

298 points by ianvonseggern 2y ago

jb1991 2y ago

Can anyone advise what is the most common language used in enterprise settings for interfacing with K8s?

JohnMakin 2y ago
IME almost exclusively golang.
- roshbhatia 2y ago
  ++, most controllers are written in go, but there's plenty of client libraries for other languages. A common pattern you'll see though is skipping writing any sort of code and instead using a higher level dsl-ish configuration usually via yaml, using tools like Kyverno.
- angio 2y ago
  I'm seeing a lot of custom operators written in Rust nowadays. Obviously biased because I do a lot of rust myself so people I'm talking to also do rust.
gadflyinyoureye 2y ago
Depends on what you mean. Helm will control a lot. You can make the yaml file in any language. Also you can admin it from command line tools. So again any language but often zsh or bash.
cortesoft 2y ago
A lot of yaml
- yen223 2y ago
  The fun kind of yaml that has a lot of {{ }} in them that breaks your syntax highlighter.
mplewis 2y ago
I have seen more Terraform than anything else.
akdor1154 2y ago
On the platform consumer side (app infra description) - well schema'd yaml, potentially orchestrated by helm ("templates to hellish extremes") or kustomize ("no templates, this is the hill we will die on"). On the platform integration/hook side (app code doing specialised platform-specific integration stuff, extensions to k8s itself), golang is the lingua franca but bindings for many languages are around and good.
bithavoc 2y ago
If you’re talking about connecting to Kubernetes and create resources programmatically, Pulumi allows you to interface with it from all the languages they support(js, ts, go, c#, python) including wrapping up Helm charts and inject secrets( my personal favorite). If you want to build your own Kubernetes Custom Resources and Controllers, Go lang works pretty well for that.

JohnMakin 2y ago

I like how this article clearly and articulately states the reasons it gains to benefit from Kubernetes. Many make the jump without knowing what they even stand to gain, or if they need to in the first place - the reasons given here are good.

nailer 2y ago
I was about to write the opposite - the logic is poor and circular - but multiple other commenters have already raised this: https://news.ycombinator.com/item?id=41194506 https://news.ycombinator.com/item?id=41194506 https://news.ycombinator.com/item?id=41194420 https://news.ycombinator.com/item?id=41194420
- JohnMakin 2y ago
  I don't really see those rebuttals as all that valid. The reasons given in this article are completely valid, from my perspective of someone who's worked heavily with Kubernetes/ECS. Helm, for instance, is a great time saver for installing software. Often software will support nothing but helm. Ease of deployment is a good consideration. Their points on networking are absolutely spot on. The scaling considerations are spot on. Killing/isolating unhealthy containers is completely valid. I could go on a lot more, but I don't see a single point listed as invalid.
- samcat116 2y ago
  They're quite specific in that they mention that teams would like to make use of existing helm charts for other software products. Telling them to build and maintain definitions for those services from scratch is added work in their mind.

dijksterhuis 2y ago

> When applied, Terraform code would spin up a template of what the service should look like by creating an ECS task set with zero instances. Then, the developer would need to deploy the service and clone this template task set [and do a bunch of manual things] > This meant that something as simple as adding an environment variable required writing and applying Terraform, then running a deploy This sounds less like a problem with ECS and more like an overcomplication in how they were using terraform + ECS to manage their deployments. I get the generating templates part for verification prior to live deploys. But this seems... dunno.

wfleming 2y ago
Very much agree. I have built infra on ECS with terraform at two companies now, and we have zero manual steps for actions like this, beyond “add the env var to a terraform file, merge it and let CI deploy”. The majority of config changes we would make are that process.
- dijksterhuis 2y ago
  Yeah.... thinking about it a bit more i just don't see why they didn't set up their CI to deploy a short lived environment on a push to a feature branch. To me that seems like the simpler solution.
roshbhatia 2y ago
I'm with you here -- ECS deploys are pretty painless and uncomplicated, but I can picture a few scenarios where this ends up being necessary, for ex; if they have a lot of services deployed on ECS and it ends up bloating the size of the Terraform state. That'd slow down plans and applies significantly, which makes sharding the Terraform state by literally cloning the configuration based on a template a lot safer.
- freedomben 2y ago
  > ECS deploys are pretty painless and uncomplicated Unfortunately in my experience, this is true until it isn't. Once it isn't true, it can quickly become a painful blackbox debugging exercise. If your org is big enough to have dedicated AWS support then they can often get help from engineers, but if you aren't then life can get really complicated. Still not a bad choice for most apps though, especially if it's just a run-of-the-mill HTTP-based app

Aeolun 2y ago

Honestly, I find the reasons they name for using Kubernetes flimsy as hell. “ECS doesn’t support helm charts!” No shit sherlock, that’s a thing literally built on Kubernetes. It’s like a government RFP that can only be fulfilled by a single client.

Carrok 2y ago
> We also encountered many smaller paper cuts, like attempting to gracefully terminate a single poorly behaving EC2 machine when running ECS on EC2. This is easy on Amazon’s Elastic Kubernetes Service (EKS), which allows you to simply cordon off the bad node and let the API server move the pods off to another machine while respecting their shutdown routines. I dunno, that seems like a very good reason to me.
- watermelon0 2y ago
  I assume that ECS Fargate would solve this, because one misbehaving ECS task would not affect others, and stopping it should still respect the shutdown routines.
  - ko_pivot 2y ago
    Fargate is very expensive at scale. Great for small or bursty workloads, but when you’re at Figma scale, you almost always go EC2 for cost-effectiveness.
    - ihkasfjdkabnsk 2y ago
      this isn't really true. It was very expensive when it was first released but now it's pretty cost competitive with EC2, especially when you consider easier scale down/up.
    - Aeolun 2y ago
      I think when you are at Figma scale you should have learned to keep things simpler. At this point I don’t think the (slightly) lower costs of EC2 weigh up against the benefits of Fargate.

vouwfietsman 2y ago

Maybe its normal for a company this size, but I have a hard time following much of the decision making around these gigantic migrations or technology efforts because the decisions don't seem to come from any user or company need. There was a similar post from Figma earlier, I think around databases, that left me feeling the same. For instance: they want to go to k8s because they want to use etcd/helm, which they can't on ECS? Why do you want to use etcd/helm? Is it really this important? Is there really no other way to achieve the goals of the company than exactly like that? When a decision is founded on a desire of the user, its easy to validate that downstream decisions make sense. When a decision is founded on a technological desire, downstream decisions may make sense in the context of the technical desire, but do they make sense in the context of the user, still? Either I don't understand organizations of this scale, or it is fundamentally difficult for organizations of this scale to identify and reason about valuable work.

WaxProlix 2y ago
People move to K8s (specifically from ECS) so that they can use cloud provider agnostic tooling and products. I suspect a lot of larger company K8s migrations are fueled by a desire to be multicloud or hybrid on-prem, mitigate cost, availability, and lock-in risk.
- timbotron 2y ago
  there's a pretty direct translation from ECS task definition to docker-compose file
- zug_zug 2y ago
  I've heard all of these lip-service justifications before, but I've yet to see anybody actually publish data showing how they saved any money. Would love to be proven wrong by some hard data, but something tells me I won't be.
  - nailer 2y ago
    Likewise. I'm not sure Kubernetes famous complexity (and the resulting staff requirements) are worth it to preemptively avoid vendor lockin, and wouldn't be solved more efficiently by migrating to another cloud provider's native tools if the need arises.

tedunangst 2y ago

How long will it take to migrate off?

codetrotter 2y ago
It’s irreversible.
hujun 2y ago
depends on how much "k8s native" code you have, there are application designed to run on k8s which uses a lot of k8s api and also if you app already micro-serviced, it is also not straight forward to change it back

wrs 2y ago

A migration with the goal of improving the infrastructure foundation is great. However, I was surprised to see that one of the motivations was to allow teams to use Helm charts rather than converting to Terraform. I haven’t seen in practice the consistent ability to actually use random Helm charts unmodified, so by encouraging its use you end up with teams forking and modifying the charts. And Helm is such a horrendous tool, you don’t really want to be maintaining your own bespoke Helm charts. IMO you’re actually better off rewriting in Terraform so at least your local version is maintainable. Happy to hear counterexamples, though — maybe the “indent 4” insanity and multi-level string templating in Helm is gone nowadays?

smellybigbelly 2y ago
Our team also suffered from the problems you described of public helm charts. There is always something you need to customise to make things work on your own environment. Our approach has been to use the public helm chart as-is and do any customisation with `kustomize —enable-helm`.
BobbyJo 2y ago
Helm is quite often the default supported way of launching containerized third-party products. I have works at two separate startups whose 'on prem' product was offered this way.
- freedomben 2y ago
  Indeed. I try hard to minimize the amount of Helm we use, but a significant amount of tools are only shipped as Helm charts. Fortunately I'm increasingly seeing people provide "raw k8s" yaml, but it's far from universal.
cwiggs 2y ago
Helm Charts and Terraform are different things IMO. Terraform is better used to deploying cloud resources (s3 bucket, EKS cluster, EKS workers, RDS, etc). Sure you can manage your k8s workloads with Terraform, but I wouldn't recommend it. Terraform having state when you already have your start in k8s makes working with Terraform + k8s a pain. Helm is purpose built for k8s, Terraform is not. I'm not a fan of Helm either though, templat-ed yaml sucks, you still have the "indent 4" insanity too. Kustomize is nice when things are simple, but once your app is complex Kustomize is worse than Helm IMO. Try to deploy an app that has a ING, with a TLS cert and external-DNS with Kustomize for multiple environments; you have to patch the resources 3 times instead of just have 1 variable you and use in 3 places. Helm is popular, Terraform is popular so they both are talked a lot, but IMO there is a tool that is yet to become popular that will replace both of these tools.

andrewguy9 2y ago

I look forward to the blog post where they get off K8, in just 18 months.

surfingdino 2y ago

ECS makes sense when you are building and breaking stuff. K8s makes sense when you are mature (as an org).

xiwenc 2y ago

I’m baffled to see so many anti-k8s sentiments on HN. Is it because most commenters are developers used to services like heroku, fly.io, render.com etc. Or run their apps on VM’s?

elktown 2y ago
I think some are just pretty sick and tired of the explosion of needless complexity we've seen in the last decade or so in software, and rightly so. This is an industry-wide problem of deeply misaligned incentives (& some amount of ZIRP gold rush), not specific to this particular case - if this one is even a good example of this to begin with. Honestly, as it stands, I think we'd be seen as pretty useless craftsmen in any other field due to an unhealthy obsession of our tooling and meta-work - consistently throwing any kind of sensible resource usage out of the window in favor of just getting to work with certain tooling. It's some kind of a "Temporarily embarrassed FAANG engineer" situation.
- cwiggs 2y ago
  I agree with this somewhat. The other day I was driving home and I saw a sprinkler head and broke on the side of the road and was spraying water everywhere. It made me think, why aren't sprinkler systems designed with HA in mind? Why aren't there dual water lines with dual sprinkler heads everywhere with an electronic component that detects a break in a line and automatically switches to the backup water line? It's because the downside of having the water spray everywhere, the grass become unhealthy or die is less than how much it would cost to deploy it HA. In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it. I blame this logic on the low interest rates for a decade. I could be wrong.
  - loire280 2y ago
    This week we had a few minutes of downtime on an internal service because of a node rotation that triggered an alert. The responding engineer started to put together a plan to make the service HA (which would have tripled the cost to serve). I asked how frequently the service went down and how many people would be inconvenienced if it did. They didn't know, but when we checked the metrics it had single-digit minutes of downtime this year and fewer than a dozen daily users. We bumped the threshold on the alert to longer than it takes for a pod to be re-scheduled and resolved the ticket.

solatic 2y ago

I don't get the hate for Kubernetes in this thread. TFA is from Figma. You can talk all day long about how early startups just don't need the kind of management benefits that Kubernetes offers, but the article isn't written by someone working for a startup, it's written by a company that nearly got sold to Adobe for $20 billion. Y'all really don't think a company like Figma stands to benefit from the flexibility that Kubernetes offers?

BobbyJo 2y ago
Kubernetes isn't even that complicated, and first party support from cloud providers often means you're doing something in K8s inleu of doing it in a cloud specific way (like ingress vs cloud specific load balancer setups). At a certain scale, K8s is the simple option. I think much of the hate on HN comes from the "ruby on rails is all you need" crowd.
- JohnMakin 2y ago
  > I think much of the hate on HN comes from the "ruby on rails is all you need" crowd. Maybe - people seem really gungho about serverless solutions here too
  - dexwiz 2y ago
    The hype for serverless cooled after that article about Prime Video dropping lambda. No one wants a product that a company won’t dogfood. I realize Amazon probably uses lambda elsewhere, but it was still a bad look.
    - LordKeren 2y ago
      I think it was much more about one specific use case of lambda that was a bad fit for the prime video team’s need and not a rejection of lambda/serverless. TBH, it kind of reflected more poorly on the team than lambda as a product
    - cmckn 2y ago

breakingcups 2y ago

I feel so out of touch when I read a blog post which casually mentions 6 CNCF projects with kool names that I've never heard of, for gaining seemingly simple functionality. I'm really wondering if I'm aging out of professional software development.

renewiltord 2y ago
Nah, there’s lots of IC work. It just means that you’re unfamiliar with one approach to org scaling: abstracting over hardware, logging, retrying handled by platform team. It’s not the only approach so you may well be familiar with others.

twodave 2y ago

TL;DR because they already ran everything in containers. Having performed a migration where this wasn’t the case, the path from non-containerized to containerized is way more effort than going from containerized non-k8s to k8s.

_pdp_ 2y ago

In my own experience, AWS Fargate is easier, more secure and way more robust then running your K8S even with EKS.

watermelon0 2y ago
Do you mean ECS Fargate? Because you can use AWS Fargate with EKS, with some limitations.
- _pdp_ 2y ago
  Yes, ECS Fargate.

ko_pivot 2y ago

I’m not surprised that the first reason they state for moving off of ECS was the lack of support for stateful services. The lack of integration between EBS and ECS has always felt really strange to me, considering that AWS already built all the logic to integrate EKS with EBS in a StatefulSet compliant way.

datatrashfire 2y ago
https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/ https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ec... This was actually added beginning of the year. Definitely was on my most wanted list for a while. You could technically use EFS, but that’s a very expensive way to run anything IO intensive.
- ko_pivot 2y ago
  This adds support for ephemeral EBS volumes. When a task is created a volume gets created, and when the task is destroyed, for whatever reason, the volume is destroyed too. It has no concept of task identity. If the task needs to be moved to a new host, the volume is destroyed.

deleted 2y ago

[deleted]

julienmarie 2y ago

I personally love k8s. I run multiple small but complex custom e-commerce shops and handle all the tech on top of marketing, finance and customer service. I was running on dedicated servers before. My stack is quite complicated and deploys were a nightmare. In the end the dread of deploying was slowing down the little company. Learning and moving to k8s took me a month. I run around 25 different services ( front ends, product admins, logistics dashboards, delivery routes optimizers, orsm, ERP, recommendation engine, search, etc.... ). It forced me to clean my act and structure things in a repeatable way. Having all your cluster config in one place allows you to exactly know the state of every service, which version is running. It allowed me to do rolling deploys with no downtime. Yes it's complex. As programmers we are used to complex. An Nginx config file is complex as well. But the more you dive into it the more you understand the architecture if k8s and how it makes sense. It forces you to respect the twelve factors to the letter. And yes, HA is more than nice, especially when your income is directly linked to the availability and stability of your stack. And it's not that expensive. I lay around 400 usd a month in hosting.

belter 2y ago
> I run multiple small but complex custom e-commerce shops How do you handle the lack of multi tenancy in Kubernetes?
maccard 2y ago
Figma were running on ECS before, so they weren't just running dedicated servers. I'm a K8S believer, but it _is_ complicated. It solves hard problems. If you're multi-cloud, it's a no brainer. If you're doing complex infra that you want a 1:1 mapping of locally, it works great. But if you're less than 100 developers and are deploying containers to just AWS, I think you'd be insane to use EKS over ECS + Fargate in 2024.
- epgui 2y ago
  I don’t know if it’s just me, but I really don’t see how kubernetes is more complex than ECS. Even for a one-man show.

xyst 2y ago

Of course there’s no mention of performance loss or gain after migration. I remember when microservices architecture was the latest hot trend that came off the presses. Small and big firms were racing to redesign/reimplement apps. But most forgot they weren’t Google/Netflix/Facebook. I remember end user experience ended up being _worse_ after the implementation. There was a saturation point where a single micro service called by all of the other micro services would cause complete system meltdown. There was also the case of an “accidental” dependency loop (S1 -> S2 -> S3 -> S1). Company didn’t have an easy way to trace logs across different services (way before distributed tracing was a thing). Turns out only a specific condition would trigger the dependency loop (maybe, 1 in 100 requests?). Good times. Also, job safety.

api 2y ago
This is a very fad driven industry. One of the things you earn after being in it for a long time is intuition for spotting fads and gratuitous complexity traps.
- makeitdouble 2y ago
  I think that aspect is indirectly covered, as one of the main motivation was to get on a popular platform that helps hiring. I agree on how it's technically a waste of time to pursue fads, but it's also a huge PITA to have a platform that good engineers actively try to avoid, as their careers would stagnate (even as they themselves know that it's half a fad)
  - sangnoir 2y ago
    I avoid working at organisations with NIH syndrome - if they are below a certain size (i.e. they lack a standing dev-eng team to support their homegrown K8s "equivalent"). Extra red flags if the said homegrown-system was developed by that guy[1] who's ostensibly a genius and has very strong opinions about his system. Give me k8s' YAML-hell any day instead, at least that bloat has transferable skills, amd I can actually Google common resolutions. 1. Has been at org for so long, management condones them flaunting the rules, like pushing straight to prod. Hates the "inefficiency" of open source platforms and purpose-built something "suitable for the company" by themselves, no documentation you have to ask them to fix issues because they don't accept code or suggestions from others. The DSL they developed is inconsistent and has no parser/linter.

jmdots 2y ago

Please just use it as a warehouse scale computer and don’t make mode groups into pets.

jokethrowaway 2y ago

In which universe migrating from docker containers in ECS to Kubernetes is an effort measured in years?

strivingtobe 2y ago

> At the time we did not auto-scale any of our containerized services and were spending a lot of unnecessary money to keep services provisioned such that they could always handle peak load, even on nights and weekends when our traffic is much lower. Huh? You've been running on AWS for how long and haven't been using auto scaling AT ALL? How was this not priority number one for the company to fix? You're just intentionally burning money at that point! > While there is some support for auto-scaling on ECS, the Kubernetes ecosystem has robust open source offerings such as Keda for auto-scaling. In addition to simple triggers like CPU utilization, Keda supports scaling on the length of an AWS Simple Queue Service (SQS) queue as well as any custom metrics from Datadog. ECS autoscaling is easy, and supports these things. Fair play if you just really wanted to use CNCF projects, but this just seems like you didn't really utilize your previous infrastructure very well.

kachapopopow 2y ago

It appears to me that people don't really understand kubernetes here. Kubernetes does not mean microservices, it does not mean containerization and isolation, hell, it doesn't even mean service discovery most of the time. The default smallest kubernetes installation provides you two things: kubelet (the scheduling agent) and kubeapi. What do these two allow you to do? KubeApi provides an API to interact with kubelet instances by telling them to do with manifests. That's all, that's all kubernetes is, just a dumb agent with some default bootstrap behavior that allows you to interact with a backend database. Now, let's get into kubernetes default extensions: - CoreDNS - linking service names to service addresses. - KubeProxy - routing traffic from host to services. - CNI(many options) - Networking between various service resources. After that, kubernetes is whatever you want it to be. It can be something that you can use to spawn few test databases. Deploy an entire production-certified clustered databases. A full distributed fs with automatic device discovery. Deploy backend services if you want to take advantage of service discovery, autoscaling and networking. Or it can be something as small as deploying monitoring (such as node-exporter) to every instance. And as a bonus, it allows you to do it from the comfort of your own local computer. This article says that figma migrated necessary services to kubernetes to improve developer experience and clearly said that things that don't need to be kubernetes aren't. For all we know they still run their services in raw instances and only use kubernetes for their storage and databases. And to add to all of that, kubernetes doesn't care where it runs, which is a great way to increase competition between cloud providers lowering the costs for all.

tbrownaw 2y ago
Is it very common to use it without containers?
- otabdeveloper4 2y ago
  It is impossible.
  - lisnake 2y ago
    it's possible with virtlet or kubevirt

ec109685 2y ago

> At high-growth companies, resources are precious Yeah, at those low growth companies, you have unlimited resources /s

datadeft 2y ago

> Migrating onto Kubernetes can take years What a heck am I reading? For who? I am not sure why companies even bother with such migrations. Where is the business value? Where is the gain for the customer? Is this one of those "L'art pour l'art" project that Figma does it just because they can?

xorcist 2y ago
It solves the "we have recently been acquired and have a lot of resources that we must put to use" problem.
kevstev 2y ago
FWIW... I was pretty taken aback by this statement as well- and also the "brag" that they moved onto K8s in less than a year. At a very well established firm ~30 years old and with the baggage that came with it, we moved to K8s in far less time- though we made zero attempt to move everything to k8s, just stuff that could benefit from it. Our pitch was more or less- move to k8s and when we do the planned datacenter move at the end of the year, you don't have to do anything aside from a checkout. Otherwise you will have to redeploy your apps to new machines or VMs and deal with all the headache around that. Or you could just containerize now if you aren't already and we take care of the rest. Most migrated and were very happy with the results. There was plenty of services that were latency sensitive or in the HPC realm where it made no sense to force a migration though, and there was no attempt to force them to shoehorn in.

deleted 2y ago

[deleted]

rayrrr 2y ago

Just out of curiosity, is there any other modern system or service that anyone here can think of, where anyone in their right might would brag about migrating to it in less than a year?

jjice 2y ago
It's a hard question to answer. Not all systems are equal in size, scope, and impact. K8s as a system is often the core of your infra, meaning everything running will be impacted. That coupled with their team constraints in the article make it sound like a year isn't awful. One system I can think of off the top of my head is when Amazon moved away from Oracle to fully Amazon/OSS RDBMSs a while ago, but that was multi year I think. If they could have done it in less than a year, they'd definitely be bragging.
therealdrag0 2y ago
I’ve seen many migrations take over a year. It’s less about the technology and more about your tech debt, integration complexity, and resourcing.

05bmckay 2y ago

I don't think this is the flex they think it is...

ravedave5 2y ago

Completely left out of this post and most of the conversation is that being on K8 makes it much, much easier to go multi-cloud. K8 is k8.

syngrog66 2y ago

k8s and "12 months" -> my priors likely confirmed. ha

sjkoelle 2y ago

The title alone is a teensie bit hilarious

Ramiro 2y ago

I love reading these "reports from the field"; I always pick up a thing or two. Thanks for sharing @ianvonseggern!

jstrong 2y ago

enjoyed the article - provides a comprehensive tour of the benefits of deploying in a tmux session (seamless restarts via ctrl-c, up, enter). for super important stuff, a systemd unit is worth considering.