6 ms·

Maybe its normal for a company this size, but I have a hard time following much of the decision making around these gigantic migrations or technology efforts be

• WaxProlix 2y ago
  People move to K8s (specifically from ECS) so that they can use cloud provider agnostic tooling and products. I suspect a lot of larger company K8s migrations are fueled by a desire to be multicloud or hybrid on-prem, mitigate cost, availability, and lock-in risk.
  • timbotron 2y ago
    there's a pretty direct translation from ECS task definition to docker-compose file
  • zug_zug 2y ago
    I've heard all of these lip-service justifications before, but I've yet to see anybody actually publish data showing how they saved any money. Would love to be proven wrong by some hard data, but something tells me I won't be.
    • nailer 2y ago
      Likewise. I'm not sure Kubernetes famous complexity (and the resulting staff requirements) are worth it to preemptively avoid vendor lockin, and wouldn't be solved more efficiently by migrating to another cloud provider's native tools if the need arises.
    • bryanlarsen 2y ago
      I'm confident Figma isn't paying published rates for AWS. The transition might have helped them in their rate negotiations with AWS, or it might not have. Hard data on the money saved would be difficult to attribute.
    • jgalt212 2y ago
      True but if AWS knows your lock-in is less locked-in, I'd bet they'd more flexible when contracts are up for renewal. I mean it's possible the blog post's primary purpose was a shot across bow to their AWS account manager.
      • logifail 2y ago
        > it's possible the blog post's primary purpose was a shot across bow to their AWS account manager Isn't it slightly depressing that this explanation is fairly (the most?) plausible?
        • jiggawatts 2y ago
          Our state department of education is one of the biggest networks in the world with about half a million devices. They would occasionally publicly announce a migration to Linux. This was just a Microsoft licensing negotiation tactic. Before he was CEO, Ballmer flew here to negotiate one of the contracts. The discounts were epic.
    • tengbretson 2y ago
      There are large swaths of the b2b space where (for whatever reason) being in the same cloud is a hard business requirement.
      • imtringued 2y ago
        There are good technical reasons for this. Anything latency or throughput sensitive is better done within the same datacenter. There have been submissions about an ffmpeg as a service company and a GPU over TCP company on HN recently that would significantly benefit from 'same cloud'.
    • vundercind 2y ago
      The vast majority of corporate decisions are never justified by useful data analysis, before or after the fact. Many are so-analyzed, but usually in ways that anyone who paid attention in high school science or stats classes can tell are so flawed that they’re meaningless. We can’t even measure manager efficacy to any useful degree, in nearly all cases. We can come up with numbers, but they don’t mean anything. Good luck with anything more complex. Very small organizations can probably manage to isolate enough variables to know how good or bad some move was in hindsight, if they try and are competent at it (… if). Sometimes an effect is so huge for a large org that it overwhelms confounders and you can be pretty confident that it was at least good or bad, even if the degree is fuzzy. Usually, no. Big organizations are largely flying blind. This has only gotten worse with the shift from people-who-know-the-work-as-leadership to professional-managers-as-leadership.
    • Alupis 2y ago
      Why would you assume it's lip-service? Being vendor-locked into ECS means you must pay whatever ECS wants... using k8s means you can feasibly pick up and move if you are forced. Even if it doesn't save money today it might save a tremendous amount in the future and/or provide a much stronger position to negotiate from.
      • greener_grass 2y ago
        Great in theory but in practice when you do K8s on AWS, the AWS stuff leaks through and you still have lock-in.
        • Alupis 2y ago
          Then don't use the AWS stuff. You can bring your own anything that they provide.
          • greener_grass 2y ago
            This requires iron discipline. Maybe with some kind of linter for Terraform / kubectl it could be done.
        • cwiggs 2y ago
          It doesn't have to be that way though. You can use the AWS ingress controller, or you can use ingress-nginx. You can use external secrets operator and tie it into AWS Secrets manager, or you can tie it into 1pass, or Hashicorp Vault. Just like picking EKS you have to be aware of the pros and cons of picking the cloud provider tool or not. Luckily the CNCF is doing a lot for reducing vender lock in and I think it will only continue.
      • elktown 2y ago
        I don't understand why this "you shouldn't be vendor-locked" rationalization is taken at face value at all? 1. The time it will take to move to another cloud is proportional to the complexity of your app. For example, if you're a Go shop using managed persistence are you more vendor locked in any meaningful way than k8s? What's the delta here? 2. Do you really think you can haggle with the fuel-producers like you're MAERKS? No, you're more likely just a car driving around for a gas station with increasingly diminishing returns.
    • WaxProlix 2y ago
      It looks like I'm implying that companies are successful in getting those things from a K8s transition, but I wasn't trying to say that, just thinking of the times when I've seen these migrations happen and relaying the stated aims. I agree, I think it can be a burner of dev time and a burden on the business as devs acquire the new skillset instead of doing more valuable work.
  • OptionOfT 2y ago
    Flexibility was a big thing for us. Many different jurisdictions required us to be conscious of where exactly data was stored & processed. K8s makes this really easy. Don't need to worry whether country X has a local Cloud data center of Vendor Y. Plus it makes hiring so much easier as you only need to understand the abstraction layer. We don't hire people for ARM64 or x86. We have abstraction layers. Multiple even. We'd be fooling us not to use them.
    • dangsux 2y ago
      [dead]
  • fazkan 2y ago
    This, most of it, I think is to support on-prem, and cloud-flexibility. Also from the customers point of view, they can now sell the entire figma "box" to controlled industries for a premium.
  • teyc 2y ago
    People move to K8s so that their resumes and job ads are cloud provider agnostic. Peoples careers stagnate when their employers platform on a home baked tech, or on specific offerings from cloud providers. Employers find Mmoving to a common platform makes recruiting easier.
  • deleted 2y ago
    [deleted]
• samcat116 2y ago
  > I have a hard time following much of the decision making around these gigantic migrations or technology efforts because the decisions don't seem to come from any user or company need I mean the blog post is written by the team deciding the company needs. They explained exactly why they can't easily use etcd on ECS due to technical limitations. They also talked about many other technical limitations that were causing them issues and increasing cost. What else are you expecting?
• Flokoso 2y ago
  Managing 500 or more VMS is a lot of work. Aline the VM upgrade, auth, backup, log rotation etc. With k8s I can give everyone a namespace, policies, volumes, have automatic log aggregation due to demon sets and k8s/cloud native stacks. Self healing and more. It's hard to describe how much better it is.
• ianvonseggern 2y ago
  Hey, author here, I think you ask a good question and I think you frame it well. I agree that, at least for some major decisions - including this one, "it is fundamentally difficult for organizations of this scale to identify and reason about valuable work." At its core we are a platform teams building tools, often for other platform teams, that are building tools that support the developers at Figma creating the actual product experience. It is often harder to reason about what the right decisions are when you are further removed from the end user, although it also gives you great leverage. If we do our jobs right the multiplier effect of getting this platform right impacts the ability of every other engineer to do their job efficiently and effectively (many indirectly!). You bring up good examples of why this is hard. It was certainly an alternative to say sorry we can't support etcd and helm and you will need to find other ways to work around this limitation. This was simply two more data points helping push us toward the conclusion that we were running our Compute platform on the wrong base building blocks. While difficult to reason about, I do think its still very worth trying to do this reasoning well. It's how as a platform team we ensure we are tackling the right work to get to the best platform we can. Thats why we spent so much time making the decision to go ahead with this and part of why I thought it was an interesting topic to write about.
  • felixgallo 2y ago
    I have a constructive recommendation for you and your engineering management for future cases such as this. First, when some team says "we want to use helm and etcd for some reason and we haven't been able to figure out how to get that working on our existing platform," start by asking them what their actual goal is. It is obscenely unlikely that helm (of all things) is a fundamental requirement to their work. Installing temporal, for example, doesn't require helm and is actually simple, if it turns out that temporal is the best workflow orchestrator for the job and that none of the probably 590 other options will do. Second, once you have figured out what the actual goal is, and have a buffet of options available, price them out. Doing some napkin math on how many people were involved and how much work had to go into it, it looks to me that what you have spent to completely rearchitect your stack and operations and retrain everyone -- completely discounting opportunity cost -- is likely not to break even in even my most generous estimate of increased productivity for about five years. More likely, the increased cost of the platform switch, the lack of likely actual velocity accrual, and the opportunity cost make this a net-net bad move except for the resumes of all of those involved.
• wg0 2y ago
  If you haven't broken down your software into 50+ different separate applications written in 15 different languages using 5 different database technologies - you'll find very little use for k8s. All you need is a way to roll out your artifact to production in a roll over or blue green fashion after the preparations such as required database alterations be it data or schema wise.
  • javaunsafe2019 2y ago
    But you do know which problems the k8s abstraction solves, right? Cause it has nothing to do with many languages nor many services but things like discovery, scaling, failover and automation …
    • wg0 2y ago
      If all you have one single application listening on port 8080 with SSL terminated elsewhere, why would you need so many abstractions in first place.
  • imiric 2y ago
    > All you need is a way to roll out your artifact to production in a roll over or blue green fashion after the preparations such as required database alterations be it data or schema wise. Easier said than done. You can start by implementing this yourself and thinking how simple it is. But then you find that you also need to decide how to handle different environments, configuration and secret management, rollbacks, failover, load balancing, HA, scaling, and a million other details. And suddenly you find yourself maintaining a hodgepodge of bespoke infrastructure tooling instead of your core product. K8s isn't for everyone. But it sure helps when someone else has thought about common infrastructure problems and solved them for you.
    • mattmanser 2y ago
      You need to remove a lot of things from that list. Almost all of that functionality is available in build tools that have been available for decades. I want to emphasize the DECADES. And then all you're left with is scaling. Which most business do not need. Almost everything you've written there is a standard feature of almost any CI toolchain, teamcity, Jenkins, Azure DevOps, etc., etc. We were doing it before k8s was even written.
• friendly_deer 2y ago
  Here's a theory about why at least some of these come about: https://lethain.com/grand-migration/ https://lethain.com/grand-migration/
• deleted 2y ago
  [deleted]
• lmm 2y ago
  > For instance: they want to go to k8s because they want to use etcd/helm, which they can't on ECS? Why do you want to use etcd/helm? Is it really this important? Is there really no other way to achieve the goals of the company than exactly like that? I'm no fan of Helm, but there are surprisingly few good alternatives to etcd (i.e. highly available but consistent datastores, suitable for e.g. the distributed equivalent of a .pid file) - Zookeeper is the only one that comes to mind, and it's a real pain on the ops side of things, requiring ancient JVM versions and being generally flaky even then.