3 ms·
Sure, that all checks out, and it's worth clarifying that I've never been all that close to k8s production ops on EKS or elsewhere - my experience with that has
by throwanem 5y ago
Sure, that all checks out, and it's worth clarifying that I've never been all that close to k8s production ops on EKS or elsewhere - my experience with that has always been primarily as a developer, and my perspective is necessarily informed by that, so it's not easy for me to speak in detail on the tradeoffs you describe.
That said, the largest production infrastructures I've worked with have all been hosted in EKS, so I'm not entirely without relevant experience, and that experience does give me still to think that k8s delivers value despite the added overhead and complexity you correctly describe.
The example that comes to mind is a multi-day downtime we took, a few years and one job ago. I don't recall the details in full at this point, but my interpretation of (what I overheard of) what our TAMs were saying is that we had scaled at a rate and to a point where their infra had failed to keep up - in any case, they and we were both clear that the problem was not of our making. Either way, though, none of our EKS-hosted infra was able to meaningfully operate, with nodes going unhealthy almost as soon as they came up.
But despite the EKS downtime lasting as long as it did, we only took a few hours' worth of revenue impact, because we were able quickly to bring up our critical-path services on GKS, cut over our LBs, and get back up and running with just enough functionality to do business. It wasn't perfect, and our CX folks in particular had a somewhat rough go of it, but we got back in business as quickly as we did because we hadn't gone all-in on AWS tools, and because one of the best infra teams I've ever had the privilege of working with had put real thought and planning into "never happen" DR cases like "what if EKS breaks?"
I've worked with some amazing teams, including the one I'm with now. Sure, I don't expect us to be as good at Amazon is in the things Amazon specializes in, and I certainly don't imagine that most teams are too likely to run into the kind of mishap I just described. But most of the stuff you're describing is almost table stakes these days - Amazon, Google, Azure, Tencent, no doubt a dozen others, hell, even DigitalOcean offers most of those capabilities these days, although they're newer enough at it that I wouldn't go straight there for a significant prod workload - I tried a side project on their managed k8s a couple years back and had a lot of trouble, and while I'm sure they're a lot better now, I'd still be a little nervous.
(They're a lot cheaper, though! Or were, last time I looked - cheap enough I didn't think twice about paying out of pocket, which I super would have to weigh carefully with one of the big players. And if my stuff is k8s native rather than platform native, it's a lot easier to think about starting cheap with DO and migrating to AWS when it's time to join the big leagues, right?)
Maybe AWS stays good forever, or maybe it doesn't. For a side project, I'd probably be fine with just using their own-brand offerings. But for a real business? I've never been a founder, and I don't suppose I can really imagine I know. But I do think that, at the point where I'd accrued enough runway to start thinking hard about the future - planning in detail for a horizon of not a month, not a year, but two or three or five years out - I think that'd be the point where I would want very much to prioritize making sure I could keep the doors open to customers even if AWS did one day catch fire, because I've seen that happen before. Sure, if everyone else is AWS-bound too, I'm probably not in worse shape for the outage than they are. But why settle for "not worse" when I can shoot for better? Why be born and die with AWS if, at the price of a little more work early on, I can buy myself a meaningfully longer runway by avoiding big-boy infra costs until I know I need big-boy infra?
I don't really know, I suppose. As I've been at pains to point out in all these conversations today, it's always a tradeoff no matter what you do, and the trick is all in getting the most upside out of whatever choices you make - especially the complicated ones. If I'd been more hands-on with AWS and knew more of what you know, maybe I'd hold an opinion much like yours. But I don't, and this is the thinking that my judgment and experience lead me to.
I think I'm not full of it, but who knows? Maybe I'll go scare up a seed round and we'll find out in the only way that really counts whether or not I'm talking nonsense. :) But in the meantime, kicking it around seems a fine way to pass an afternoon when I'm too ill to work.
- dolni 5y agoThanks for the detailed reply! It's been fun engaging with you. You have really hit the nail on the head about risk, and what it means to go "all-in" on a particular provider. There's a very relevant case to be made for electing to remain provider agnostic if your business is sensitive to outages. There is a balance between avoiding administrative overhead and mitigating risk. I like to think about it like buying insurance (I'd put security posture in the same bucket, too). Pursuing a cloud-agnostic design has some cost per month. A cloud vendor outage has some chance of happening per month, and if it does occur it will cost you $X. The business needs to figure out when: cost(cloud-agnostic design) < risk(cloud vendor outage) * cost(cloud vendor outage) Figuring that out isn't trivial, especially because a large part of cost(cloud-agnostic design) will be in employee hours. At best you probably get a ballpark estimate.