4 ms·

Yes! My startup of 5 people did. We started out with a managed Kubernetes cluster on DigitalOcean, but there were a number of reasons that caused us to not be v

Yes! My startup of 5 people did. We started out with a managed Kubernetes cluster on DigitalOcean, but there were a number of reasons that caused us to not be very comfortable with that setup. - Taking random .yml configs from The InternetTM to install an Nginx Ingress with automatic LetsEncrypt certs felt not-exactly-great. It's no better than piping curl to bash, except the potential impact is not that your computer is dead, but the entirety of prod goes down. - Because of this, upgrades of Kubernetes are a pain. The DigitalOcean admin panel will complain about problems in 'our' configs, that aren't actually OUR configs. We don't know how to fix that, or if ignoring the warnings and upgrading will break our production apps. - Upgrades of Kubernetes itself aren't actually zero downtime, and we couldn't figure out how to do that (even after investing a significant amount of research time). - We were using only a tiny subset of the functionality in Kubernetes. Specifically we wanted high-availability application servers (2+ pods in parallel) with zero-downtime deployments, connecting to a DO managed PostgreSQL instance, with a webserver that does SSL-termination in front of it. - Setting up deployments from a GitLab CI/CD pipeline was pretty hard, and it turned out the functionality for managing a Kubernetes cluster from GitLab was not really done with our use case in mind (I think?). - It would be bad enough if DigitalOcean shit the bed, but the biggest problem was that we couldn't reliably recognize if something was a problem caused by us, or by DO. Try explaining that one to your customers. Summarizing: it was just too complex and fragile, even once you wrap your head around what the hell a Pod, a Deployment, an Ingress and Ingress Controller, and all of the other Kubernetes lingo actually means. I suspect you need a dedicated infra person who knows their stuff to make this work, so it could very well make sense for larger companies, but for our situation it was overkill. We were not intellectually in control of this setup, and I do not feel comfortable running production workloads (systems used by 20k high-school students, mission-critical applications used by logistical companies) on something we couldn't quite grasp. We went to a much simpler setup on Fly.io, and have been happy since. It's a shame they seem to be too young of a company to really be super reliable, but I suspect this is only a matter of time. In terms of feature set, it's all we need.

dimitar 4y ago
Well all of those issues are fixable, but I think it is a totally valid reason not to use k8s if you don't have a dedicated infra person/team.
- superice 4y ago
  Yeah, definitely! This is why I am not that harsh of Kubernetes as a tool at all, I'm just saying that it's not suitable for us for these reasons. In our context of <4 FTE of dev power it just isn't worth the manpower we have to throw at it to make it work, I'd much rather invest that time into moving our core business forward. I might see ourselves moving back to it in the future, but in the meantime we really just need a Heroku / Fly.io / DO apps / AWS ELB or so. At my previous employer (~50 FTE of devs, 2-ish FTE dedicated to infra) Kubernetes worked perfectly fine, and I think in that context it made a lot more sense.
j16sdiz 4y ago
Sounds like you have never understand what you have deployed. Kubernete is complex, you need somebody know how in your team. Meanwhile, going fly.io sounds sensible to me.
- superice 4y ago
  Yep! And we had to make a decision whether we would focus on our core business of developing great applications for end-users, or spend more time running infra and try to wrap our heads around the mountain of complexity that is k8s. That choice at our size is a no-brainer, although that trade-off might be very different for larger teams.
dbingham 4y ago
For context, I ran a DevOps team for the last 4 years that managed two products on AWS - one on EKS and one on ECS. I also just finished building out more or less that exact infrastructure on DO. I can pretty confidently say, that's not K8s, that's Digital Ocean. On AWS, we ran the EKS infrastructure (which was not simple) with basically half a dev's time for years. It was only when it started to scale to millions of users that we needed to build a team to support it. It was still a much smaller team than the one that supported the ECS product (two devops). I was mostly managing and not coding by the time Kubernetes was in our stack, so while I'm very familiar with infrastructure in general (and I know ECS inside and out unfortunately), I hadn't used Kubernetes directly much before I build this DO infrastructure. But I got it up in a week and though DO is a nightmare, k8s is an absolute joy as a DevOps. Holy shit it's perfect. It does exactly what it needs to, with exactly the right abstractions, with perfectly reasonable defaults. The reality is that infrastructure work is just that complicated. You wouldn't try to have a team of front end engineers build your rest backend. It's not reasonable to expect javascript engineers to know how to build and operate an infrastructure - at least not with out dedicating themselves to learning the tooling and space full time for a while. Think of it from the perspective of a frontend engineer learning Python and Django to build out a rest backend, and then multiply the complexity by 4. That's just infrastructure regardless of what you're using. That said, if something like Fly.io can fit your needs, that's great! I haven't used them so I can't speak to them directly, but I know that with Heroku, the trade off was cost and, eventually, being limited in what you could build. Eventually you would need to build something that just couldn't be built with Heroku. A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply.
- superice 4y ago
  That's fair enough. We took a look at 'native' AWS, but there are a multitude of reasons why just dealing with AWS at all is a huge upfront time investment too if you don't hire somebody already skilled at this (complicated billing, just figuring out the product names for their various services, to name a few). > The reality is that infrastructure work is just that complicated. Yes, if you need the flexibility of running anything in any setup. What we really wanted was 'yeet a docker image with a web server in it + env vars at some magic beast that'll run it for me, slap an SSL-cert on it, and make sure it's always online'. We tried to replicate this with Kubernetes, so we got the full complexity of k8s unloaded upon us. Heroku was what we really wanted, but it was always too expensive. Fly.io strikes a good balance here, the defaults are sane, it's still flexible enough for other services, and it's relatively cheap (spend is similar to DO K8s). > You wouldn't try to have a team of front end engineers build your rest backend. Well, yes and no. I wouldn't expect frontend engineers to know the ins and outs of everything backend, but to build on your metaphore a bit further: Setting up a basic Node backend with express serving static files shouldn't take multiple weeks, even for a frontend engineer. I feel like I was trying to do the infra equivalent of that, and it did take me forever. > A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply The build limits could be an issue but really isn't for us right now. It's fairly easy to build locally though (in our case: in our GitLab CI/CD runners)

dimitar 4y ago

Well all of those issues are fixable, but I think it is a totally valid reason not to use k8s if you don't have a dedicated infra person/team.

superice 4y ago
Yeah, definitely! This is why I am not that harsh of Kubernetes as a tool at all, I'm just saying that it's not suitable for us for these reasons. In our context of <4 FTE of dev power it just isn't worth the manpower we have to throw at it to make it work, I'd much rather invest that time into moving our core business forward. I might see ourselves moving back to it in the future, but in the meantime we really just need a Heroku / Fly.io / DO apps / AWS ELB or so. At my previous employer (~50 FTE of devs, 2-ish FTE dedicated to infra) Kubernetes worked perfectly fine, and I think in that context it made a lot more sense.

j16sdiz 4y ago

Sounds like you have never understand what you have deployed. Kubernete is complex, you need somebody know how in your team. Meanwhile, going fly.io sounds sensible to me.

superice 4y ago
Yep! And we had to make a decision whether we would focus on our core business of developing great applications for end-users, or spend more time running infra and try to wrap our heads around the mountain of complexity that is k8s. That choice at our size is a no-brainer, although that trade-off might be very different for larger teams.

dbingham 4y ago

For context, I ran a DevOps team for the last 4 years that managed two products on AWS - one on EKS and one on ECS. I also just finished building out more or less that exact infrastructure on DO. I can pretty confidently say, that's not K8s, that's Digital Ocean. On AWS, we ran the EKS infrastructure (which was not simple) with basically half a dev's time for years. It was only when it started to scale to millions of users that we needed to build a team to support it. It was still a much smaller team than the one that supported the ECS product (two devops). I was mostly managing and not coding by the time Kubernetes was in our stack, so while I'm very familiar with infrastructure in general (and I know ECS inside and out unfortunately), I hadn't used Kubernetes directly much before I build this DO infrastructure. But I got it up in a week and though DO is a nightmare, k8s is an absolute joy as a DevOps. Holy shit it's perfect. It does exactly what it needs to, with exactly the right abstractions, with perfectly reasonable defaults. The reality is that infrastructure work is just that complicated. You wouldn't try to have a team of front end engineers build your rest backend. It's not reasonable to expect javascript engineers to know how to build and operate an infrastructure - at least not with out dedicating themselves to learning the tooling and space full time for a while. Think of it from the perspective of a frontend engineer learning Python and Django to build out a rest backend, and then multiply the complexity by 4. That's just infrastructure regardless of what you're using. That said, if something like Fly.io can fit your needs, that's great! I haven't used them so I can't speak to them directly, but I know that with Heroku, the trade off was cost and, eventually, being limited in what you could build. Eventually you would need to build something that just couldn't be built with Heroku. A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply.

superice 4y ago
That's fair enough. We took a look at 'native' AWS, but there are a multitude of reasons why just dealing with AWS at all is a huge upfront time investment too if you don't hire somebody already skilled at this (complicated billing, just figuring out the product names for their various services, to name a few). > The reality is that infrastructure work is just that complicated. Yes, if you need the flexibility of running anything in any setup. What we really wanted was 'yeet a docker image with a web server in it + env vars at some magic beast that'll run it for me, slap an SSL-cert on it, and make sure it's always online'. We tried to replicate this with Kubernetes, so we got the full complexity of k8s unloaded upon us. Heroku was what we really wanted, but it was always too expensive. Fly.io strikes a good balance here, the defaults are sane, it's still flexible enough for other services, and it's relatively cheap (spend is similar to DO K8s). > You wouldn't try to have a team of front end engineers build your rest backend. Well, yes and no. I wouldn't expect frontend engineers to know the ins and outs of everything backend, but to build on your metaphore a bit further: Setting up a basic Node backend with express serving static files shouldn't take multiple weeks, even for a frontend engineer. I feel like I was trying to do the infra equivalent of that, and it did take me forever. > A quick glance at Fly, the pricing looks reasonable, but I'm guessing the build limits will still apply The build limits could be an issue but really isn't for us right now. It's fairly easy to build locally though (in our case: in our GitLab CI/CD runners)