4 ms·
I've moved to a company that doesn't use Kubernetes at the moment (and that's a 100% calculated and rational decision). What I see, is that a lot of effort is p
by kmac_ 4y ago
I've moved to a company that doesn't use Kubernetes at the moment (and that's a 100% calculated and rational decision). What I see, is that a lot of effort is put to provide functionalities that Kubernetes brings. In case of running a bunch of services, when you wish to do that in a stable and secure way, Kubernetes cuts down running costs. It covers so much cross cutting concerns that reimplementation of those capabilities is not possible unless you have heavy $$$ to spend.
- mountainriver 4y agoSame, we are on ecs and there is a lot of reinventing the wheel
- tootie 4y agoGenuinely curious what things k8s solves that you are reinventing. I run ECS and find that using microservices and their managed offerings (ie RDS, SQS) we don't any complicated topography to do complex work.
- mountainriver 4y ago1. Creating confined development environments containing multiple services. With k8s we can spin up a cluster locally, install all the dependencies and develop on it. 2. Remote development. With k8s we can develop right out of the cluster, ECS has no comparative. 3. Installing OSS software. K8s has loads of supported packages for OSS tooling.
- spmurrayzzz 4y agoI think you're right to point out how much ground k8s covers and to replace every vertical that it integrates could be challenging/costly. But k8s is not a zero-cost abstraction, so I think the calculus here is often more nuanced. In the case of my org, we optimized for the features we thought were valuable and amortized that effort over time. Notably this was early in k8s history (2014/2015), but the fruits of those efforts have aged well so far (8 years or so). Small code footprint to cover service discovery, cert provisioning, release orchestration, and configuration management. The whole devops stack is less than 3k SLOC. Service ecosystem is ~150 distributed systems, roughly about 5 million SLOC, running on just over 1k servers on AWS. I think if the aim is not to completely replace what k8s does, but to cherry pick the features that give you some pareto distribution of value, sometimes its worth it to build in-house. Nothing wrong of course with going with k8s for many orgs, but in our case we didn't have to reinvent the whole wheel to live without it.
- kmac_ 4y ago3k SLOC of devops code to cover a system of that scale is super impressive. And I agree, there's no reason to invest in k8s when only a small fraction of its capabilities is necessary (or we have a team that's already experienced). Otherwise we may end up bending our requirements to k8s abstractions (even though they are well designed).
- roflyear 4y agoAt least for most hosted solutions, Kubernetes seems to be "cheap" (compared to other offerings at that provider) after you pass some reasonable threshold: something like 6-8 services each running 3-4 instances or so. This threshold seems to roughly end up being $500/m.
- rcoder 4y agoWhen I hear numbers like this I wonder what percentage of the compute and memory resources of that 18-32 node cluster — not to mention the engineering that went into making it work — goes into the “hard” problems of horizontal scaling, cramming stateful services into an architecture designed for stateless ones, etc. You can actually get a couple of pretty beefy bare metal boxes for that budget. Or a couple of more modest ones for app servers plus a nice big RDS instance with all the trimmings. Based on past experience, that’ll get you to a few hundred rps for even a fairly complicated, poorly-tuned Rails or PHP app; your well-factored Go API server should handle 10x that pretty easily. You might have to write some Bash or systemd unit files instead of a bunch of YAML, which may or may not bug you. I find shell easier to understand and debug than YAML-based scripting but YMMV.
- roflyear 4y agoRight, I don't think you can beat buying the machines, of course.