8 ms·
> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applicatio
by ajross 8y ago
> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applications within our Kubernetes clusters.
Sounds big. But then per wikipedia:
> uSwitch.com [...] allows consumers to compare prices for a range of energy, personal finance, insurance and communications services.
And:
> On 30 April 2015, the property website firm Zoopla agreed to purchase uSwitch from LDC for £160 million
So... a low bandwidth business (we're hardly talking Netflix here!) doing maybe, what, $10M in revenue annually and not growing fast enough to justify venture investment or IPO funding (they were a private acquisition!)...
Seriously, I'm sure they like it. But do they really, truly need Kubernetes? This really sounds like the kind of scale that can be achieved with 2-3 hand-managed servers, or maybe twice that number of AWS boxen.
- marcc 8y agoShould any business that has SLAs and needs to be reliable ever rely on “2-3 hand-managed servers”? No. Kubernetes isn’t only about scale. It also provides rolling upgrades and rollbacks. And failover. And DNS based service discovery. And there’s more. You can find solutions to these without Kubernetes but a lot of Kubernetes use is to get these, not simply for scale issues.
- oso2k 8y agoFull-disclosure: I’m a Consulting Architect at Red Hat focused on OpenShift. I like to say Kubernetes & OpenShift focus on availability of the cluster and applications as their primary concern. Many other concerns my customers want to impose are actually detrimental to the goal(s) they’re trying to achieve.
- bonesss 8y ago> And there’s more Being able to reliably create your entire infrastructure on another platform in 'minutes', for example. Not to mention applications architected from the ground up around cloud-friendly and scale-friendly primitives... 2-3 hand-managed servers are great, but will absolutely warp your application and will slowly accrue configuration cruft. That's not terrible, but for many Real World issues portability and freedom to fire up wholly valid test-environments are game-changers. Even the acquisition stories are nicer.
- detaro 8y agoOn the other hand, operating a bunch of price comparison sites sounds like the kind of thing that has tons of small moving parts for data import and such, which seems like a reasonable fit for an orchestration system like this. Especially if there's 30 teams making pieces of it.
- accnumnplus1 8y ago30 teams' services fit on 2-3 servers? We don't have anywhere near 30 teams, but at my last count end of last year there were ~250 repos, and everyday I get auto-emailed telling me I've been subscribed to a new repo. No way could we have only 2-3 servers.
- tonyarkles 8y agoI'm perpetually curious about what kind of resource usage you'd expect given those 250 repos, and which runtime you're deploying. I've recently been playing with Go and looking at existing JVM-based services (Spring Boot + starters + our microservices). We've seen rewrites of small services change from ~500MB of RAM to ~10MB of RAM. 250 Go-based services would probably fit very comfortably on a small cluster of cheap-ish servers, depending on what they're doing and how much traffic they're handling.
- chairmanwow 8y agoCan you go into detail about what things changed in the rewrites that enabled this reduction in RAM usage?
- jacques_chester 8y agoThe JVM is traditionally memory-hungry. There are lines of engineering underway which will change that quite a bit (GraalVM looks particularly promising), but for now seeing a JVM process merrily consuming hundreds of Mb of RAM as the base case is not unusual.
- tonyarkles 8y agoThat's exactly it. Plus add in all of the Spring Boot stuff (autodiscovery/autowiring, behaviour that gets added simply because a .jar is on the classpath, etc etc) and you end up with a pretty large footprint for a pretty tiny app. For reference too, the deployment artifacts with Go are ~5MB vs. a 70MB .war file. We've also just used really simple packages (the built-in net/http server for Go vs. embedded Tomcat, gorilla/mux vs. the Spring class annotations for routing), and I suspect that dramatically cuts down on the footprint too.
- manigandham 8y agoScale isn't everything. Removing "hand-managed" from the entire process is an even bigger benefit. It's really not that complicated, it's clustering software that lets you encapsulate your applications into containers and just write simple declarative YAML files while it takes care of actually running it as specified, and keeping it running regardless of what happens to the hardware. Given all the managed offerings now with free master nodes, why would you purposely take on more ops overhead?
- tannhaeuser 8y ago> simple declarative YAML files If the config files and underlying infrastructure are changing all the time and have varying degrees of documentation, that's just shifting know-how from established tools to the newest fad, especially when k8s know-how is extremely scarce/expensive, and will leave you in a trial-and-error situation with unclear diagnostics if anything goes wrong due to the sheer complexity. Automation is of course not limited to k8s at all. I've witnessed moving perfectly running Terraform-like setups to more expensive k8s setups by admins just so that k8s appears on their resume. There are valid reasons to use k8s, mesos or openshift in big shops, but at this point k8s is just oversold and overhyped IMHO, and not a good match for startups ("our k8s guy comes next tuesday"). Especially when it only allows Docker containers, which is another "political" marchitecture solution with awkward technical constraints to compensate rather than a solution based on merit IMHO.
- manigandham 8y ago> k8s is just oversold and overhyped Yes, like anything else is, but it also removes almost all ops overhead and is much faster and easier to maintain at the container level than recreating images and redeploying VMs. Since the vast majority of startups just want to run apps and aren't doing any sophisticated lower-level infrastructure, trading terraform/chef/puppet for k8s yaml is a net win, especially since it also replaces several other accessory software like load balancers, service discovery, rolling deployments, etc. that you might need otherwise.
- 8y ago
- twblalock 8y agoHand-managed servers should not exist anymore outside of proof-of-concept work. There is no shortage of tools to automate provisioning and deployment, and Kubernetes is one of many options. Bringing up servers should be as automated and reproducible as your CI pipeline for building software -- when you discover problems, go ahead and fix them manually, but then you should update your automated processes to include them.
- tnolet 8y agoYour tone is snarky and dismissive, but I kinda understand what you are saying and it is very relevant in essence. Kubernetes as fashion is a thing and for many teams probably a trap where the time/money invested in the IT infrastructure has no relation to the business value it brings.
- lima 8y agoI've seen way too many companies who started out with 2-3 hand-managed servers and suddenly found themselves with 100 hand-managed servers. Configuration management and deployment is one of these things you should get right from the start.
- erikb 8y agoInteresting to see this post downvoted. In my 1.5 years hands on experience with k8s I found that most of the problems we faced (even in much bigger scenarios) would have been solved quicker with simple Linux administration. We also found that most people don't just run k8s but run it on VMs, with each k8s node exactly being one VM. So the overhead is even bigger to employ k8s in these scenarios (instead of reducing overhead which is one of the key claims of using containers vs VMs).
- mmt 8y ago> 2-3 hand-managed servers I think too many people in this sub-thread are harping on this excerpt, especially with regard to the "hand-managed" and failing to take the most charitable reading, as exhorted by the guidelines. What commenters are, perhaps, failing to grasp, is that if there are only 3 (or even 6) servers in the environment, and each one has a unique configuration, then they're all, by some definition, hand-managed. No amount of "automation" will reduce that fundamental administrative/cognitive burden. Sure, there are some sensible best practices, even for that scenario, like, at minimum, storing config in version control. However, going all-in on a CM system (or, perhaps, Kubernetes) for 100 servers when all you have is 3 is, at best, premature optimization. There's a well-known aphorism in tech about that.
- ajross 8y agoHeh, yeah. I realized after the third reply that my phrasing had turned into a downvote magnet. There's also the problem that the cluster itself becomes a failure point. It too needs to run on physical servers that need to be provisioned and managed. Anyone who can fat-finger a regular iron server configuration can muck up a kubernetes deployment. Ultimately, the top of the stack is always "hand-managed" in some sense. Even AWS has had user-induced failures.