3 ms·
My team maintains a platform that runs in datacenters across four continents, a private cloud. 1000s of VMs We use VMware and have its API hooked up to Ansible
by aprdm 7y ago
My team maintains a platform that runs in datacenters across four continents, a private cloud. 1000s of VMs
We use VMware and have its API hooked up to Ansible as well as Consul and monitoring.
We then make Ansible playbooks available for devs to create services which takes around 5 minutes to run. Since it has consul and monitoring hooked in we automatically create entries in DNS and can auto scale.
We have services running in docker (without kubernetes), consul is ultimately responsible for, based on service metadata, hooking containers to the ingress traffic through the Fabio load balancer.
I would like to understand what we are missing by not using kubernetes, every time I think about it and talk with developers I feel it isn't worth it.
My team would have to maintain an extra layer of complexity that can get us called at night for no clear business gain.
Would love to know what we're missing.
- jeremyjh 7y agoIt sounds like you've written your own Kubernetes. Congratulations, I guess.
- aprdm 7y agoWe have to support a lot of non docker workflows in this platform. Some of the services are more than 10 years old.
- jacques_chester 7y ago> Would love to know what we're missing. Having an upstream who isn't you.
- aprdm 7y agoSorry what do you mean by that? Consul, Ansible, Prometheus and other parts of the stack we maintain all have their upstreams. If you mean running a managed Kubernetes in the cloud, would love to that but for our workflows we have to run on prems for most of the infrastructure. It's a lot of data and compute. A bit like saying google should use aws for their internal services.
- jacques_chester 7y agoI mean assembling the stack yourself, basically, especially since you're adding features by creating particular arrangements out of the ensemble. There are always more edge cases. It's easier to let someone else handle them. You'll note that Google don't manufacture their own RAM, don't operate hydroelectric plants, buy CPUs from Intel and so on. That they could plausibly do any of these things themselves satisfactorily, possibly even better than their suppliers can, doesn't make it the right choice. There is an opportunity cost for doing so. It's economics 101: gains from trade.
- aprdm 7y agoRight and I would love to leverage Kubernetes instead of our own plumbing of the OSS solutions, I just feel that operating it on prems requires a lot of overhead that simpler tools arranged together do not have, also, when they fail it isn't catastrophic for the most part (except for Consul).