3 ms·
On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense. At work, we mov
by jhgg 3y ago
On the other hand, at a certain scale (running hundreds of ES nodes across 80 or so ES clusters), Kubernetes actually does make a lot of sense.
At work, we moved from hosting elastic search on bare VMs to kubernetes. By leveraging scheduler policies we are able to pack / over-provision ES node pods of different clusters onto the same Kubernetes nodes, allowing for far greater resource efficiency, while being able to handle node failure while maintaining availability across all clusters. Additionally, this simplified operations significantly as we can now leverage the operator to do cluster wide operations (e.g rolling restarts, node OS upgrades, ES version upgrades, etc...) fairly easily.
We did, however, go 6 years (and several hundred million users and trillions of documents indexed) without needing to use Kubernetes!
We will blog about this at some point this year.
- jen20 3y agoA big part of the problem with Kubernetes is it doesn't make a ton of sense at small scale, and it just plain doesn't work at large scale. Nomad is generally speaking a much more appropriate technology when you hit the point of needing such a system.
- jhgg 3y agoI would consider our scale pretty large here, and it works just fine.
- marcosdumay 3y agoDid you keep adding and removing replicas into your cluster based on a scheduler policy? How often did you adjust the number of nodes (and how long did it take to make a node available)? At the high-level you are describing your setup, it doesn't make sense. You'd spend way more resources managing any cluster than what you would gain from a normal-looking policy. I seem to be missing some important detail.
- jhgg 3y agoYou must be. It is very easy. We just let kubernetes scheduler take care of everything. We use anti affinities to disallow scheduling nodes from the same ES cluster on the same K8S node. We use the ES operator which will apply the appropriate PDBs to ensure that operations done to the k8s cluster as a whole doesn't end up turning off too many things, and threatening availability. Things literally have just worked. We are about to do a scale up operation of about 6 nodes each for 16 clusters (total of 96 ES data nodes being added.) This was editing a variable in a config file, and pushing that config to kubernetes. The operation will probably take a day to complete as we throttle the speed at which data transfers to new nodes to not adversely affect latency or ingest rate. The amount of "human time" to kick off this operation and begin the scale up is measured in minutes, then it's just letting it do its thing.