4 ms·
Did you keep adding and removing replicas into your cluster based on a scheduler policy? How often did you adjust the number of nodes (and how long did it take
by marcosdumay 3y ago
Did you keep adding and removing replicas into your cluster based on a scheduler policy? How often did you adjust the number of nodes (and how long did it take to make a node available)?
At the high-level you are describing your setup, it doesn't make sense. You'd spend way more resources managing any cluster than what you would gain from a normal-looking policy. I seem to be missing some important detail.
- jhgg 3y agoYou must be. It is very easy. We just let kubernetes scheduler take care of everything. We use anti affinities to disallow scheduling nodes from the same ES cluster on the same K8S node. We use the ES operator which will apply the appropriate PDBs to ensure that operations done to the k8s cluster as a whole doesn't end up turning off too many things, and threatening availability. Things literally have just worked. We are about to do a scale up operation of about 6 nodes each for 16 clusters (total of 96 ES data nodes being added.) This was editing a variable in a config file, and pushing that config to kubernetes. The operation will probably take a day to complete as we throttle the speed at which data transfers to new nodes to not adversely affect latency or ingest rate. The amount of "human time" to kick off this operation and begin the scale up is measured in minutes, then it's just letting it do its thing.