4 ms·

At this scale you automate away 99.9% of the things you respond to. We are not even near this scale yet at my place of work, and we are moving towards this met

by jhgg 4y ago

At this scale you automate away 99.9% of the things you respond to. We are not even near this scale yet at my place of work, and we are moving towards this methodology of strong automation to orchestrate a cluster. We hire software engineers to operate our database clusters, and the expectation is to somewhat be selfishly motivated to write programs to remediate issues so you don't get paged constantly. We do not expect to grow our headcount proportionally to the number of clusters or nodes we operate. You must treat your nodes like cattle not pets. If a node fails, automation kicks in and re-bootstraps it. It is not worth figuring out how to nurse it back to health. When you are performing rolling or scale up operations on the cluster you are just invoking automation to do everything for you.

notacoward 4y ago
> You must treat your nodes like cattle not pets. Do not cite the deep magic to me. I was there when it was created. Really. I know all about Randy's "cattle and pets" metaphor, and automated responses, and not nursing individual nodes back to health, and I had already accounted for those in my comment. I did mention that I'd been on a team responsible for that many nodes, right? <checks> Yep, I did. > If a node fails, automation kicks in and re-bootstraps it. You think we didn't have such systems at Facebook? Yes, we did. And yet, amazingly, there were various kinds of failure or degradation that they didn't always catch or respond to correctly. These kinds of systems only work for syndromes that have already been seen, understood, and coded for. When you have a hundred thousand nodes or more, some nodes regularly fail in novel ways or get stranded in weird automation states. That requires human time to sort out. > you are just invoking automation to do everything for you Maybe that works for teeny-tiny clusters. I hope for your sake that it does. For 10K-node clusters, it's not uncommon for that "rare" condition to occur a few times, requiring a human to prod things along a bit. Yes, the automation does 99% of the work, but 1% can still add up pretty quickly. > We do not expect Only time will tell whether your expectation is realistic. Maybe it is, at your scale. It wouldn't have been at the scale I worked at, which seems a lot closer to what the OP is about.

by jhgg 4y ago

notacoward 4y ago

> You must treat your nodes like cattle not pets. Do not cite the deep magic to me. I was there when it was created. Really. I know all about Randy's "cattle and pets" metaphor, and automated responses, and not nursing individual nodes back to health, and I had already accounted for those in my comment. I did mention that I'd been on a team responsible for that many nodes, right? <checks> Yep, I did. > If a node fails, automation kicks in and re-bootstraps it. You think we didn't have such systems at Facebook? Yes, we did. And yet, amazingly, there were various kinds of failure or degradation that they didn't always catch or respond to correctly. These kinds of systems only work for syndromes that have already been seen, understood, and coded for. When you have a hundred thousand nodes or more, some nodes regularly fail in novel ways or get stranded in weird automation states. That requires human time to sort out. > you are just invoking automation to do everything for you Maybe that works for teeny-tiny clusters. I hope for your sake that it does. For 10K-node clusters, it's not uncommon for that "rare" condition to occur a few times, requiring a human to prod things along a bit. Yes, the automation does 99% of the work, but 1% can still add up pretty quickly. > We do not expect Only time will tell whether your expectation is realistic. Maybe it is, at your scale. It wouldn't have been at the scale I worked at, which seems a lot closer to what the OP is about.