4 ms·

The operational complexity of managing thousands of clusters must be mind boggling. I've been on two projects managing dozens of storage clusters, the second wi

The operational complexity of managing thousands of clusters must be mind boggling. I've been on two projects managing dozens of storage clusters, the second with more data than this in some individual clusters and adding up almost this many total nodes. There were technical problems with scaling up to 10K nodes per cluster, but the operational issues mostly scaled according to number of clusters. For example, how many alerts per hour/day can you stand? Too many and you're overwhelmed; too few and you miss stuff. Walking that fine line became successively more difficult as clusters were added. Same thing with graphs and dashboards. Also, when your storage and IOPS are siloed this much you have no elasticity, so you're going to be chasing capacity or load problems much more often. On the plus side, this probably means each tenant has their own cluster, so you don't have so many worries about them affecting each other. The big question I'd have is: how many people (including on client teams) does it take to manage this much sprawl?

sdevoid 4y ago
I'm going to guess somewhere between 50 and 400 people. Smaller than 50: this is probably an org where "we maintain the source code and documentation. Running, debugging, and scaling your cluster is YOUR problem". Larger than 400: the org probably fulfills other functions besides "running thousands of production Cassandra clusters." This is a lot, and is one of the big things that surprised me when I first joined a large organization. But here's an example breakdown in no particular order: - 10 to 30 people for managing, contributing, and maintaining the open source aspect. Just a handful of engineers could contribute features and fixes to the project, but once the open source project and community gets larger it becomes a full time job. System component maintainers, foundation boards, committees, conferences, etc. add up. - 10 to 70 people for "operations". As you mentioned, the load here tends to scale with the number of clusters (customers). At the large end this is several teams, with say a team dedicated to (a) fleet-management of the individual machines in a cluster, (b) cluster lifecycle management, and (c) macro level operations above the cluster level. Alerts can't all go to this team, so some of the work is writing alerts that go to customers in a self-serve model. - 10 - 40 people for "scale projects." At this scale you have 1 - 10 customers that are on the verge of toppling the system over. They've grown and hit various system bottlenecks that need to be addressed. And you'd be lucky if they're all hitting the same bottleneck. With this many customers, it's likely that they've all adopted orthogonal anti-patterns that you need to find a fix for: too many rows, too many columns, bad query optimization, too many schema changes, too many cluster creates and destroys, etc. So you probably have multiple projects ongoing for these. - 10 - 30 people for "testing infrastructure". Everyone writes unit tests, but once you get to integration and scale testing, you need a team that writes and maintains fixtures to spin up a small cluster for some tests and a large cluster for scale tests (which your "scale projects" teams need, btw). And your customers probably need ways of getting access to small test Cassandra clusters (or mocks of the same) for THEIR integration and scale tests, since Cassandra is just a small part of their system. - 10 - 30 people for automating resource scaling and handling cost attribution. These may not be one function, but I'm lumping them together. "Operations" might handle some of the resource scale problems, but at some point it's probably worth a team to continually look for ways to manage the multi-dimensional scaling problem that large cluster software systems inevitably create. (Is it better to have few large nodes, or many small nodes?) You need some way of attributing cost back to customer organizations, otherwise you're paying $50M because one engineer on the weather team forgot to tear down a test cluster in one automated test 6 months ago and... You need to make sure that growth projections for customers are defined and tracked so you have enough machines on hand. - I'll add that it'll be worth adding whole teams for some of the more complex internal bits of this system, even if the actual rate of change in that sub-system is not very high. At this scale organizations need to optimize for stability, not efficiency. You don't want to be in the situation where the only person who understands the FizzBuzz system leaves and now dozens of people/projects are blocked because nobody understands how to safely integrate changes into FizzBuzz. - Things not covered: security, auditing, internal documentation, machine provisioning, datacenter operations, operating system maintenance, firmware maintenance, new hardware qualification, etc. Maybe there's an entire organization dedicated to each of these, in which case you get it for free. If not, some of your time needs to be spent on these. (Even "free" might have a cost as you need to integrate with those services and update when those services change.)
- notacoward 4y ago
  Spot on, and thank you. My second team was ~40 (might have peaked at ~50) split across four sub-teams, for software that ran at similar scale and was designed and developed to rely heavily on other in-house infra. Maybe half a dozen people on adjacent teams (including customers) who had more than trivial knowledge of our system. Some in our team were almost pure developers, some were almost pure operators, most were at various points in between. I think the reason you and I (we know each other on Twitter BTW) are so at odds with some of the other commenters is that they haven't maxed out on automation yet and don't realize that's A Thing. Automation is absolutely fantastic and essential for running anything at this scale, but it's no panacea. While it usually helps you get more work done faster, sometimes it causes damage faster. Some of our most memorable incidents involved automation run amok, like suddenly taking down 1000 machines in a cluster for trivial reasons or even false alarms while we were already fighting potential-data-loss or load-storm problems. That, in turn, was largely the result of the teams responsible for that infra only thinking about ephemeral web workers and caches, hardly even trying to understand the concerns of permanent data storage. But I digress. The point, still, is that when you've maxed out on what automation can do for you, your remaining workload tends to scale by cluster. And having thousands instead of dozens of clusters sounds like a nightmare. There are many ways to scale such systems. Increasing the size of individual clusters sure as hell ain't easy - I joined that team because it seemed like a hard and fun challenge - but ultimately it pays off by avoiding the operational challenge of Too Many Clusters.
  - sdevoid 4y ago
    Exactly. - There's nothing scarier than "the automation had no rate-limiting or health-checking". Of course, what do we mean by automation? At some point it becomes impractical to slow every change down to a crawl, so some judgement is required. But "slow enough to mount a human response to stop it" is the standard I've applied to critical systems. - Thankfully I've avoided having to support "real" storage systems. The challenges of "simple" distributed databases storing is enough for me. :-) On the "pets vs. cattle" metaphor, I think most people fail to grok the second component of that. I don't think there are many successful cattle ranchers that will just "take the cattle out of line and shoot it in the head." The point of the metaphor is: When you have thousands of head of cattle, you need to think about the problems of feeding and caring for cattle operationally, not as a one-off. Despite what https://xkcd.com/1737 https://xkcd.com/1737 might make one believe, people don't just throw out servers when one part goes bad, or (intentionally) burn down datacenters. What the "hyperscalers" do is decouple the operational challenges of running machines from the operational challenges of running services (or at least try to). Of course this results in a TON of work on both ends.
redanddead 4y ago
They for sure have in-house software to handle these nodes. The software may be pretty good at what it does considering they know what their own DBs are susceptible to, that might reduce a whole bunch of the human management and the team they need may be not that big but likely highly skilled.
magnawave 4y ago
One of the hard parts of quantifying that, is you have people who wear many hats. So sure you have Cassandra gurus, and probably a decent number of them. But this is in the league where really hardcore automation kicks in to keep staffing sane, and operations possible. But outside that, how do you count datacenter folks, client folks, networking folks, etc who only spend a little fraction of time on the database parts. But I think I can say with sufficient knowledge, all things considered still, “way fewer than you might think”.
jhgg 4y ago
At this scale you automate away 99.9% of the things you respond to. We are not even near this scale yet at my place of work, and we are moving towards this methodology of strong automation to orchestrate a cluster. We hire software engineers to operate our database clusters, and the expectation is to somewhat be selfishly motivated to write programs to remediate issues so you don't get paged constantly. We do not expect to grow our headcount proportionally to the number of clusters or nodes we operate. You must treat your nodes like cattle not pets. If a node fails, automation kicks in and re-bootstraps it. It is not worth figuring out how to nurse it back to health. When you are performing rolling or scale up operations on the cluster you are just invoking automation to do everything for you.
- notacoward 4y ago
  > You must treat your nodes like cattle not pets. Do not cite the deep magic to me. I was there when it was created. Really. I know all about Randy's "cattle and pets" metaphor, and automated responses, and not nursing individual nodes back to health, and I had already accounted for those in my comment. I did mention that I'd been on a team responsible for that many nodes, right? <checks> Yep, I did. > If a node fails, automation kicks in and re-bootstraps it. You think we didn't have such systems at Facebook? Yes, we did. And yet, amazingly, there were various kinds of failure or degradation that they didn't always catch or respond to correctly. These kinds of systems only work for syndromes that have already been seen, understood, and coded for. When you have a hundred thousand nodes or more, some nodes regularly fail in novel ways or get stranded in weird automation states. That requires human time to sort out. > you are just invoking automation to do everything for you Maybe that works for teeny-tiny clusters. I hope for your sake that it does. For 10K-node clusters, it's not uncommon for that "rare" condition to occur a few times, requiring a human to prod things along a bit. Yes, the automation does 99% of the work, but 1% can still add up pretty quickly. > We do not expect Only time will tell whether your expectation is realistic. Maybe it is, at your scale. It wouldn't have been at the scale I worked at, which seems a lot closer to what the OP is about.
notacoward 4y ago
Funny how every single reply missed the part where I mentioned I have worked at this scale. Larger in some ways. I know what that kind of automation looks like, and what it can do for you. My point is that, even with that, managing thousands of smaller clusters is going to involve way more operational work than running fewer larger clusters (based on appropriate technology). Try to address that, not the strawman of someone who might suggest trying to do this without modern automation.
deleted 4y ago
[deleted]
kshacker 4y ago
> how many people (including on client teams) does it take to manage this much sprawl? Being remotely familiar something with similar complexity, I say "very few". Automation is not one time, it is ongoing. You start with something small, and you build, and build, and build. I saw a presentation from maybe 5-6 years back talking about Apple Cassandra, and have talked to some engineers too when they used to have DataStax Cassandra summits. Their automation has probably scaled up from those days. Maybe their infrastructure grows by X% every year (X probably being a double digit number) and their automation increases their productivity / reduces their overhead by X/2%, it allows them to still retain a huge growth trajectory at a smaller increase in people costs. Improve productivity by X% or more (with new automation), and you have hit a home run. And they should not be asking "by what percentage should I optimize", but simply "where do we spend effort today and how can we reduce it" ... which any business or group should be doing it anyways. The biggest challenges with such things come when management changes strategy for various reasons because then you lose a year or 2, and then you fall really behind and need to do so much catch up just to catch breath.

sdevoid 4y ago

I'm going to guess somewhere between 50 and 400 people. Smaller than 50: this is probably an org where "we maintain the source code and documentation. Running, debugging, and scaling your cluster is YOUR problem". Larger than 400: the org probably fulfills other functions besides "running thousands of production Cassandra clusters." This is a lot, and is one of the big things that surprised me when I first joined a large organization. But here's an example breakdown in no particular order: - 10 to 30 people for managing, contributing, and maintaining the open source aspect. Just a handful of engineers could contribute features and fixes to the project, but once the open source project and community gets larger it becomes a full time job. System component maintainers, foundation boards, committees, conferences, etc. add up. - 10 to 70 people for "operations". As you mentioned, the load here tends to scale with the number of clusters (customers). At the large end this is several teams, with say a team dedicated to (a) fleet-management of the individual machines in a cluster, (b) cluster lifecycle management, and (c) macro level operations above the cluster level. Alerts can't all go to this team, so some of the work is writing alerts that go to customers in a self-serve model. - 10 - 40 people for "scale projects." At this scale you have 1 - 10 customers that are on the verge of toppling the system over. They've grown and hit various system bottlenecks that need to be addressed. And you'd be lucky if they're all hitting the same bottleneck. With this many customers, it's likely that they've all adopted orthogonal anti-patterns that you need to find a fix for: too many rows, too many columns, bad query optimization, too many schema changes, too many cluster creates and destroys, etc. So you probably have multiple projects ongoing for these. - 10 - 30 people for "testing infrastructure". Everyone writes unit tests, but once you get to integration and scale testing, you need a team that writes and maintains fixtures to spin up a small cluster for some tests and a large cluster for scale tests (which your "scale projects" teams need, btw). And your customers probably need ways of getting access to small test Cassandra clusters (or mocks of the same) for THEIR integration and scale tests, since Cassandra is just a small part of their system. - 10 - 30 people for automating resource scaling and handling cost attribution. These may not be one function, but I'm lumping them together. "Operations" might handle some of the resource scale problems, but at some point it's probably worth a team to continually look for ways to manage the multi-dimensional scaling problem that large cluster software systems inevitably create. (Is it better to have few large nodes, or many small nodes?) You need some way of attributing cost back to customer organizations, otherwise you're paying $50M because one engineer on the weather team forgot to tear down a test cluster in one automated test 6 months ago and... You need to make sure that growth projections for customers are defined and tracked so you have enough machines on hand. - I'll add that it'll be worth adding whole teams for some of the more complex internal bits of this system, even if the actual rate of change in that sub-system is not very high. At this scale organizations need to optimize for stability, not efficiency. You don't want to be in the situation where the only person who understands the FizzBuzz system leaves and now dozens of people/projects are blocked because nobody understands how to safely integrate changes into FizzBuzz. - Things not covered: security, auditing, internal documentation, machine provisioning, datacenter operations, operating system maintenance, firmware maintenance, new hardware qualification, etc. Maybe there's an entire organization dedicated to each of these, in which case you get it for free. If not, some of your time needs to be spent on these. (Even "free" might have a cost as you need to integrate with those services and update when those services change.)

notacoward 4y ago
Spot on, and thank you. My second team was ~40 (might have peaked at ~50) split across four sub-teams, for software that ran at similar scale and was designed and developed to rely heavily on other in-house infra. Maybe half a dozen people on adjacent teams (including customers) who had more than trivial knowledge of our system. Some in our team were almost pure developers, some were almost pure operators, most were at various points in between. I think the reason you and I (we know each other on Twitter BTW) are so at odds with some of the other commenters is that they haven't maxed out on automation yet and don't realize that's A Thing. Automation is absolutely fantastic and essential for running anything at this scale, but it's no panacea. While it usually helps you get more work done faster, sometimes it causes damage faster. Some of our most memorable incidents involved automation run amok, like suddenly taking down 1000 machines in a cluster for trivial reasons or even false alarms while we were already fighting potential-data-loss or load-storm problems. That, in turn, was largely the result of the teams responsible for that infra only thinking about ephemeral web workers and caches, hardly even trying to understand the concerns of permanent data storage. But I digress. The point, still, is that when you've maxed out on what automation can do for you, your remaining workload tends to scale by cluster. And having thousands instead of dozens of clusters sounds like a nightmare. There are many ways to scale such systems. Increasing the size of individual clusters sure as hell ain't easy - I joined that team because it seemed like a hard and fun challenge - but ultimately it pays off by avoiding the operational challenge of Too Many Clusters.
- sdevoid 4y ago
  Exactly. - There's nothing scarier than "the automation had no rate-limiting or health-checking". Of course, what do we mean by automation? At some point it becomes impractical to slow every change down to a crawl, so some judgement is required. But "slow enough to mount a human response to stop it" is the standard I've applied to critical systems. - Thankfully I've avoided having to support "real" storage systems. The challenges of "simple" distributed databases storing is enough for me. :-) On the "pets vs. cattle" metaphor, I think most people fail to grok the second component of that. I don't think there are many successful cattle ranchers that will just "take the cattle out of line and shoot it in the head." The point of the metaphor is: When you have thousands of head of cattle, you need to think about the problems of feeding and caring for cattle operationally, not as a one-off. Despite what https://xkcd.com/1737 https://xkcd.com/1737 might make one believe, people don't just throw out servers when one part goes bad, or (intentionally) burn down datacenters. What the "hyperscalers" do is decouple the operational challenges of running machines from the operational challenges of running services (or at least try to). Of course this results in a TON of work on both ends.

redanddead 4y ago

They for sure have in-house software to handle these nodes. The software may be pretty good at what it does considering they know what their own DBs are susceptible to, that might reduce a whole bunch of the human management and the team they need may be not that big but likely highly skilled.

magnawave 4y ago

One of the hard parts of quantifying that, is you have people who wear many hats. So sure you have Cassandra gurus, and probably a decent number of them. But this is in the league where really hardcore automation kicks in to keep staffing sane, and operations possible. But outside that, how do you count datacenter folks, client folks, networking folks, etc who only spend a little fraction of time on the database parts. But I think I can say with sufficient knowledge, all things considered still, “way fewer than you might think”.

jhgg 4y ago

At this scale you automate away 99.9% of the things you respond to. We are not even near this scale yet at my place of work, and we are moving towards this methodology of strong automation to orchestrate a cluster. We hire software engineers to operate our database clusters, and the expectation is to somewhat be selfishly motivated to write programs to remediate issues so you don't get paged constantly. We do not expect to grow our headcount proportionally to the number of clusters or nodes we operate. You must treat your nodes like cattle not pets. If a node fails, automation kicks in and re-bootstraps it. It is not worth figuring out how to nurse it back to health. When you are performing rolling or scale up operations on the cluster you are just invoking automation to do everything for you.

notacoward 4y ago
> You must treat your nodes like cattle not pets. Do not cite the deep magic to me. I was there when it was created. Really. I know all about Randy's "cattle and pets" metaphor, and automated responses, and not nursing individual nodes back to health, and I had already accounted for those in my comment. I did mention that I'd been on a team responsible for that many nodes, right? <checks> Yep, I did. > If a node fails, automation kicks in and re-bootstraps it. You think we didn't have such systems at Facebook? Yes, we did. And yet, amazingly, there were various kinds of failure or degradation that they didn't always catch or respond to correctly. These kinds of systems only work for syndromes that have already been seen, understood, and coded for. When you have a hundred thousand nodes or more, some nodes regularly fail in novel ways or get stranded in weird automation states. That requires human time to sort out. > you are just invoking automation to do everything for you Maybe that works for teeny-tiny clusters. I hope for your sake that it does. For 10K-node clusters, it's not uncommon for that "rare" condition to occur a few times, requiring a human to prod things along a bit. Yes, the automation does 99% of the work, but 1% can still add up pretty quickly. > We do not expect Only time will tell whether your expectation is realistic. Maybe it is, at your scale. It wouldn't have been at the scale I worked at, which seems a lot closer to what the OP is about.

notacoward 4y ago

Funny how every single reply missed the part where I mentioned I have worked at this scale. Larger in some ways. I know what that kind of automation looks like, and what it can do for you. My point is that, even with that, managing thousands of smaller clusters is going to involve way more operational work than running fewer larger clusters (based on appropriate technology). Try to address that, not the strawman of someone who might suggest trying to do this without modern automation.

deleted 4y ago

[deleted]

kshacker 4y ago

> how many people (including on client teams) does it take to manage this much sprawl? Being remotely familiar something with similar complexity, I say "very few". Automation is not one time, it is ongoing. You start with something small, and you build, and build, and build. I saw a presentation from maybe 5-6 years back talking about Apple Cassandra, and have talked to some engineers too when they used to have DataStax Cassandra summits. Their automation has probably scaled up from those days. Maybe their infrastructure grows by X% every year (X probably being a double digit number) and their automation increases their productivity / reduces their overhead by X/2%, it allows them to still retain a huge growth trajectory at a smaller increase in people costs. Improve productivity by X% or more (with new automation), and you have hit a home run. And they should not be asking "by what percentage should I optimize", but simply "where do we spend effort today and how can we reduce it" ... which any business or group should be doing it anyways. The biggest challenges with such things come when management changes strategy for various reasons because then you lose a year or 2, and then you fall really behind and need to do so much catch up just to catch breath.