8 ms·

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have les

> If you’ve created a database before, you probably had to estimate how many servers to use based on the expected traffic. The answer is "one". If you have less than 10k req/s you shouldn't even start to think about multiple DB servers or migrating from bog-standard MySQL/MariaDB or Postgres. I will never understand this obsession with "scaling". Modern web dev seriously over-complicates so many things, it's not even funny anymore.

deleted 5y ago
[deleted]
wyager 5y ago
Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.
mjb 5y ago
What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability. What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility. What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.
- phoboslab 5y ago
  Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.
  - mjibson 5y ago
    The setup you describe is very much not simple. I worked at a place with very good DBAs and our replication setup caused us more downtime than anything else. Cockroach and Spanner exist because many programmers observed that what you describe is hard.
    - Groxx 5y ago
      As a counter-anecdote: multiple startup projects I've worked on with separate MySQL setups where each had just a single master + two slaves (one warm for fast failover in case of hardware failure or upgrades, one cold for slow analytics-style queries) did just fine with millions (to tens of millions) of users. No downtime at all for years on end. MySQL and Postgres are massively more widely-used than Cockroach and Spanner, broadly very successfully. It's entirely feasible to run it with high uptime.
      - dboreham 5y ago
        Very few deployments experience actual failures. Could be some fridge-door/light situation going on.
        ketanip 5y ago
        > fridge-door/light situation going on what does it mean ?
        maybevain 5y ago
        I think that is meant to be parsed as: Just like you can't check if the fridge light is on without opening the door (which of course turns it on), it's hard to know if a system is resilient to failure without having one. It just might be that there hasn't been a situation that would cause a failure.
        deleted 5y ago
        [deleted]
- _vvhw 5y ago
  This is probably one of the best motivations for a distributed database that I've read. I find that it's not often that people grasp that distribution is about availability. It's obvious when you say it, but for a long time my own intuition was that distribution is about mostly durability or consensus protocols to provide total order across multiple machines. Yet these build together into availability. In fact, I first noticed this distinction when reading Brian M. Oki's seminal 1988 paper on Viewstamped Replication, the work that would pioneer the field of consensus—a year before Paxos but with an intuitive protocol essentially identical to Raft. The surprising thing is that today many of us might have titled the paper something about "consensus" or "total order" (which it practically invented, and which was the major breakthrough, at least how to do this in the presence of network partitions) but that he titled it "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems". I did a short intro talk to Viewstamped Replication (and particularly why FTP or nightly backups or manual failover are not a solution): https://www.youtube.com/watch?v=_Jlikdtm4OA https://www.youtube.com/watch?v=_Jlikdtm4OA The talk is followed by interviews with Brian M. Oki and James Cowling (authors of the 1988 and 2012 papers respectively).
brokencode 5y ago
The question isn’t whether you have 10k req/s now, but whether you expect to in the future. If you are designing a blog, then yeah, you probably don’t need to worry about it. If you are starting a social network or SAAS business application, then you probably do.
- GuB-42 5y ago
  The future will be different. A lot of successful businesses start with things that are not scalable, and it is a strength, not a weakness. If you start a social network for instance, you can't beat Facebook at its own game. You have to do something that Facebook can't do because it is too big. Scalability problems will be tackled as you grow. Among the many things Facebook can't do is running its service on a single database. It makes things much harder on them, thankfully, you are much smaller than Facebook and you can. Take that advantage.
psadri 5y ago
Unfortunately, easy to hit that with say GraphQL where each client request can resolve to dozens of db selects vs a single hand written/tuned SQL select.
- aaaaaaaaaaab 5y ago
  Maybe that's a good reason to avoid GraphQL then?
  - TheGuyWhoCodes 5y ago
    That's not the reason, there are many but this a nice feature where a client can send one request and it will resolve internally (so less client side requests), you can request only the fields you need and it's easier when hitting multiple micro services. Now you get into the issues of tracing, request failures and retries, payload size (only json? really..) etc. But I do agree that a super optimized sql for a a specific purpose has it's performance benefit
  - psadri 5y ago
    In theory the graphql server could be smart enough to assemble a single SQL select to satisfy the query. Does anyone know of an implementation that does this?
- infogulch 5y ago
  If you're using GraphQL with SQL -- postgres specifically -- I would say use Hasura, but support between Hasura & CockroachDB seems to have stalled due to missing triggers [0] [1]. CRDB supports a feature called "changefeeds" [2] which is claimed might cover some of Hasura's use-cases, but that's a proprietary extension not present in base PostgreSQL. [0]: https://github.com/hasura/graphql-engine/issues/678 https://github.com/hasura/graphql-engine/issues/678 [1]: https://github.com/cockroachdb/cockroach/issues/28296 https://github.com/cockroachdb/cockroach/issues/28296 [2]: https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-...
  - 5y ago
bitwize 5y ago
(robot voice) Does /dev/null support sharding? Sharding is the secret ingredient in the web scale sauce.
- zkldi 5y ago
  it's impressive how well that video has aged with modern web dev.
dcposch 5y ago
This is a good argument against running a k8s cluster when u don't need one, but not a good argument against this new serverless Cockroach product. Serverless is not just about auto scaling up from 1 to n, it's about autoscaling down from 1 to 0. If Cockroach provides a robust SQL DB at a marginal cost of ~$0/mo for small request volumes, that is a real value add over running your own pg server. Not having to deal with administration or backups is another big value add. This offering looks like it compares very nicely to say running an rds instance with auto backups enabled.
- gkop 5y ago
  Tradeoffs are tradeoffs. In your k8s example, by running a k8s cluster when you don't need one, a cost you pay is the overhead. In the Cockroach serverless case, costs that come to mind include vendor lock-in when you evolve a pattern of production traffic that is hard to migrate to other solutions, and security and compliance challenges due to the virtualized instances running on shared clusters. In many cases these tradeoffs may be worthwhile. My point is that looking at it only in the dimension of scaling up and down, doesn't tell the whole story. The OP doesn't talk about tradeoffs, so in the comment section we must.
  - jungturk 5y ago
    In the case you mention, you've made a tradeoff to defer developing an in-house solution for supporting a potential future pattern of production traffic, and that can be a huge accelerator. Further, once you're experiencing the types of success that demands a superior solution the priority to invest in such a solution is clear. It isn't "lock-in" - there's enough experience and capability in the market to solve those problems _once you have to_. Solutions like this let you decide whether the right time is at start-up or scale-up.
    - deleted 5y ago
      [deleted]
- jayd16 5y ago
  k8s lets you put multiple deployments on virtualized hardware in a cloud agnostic/consistent way. You can do all that on one machine. Even if you're running a single instance deployment you still probably want at least two environments. K8s isn't without value in that respect.
lumost 5y ago
The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The following questions are likely to come up 1) My t3.xl DB is down, how much bigger can I make it? 2) My r3.24xl DB can only handle 100 TPS and now my site is down, what can I do? 3) My 2x r3.24xl DB cluster costs a lot of money, Are other solutions cheaper? 4) My latency is high, are other solutions faster? For someone who hasn't dealt with these questions before, these will become long and painful lessons with massive material impacts to the business. It's appealing to use Dynamo as it takes the magic out of scaling. It's appealing to use serverless RDBMS as you don't have to think about it anymore unless it has high costs/latency.
- Groxx 5y ago
  Why would you assume that the person responsible for a thing doesn't know what they're doing?
- mdekkers 5y ago
  > The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The answer is very clear-cut: Work with professionals.
  - mr_toad 5y ago
    The amount of data being collected in the world is growing much faster than the number of database engineers is.
  - lumost 5y ago
    The number of professionals who know and care about scaling who also want to work on small scale applications is relatively small. Hiring them will pose a challenge.
didip 5y ago
I disagree. Even the simplest master-slave setup bound to cause an outage once or twice. Now, if you setup your DB using public cloud's flavor of PostgreSQL... That's a different story.
unobatbayar 5y ago
But wouldn't you agree that the initial architecture is really important; It should be at least designed with scaling in mind? Since it can get difficult changing things afterwards.
- jayd16 5y ago
  Sure if "in mind" includes realistic capacity planning.
anamax 5y ago
During a technical interview many years ago, I commented that the system architecture didn't scale. The interviewer responded "When that matters, I won't even be managing the person whose problem that is."
- emilsedgh 5y ago
  Isn't that the right mindset?
  - YZF 5y ago
    This will be somebody else's problem? Ehm, no it's not. - When the bridge collapses I'll be managing someone else. - When they find out this airplane has critical design flaws I'll be managing someone else. - When this software I'm working on is hacked I'll be managing someone else.
    - anamax 5y ago
      All bridges have a capacity limit. "Doesn't scale" doesn't imply "inadequate." Overbuilding is waste.
      - Tanjreeve 5y ago
        Software isn't bridges though. If you run into scaling issues if you're lucky it's just a case of swapping out a database for a bigger one. More likely it isn't just that though and being able to do things like scale up distributed workers coherently with a change of a config file requires upfront thought and design that YAGNI would say isn't necessary. People just breezily saying "we can optimise it later" for a scale up of orders of magnitude are nearly always wrong. I'd argue Reddit is a perfect example of a site that's clearly running into scaling issues but are locked into an architecture with few escape hatches built in When a bridge fails it'll just fail. When software runs into a scaling limit it'll degrade and fall on its arse constantly and be absolutely terrible as long as it takes for the software team to completely rework their entire architecture often having to learn completely new technologies.
truetraveller 5y ago
Side note: this guy is Dominic Szablewski, author of the amazing ImpactJS, JSMpeg, QuakeVR. Probably a 10x developer. He rivals Fabrice Bellard of ffmpeg fame. Check out his site: https://phoboslab.org/ https://phoboslab.org/
earleybird 5y ago
And the rational that validates this was written back around 2006[0] [0] https://web.archive.org/web/20090306191715/http://www.my-idconcept.de/downloads/You_Probably_Dont_Need_RAC.pdf https://web.archive.org/web/20090306191715/http://www.my-idc...
chillfox 5y ago
I feel like you forgot SQLite, and for really small scale you could just use whatever storage/serialization solution comes built in to your favorite language.
mgartner 5y ago
It seems like you’re defining “scaling” as growth of a workload to the point that it cannot be handled by a single-server DB. But with any service without a constant workload (I’d wager almost all services besides prototypes that get no users) you’re going to have to literally scale that one machine, by replacing it with a bigger machine. When you have 50 users you’re not going to be paying for some yy.24xlarge. You’ll start with something much more affordable. When the service grows to 50,000 users, you certainly won’t be at “Facebook scale”, but that t3.small isn’t going to cut it. Should your service ever decline, it’d be nice to scale that machine down to save on costs. At a previous job, we spent many human hours continually ratcheting up the size of our Postgres machine a few times a year. Not only did this take non-trivial engineering hours and mind-space, it also caused maintenance downtime due to the limitations of traditional DBMSs. Self-managed CockroachDB eliminates the downtime needed to scale. To handle a more intense workload, add machines. If you want to vertically scale each machine, that can be done without downtime too. CockroachDB Serverless takes this a step further by scaling up and down to suit the demands of a highly dynamic workload, while minimizing costs. Maybe what looks like an mega-scale obsession to you is actually a bunch of people trying to avoid the common headaches of managing a moderately sized, dynamic service.

deleted 5y ago

[deleted]

wyager 5y ago

Alternatively, people really do need to put a lot of thought into scaling, but only because they did something like write some core web service in an interpreted language framework that maxes out at 100 requests per second.

mjb 5y ago

What happens when that database fails? Are you OK losing some data, or do you want the data to be synchronously replicated off the machine and be available somewhere else after failure? Distribution isn't only about scale, it's also about availability. What happens when that database loses some data? Do you want an up-to-the second backup, or point-in-time recovery? Or are you OK restoring last night's backup? Distribution isn't only about scale, it's also about durability. What happens when you need to run an expensive business process ad-hoc? Do you want it to be easy to scale out reads, or to export that data to an analytics system? Or are you OK building something else to handle that case? Distribution isn't only about scale, it's also about flexibility. What happens when you want to serve customers in one market, and make sure that their data stays local for regulatory compliance reasons or latency? Are you OK with having separate databases? Distribution isn't only about scale, it's also about locality.

phoboslab 5y ago
Fair points. I would argue that for most people a simple master-slave setup with manual failover will produce far fewer headaches than a "serverless" architecture. When you are big enough to worry about the other issues, you surely are big enough to handle the requirements in-house. I see the dependence on some specific companies as the bigger threat to reliability.
- mjibson 5y ago
  The setup you describe is very much not simple. I worked at a place with very good DBAs and our replication setup caused us more downtime than anything else. Cockroach and Spanner exist because many programmers observed that what you describe is hard.
  - Groxx 5y ago
    As a counter-anecdote: multiple startup projects I've worked on with separate MySQL setups where each had just a single master + two slaves (one warm for fast failover in case of hardware failure or upgrades, one cold for slow analytics-style queries) did just fine with millions (to tens of millions) of users. No downtime at all for years on end. MySQL and Postgres are massively more widely-used than Cockroach and Spanner, broadly very successfully. It's entirely feasible to run it with high uptime.
    - dboreham 5y ago
      Very few deployments experience actual failures. Could be some fridge-door/light situation going on.
      - ketanip 5y ago
        > fridge-door/light situation going on what does it mean ?
        maybevain 5y ago
        I think that is meant to be parsed as: Just like you can't check if the fridge light is on without opening the door (which of course turns it on), it's hard to know if a system is resilient to failure without having one. It just might be that there hasn't been a situation that would cause a failure.
        deleted 5y ago
        [deleted]
_vvhw 5y ago
This is probably one of the best motivations for a distributed database that I've read. I find that it's not often that people grasp that distribution is about availability. It's obvious when you say it, but for a long time my own intuition was that distribution is about mostly durability or consensus protocols to provide total order across multiple machines. Yet these build together into availability. In fact, I first noticed this distinction when reading Brian M. Oki's seminal 1988 paper on Viewstamped Replication, the work that would pioneer the field of consensus—a year before Paxos but with an intuitive protocol essentially identical to Raft. The surprising thing is that today many of us might have titled the paper something about "consensus" or "total order" (which it practically invented, and which was the major breakthrough, at least how to do this in the presence of network partitions) but that he titled it "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems". I did a short intro talk to Viewstamped Replication (and particularly why FTP or nightly backups or manual failover are not a solution): https://www.youtube.com/watch?v=_Jlikdtm4OA https://www.youtube.com/watch?v=_Jlikdtm4OA The talk is followed by interviews with Brian M. Oki and James Cowling (authors of the 1988 and 2012 papers respectively).

brokencode 5y ago

The question isn’t whether you have 10k req/s now, but whether you expect to in the future. If you are designing a blog, then yeah, you probably don’t need to worry about it. If you are starting a social network or SAAS business application, then you probably do.

GuB-42 5y ago
The future will be different. A lot of successful businesses start with things that are not scalable, and it is a strength, not a weakness. If you start a social network for instance, you can't beat Facebook at its own game. You have to do something that Facebook can't do because it is too big. Scalability problems will be tackled as you grow. Among the many things Facebook can't do is running its service on a single database. It makes things much harder on them, thankfully, you are much smaller than Facebook and you can. Take that advantage.

psadri 5y ago

Unfortunately, easy to hit that with say GraphQL where each client request can resolve to dozens of db selects vs a single hand written/tuned SQL select.

aaaaaaaaaaab 5y ago
Maybe that's a good reason to avoid GraphQL then?
- TheGuyWhoCodes 5y ago
  That's not the reason, there are many but this a nice feature where a client can send one request and it will resolve internally (so less client side requests), you can request only the fields you need and it's easier when hitting multiple micro services. Now you get into the issues of tracing, request failures and retries, payload size (only json? really..) etc. But I do agree that a super optimized sql for a a specific purpose has it's performance benefit
- psadri 5y ago
  In theory the graphql server could be smart enough to assemble a single SQL select to satisfy the query. Does anyone know of an implementation that does this?
infogulch 5y ago
If you're using GraphQL with SQL -- postgres specifically -- I would say use Hasura, but support between Hasura & CockroachDB seems to have stalled due to missing triggers [0] [1]. CRDB supports a feature called "changefeeds" [2] which is claimed might cover some of Hasura's use-cases, but that's a proprietary extension not present in base PostgreSQL. [0]: https://github.com/hasura/graphql-engine/issues/678 https://github.com/hasura/graphql-engine/issues/678 [1]: https://github.com/cockroachdb/cockroach/issues/28296 https://github.com/cockroachdb/cockroach/issues/28296 [2]: https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-...
- 5y ago

bitwize 5y ago

(robot voice) Does /dev/null support sharding? Sharding is the secret ingredient in the web scale sauce.

zkldi 5y ago
it's impressive how well that video has aged with modern web dev.

dcposch 5y ago

This is a good argument against running a k8s cluster when u don't need one, but not a good argument against this new serverless Cockroach product. Serverless is not just about auto scaling up from 1 to n, it's about autoscaling down from 1 to 0. If Cockroach provides a robust SQL DB at a marginal cost of ~$0/mo for small request volumes, that is a real value add over running your own pg server. Not having to deal with administration or backups is another big value add. This offering looks like it compares very nicely to say running an rds instance with auto backups enabled.

gkop 5y ago
Tradeoffs are tradeoffs. In your k8s example, by running a k8s cluster when you don't need one, a cost you pay is the overhead. In the Cockroach serverless case, costs that come to mind include vendor lock-in when you evolve a pattern of production traffic that is hard to migrate to other solutions, and security and compliance challenges due to the virtualized instances running on shared clusters. In many cases these tradeoffs may be worthwhile. My point is that looking at it only in the dimension of scaling up and down, doesn't tell the whole story. The OP doesn't talk about tradeoffs, so in the comment section we must.
- jungturk 5y ago
  In the case you mention, you've made a tradeoff to defer developing an in-house solution for supporting a potential future pattern of production traffic, and that can be a huge accelerator. Further, once you're experiencing the types of success that demands a superior solution the priority to invest in such a solution is clear. It isn't "lock-in" - there's enough experience and capability in the market to solve those problems _once you have to_. Solutions like this let you decide whether the right time is at start-up or scale-up.
  - deleted 5y ago
    [deleted]
jayd16 5y ago
k8s lets you put multiple deployments on virtualized hardware in a cloud agnostic/consistent way. You can do all that on one machine. Even if you're running a single instance deployment you still probably want at least two environments. K8s isn't without value in that respect.

lumost 5y ago

The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The following questions are likely to come up 1) My t3.xl DB is down, how much bigger can I make it? 2) My r3.24xl DB can only handle 100 TPS and now my site is down, what can I do? 3) My 2x r3.24xl DB cluster costs a lot of money, Are other solutions cheaper? 4) My latency is high, are other solutions faster? For someone who hasn't dealt with these questions before, these will become long and painful lessons with massive material impacts to the business. It's appealing to use Dynamo as it takes the magic out of scaling. It's appealing to use serverless RDBMS as you don't have to think about it anymore unless it has high costs/latency.

Groxx 5y ago
Why would you assume that the person responsible for a thing doesn't know what they're doing?
mdekkers 5y ago
> The answer is unfortunately less clear cut. Particularly if you assume that whoever is tasked with scaling this hypothetical DB doesn't know what they are doing a-priori. The answer is very clear-cut: Work with professionals.
- mr_toad 5y ago
  The amount of data being collected in the world is growing much faster than the number of database engineers is.
- lumost 5y ago
  The number of professionals who know and care about scaling who also want to work on small scale applications is relatively small. Hiring them will pose a challenge.

didip 5y ago

I disagree. Even the simplest master-slave setup bound to cause an outage once or twice. Now, if you setup your DB using public cloud's flavor of PostgreSQL... That's a different story.

unobatbayar 5y ago

But wouldn't you agree that the initial architecture is really important; It should be at least designed with scaling in mind? Since it can get difficult changing things afterwards.

jayd16 5y ago
Sure if "in mind" includes realistic capacity planning.

anamax 5y ago

During a technical interview many years ago, I commented that the system architecture didn't scale. The interviewer responded "When that matters, I won't even be managing the person whose problem that is."

emilsedgh 5y ago
Isn't that the right mindset?
- YZF 5y ago
  This will be somebody else's problem? Ehm, no it's not. - When the bridge collapses I'll be managing someone else. - When they find out this airplane has critical design flaws I'll be managing someone else. - When this software I'm working on is hacked I'll be managing someone else.
  - anamax 5y ago
    All bridges have a capacity limit. "Doesn't scale" doesn't imply "inadequate." Overbuilding is waste.
    - Tanjreeve 5y ago
      Software isn't bridges though. If you run into scaling issues if you're lucky it's just a case of swapping out a database for a bigger one. More likely it isn't just that though and being able to do things like scale up distributed workers coherently with a change of a config file requires upfront thought and design that YAGNI would say isn't necessary. People just breezily saying "we can optimise it later" for a scale up of orders of magnitude are nearly always wrong. I'd argue Reddit is a perfect example of a site that's clearly running into scaling issues but are locked into an architecture with few escape hatches built in When a bridge fails it'll just fail. When software runs into a scaling limit it'll degrade and fall on its arse constantly and be absolutely terrible as long as it takes for the software team to completely rework their entire architecture often having to learn completely new technologies.

truetraveller 5y ago

Side note: this guy is Dominic Szablewski, author of the amazing ImpactJS, JSMpeg, QuakeVR. Probably a 10x developer. He rivals Fabrice Bellard of ffmpeg fame. Check out his site: https://phoboslab.org/ https://phoboslab.org/

earleybird 5y ago

And the rational that validates this was written back around 2006[0] [0] https://web.archive.org/web/20090306191715/http://www.my-idconcept.de/downloads/You_Probably_Dont_Need_RAC.pdf https://web.archive.org/web/20090306191715/http://www.my-idc...

chillfox 5y ago

I feel like you forgot SQLite, and for really small scale you could just use whatever storage/serialization solution comes built in to your favorite language.

mgartner 5y ago

It seems like you’re defining “scaling” as growth of a workload to the point that it cannot be handled by a single-server DB. But with any service without a constant workload (I’d wager almost all services besides prototypes that get no users) you’re going to have to literally scale that one machine, by replacing it with a bigger machine. When you have 50 users you’re not going to be paying for some yy.24xlarge. You’ll start with something much more affordable. When the service grows to 50,000 users, you certainly won’t be at “Facebook scale”, but that t3.small isn’t going to cut it. Should your service ever decline, it’d be nice to scale that machine down to save on costs. At a previous job, we spent many human hours continually ratcheting up the size of our Postgres machine a few times a year. Not only did this take non-trivial engineering hours and mind-space, it also caused maintenance downtime due to the limitations of traditional DBMSs. Self-managed CockroachDB eliminates the downtime needed to scale. To handle a more intense workload, add machines. If you want to vertically scale each machine, that can be done without downtime too. CockroachDB Serverless takes this a step further by scaling up and down to suit the demands of a highly dynamic workload, while minimizing costs. Maybe what looks like an mega-scale obsession to you is actually a bunch of people trying to avoid the common headaches of managing a moderately sized, dynamic service.