5 ms·

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorithm just adds servers magically when your load gets high. Of course the system still has limits. But if you go from zonal to regional to global autoscaling, and you architect specifically to scale up each point in the system using autoscaling, it kind of doesn't have a limit? (that any one product would hit, at a global level) In the past I have spun up a duplicate distributed system in another region and just split the traffic between the two. IaC is crazy.

secondcoming 3y ago
That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.
- tonyarkles 3y ago
  100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If you start auto-scaling because a downstream service is struggling, you're just going to end up hitting it even harder and essentially contributing to a DoS attack against them. No silver bullets and all that. Still need to do engineering analysis to figure out what's actually happening before you start shooting.
  - lanstin 3y ago
    Priority queues for your downstream calls to rate limit them. A large enough cluster can take down most any data store, except maybe spanner. Even then you could increase latency a lot while it scales up.
manuelabeledo 3y ago
> Nobody seems to talk about this, I guess because it's taken for granted? No, because it has a theoretical limit, same as queues, back pressure, etc. One cannot simply scale up indefinitely because it is not profitable.
- 0xbadcafebee 3y ago
  There's no product in the world that would hit a limit if autoscaled globally on AWS. Sure, you could write an app whose literal sole purpose is "take up all memory, CPU and bandwidth, recursively, infinitely", but nobody is making that product. Real products built today have a finite amount of demand, and global cloud capacity is larger than that. You can't say what architecture is or isn't profitable in general, business is more complicated than that. But besides the realities of commerce, you can easily architect a system such that the autoscaling cost is a fraction of opex.
  - gilbetron 3y ago
    Not really true (I work for one that can, and does, hit limits in AWS, let alone GCP/Azure), but the general point is mostly true. You just might not actually get what you want fast enough, however.
  - smaudet 3y ago
    > There's no product in the world that would hit a limit if autoscaled globally on AWS > but nobody is making that product Nobody is making that product because nobody has money for it...(or the demand to supply that money). You can auto-scale without AWS, actually, its called measuring server load and auto-ordering new machines when your load increases (and keeping machines turned off on standby for spikes). Or you can go hybrid, and load-shed to a cloud...which would be the smart thing to do.
  - vault_ 3y ago
    > Real products built today have a finite amount of demand, and global cloud capacity is larger than that. This isn't really true, and it's especially not true when specialized hardware comes into play. If you have a "web-scale" GPU workload, it's not unlikely that you'll hit resource availability constraints from time to time. The question isn't whether cloud capacity is larger than your demand for a particular resource, it's whether cloud capacity is larger than the aggregate peak demand for that resource. Cloud providers aren't magic. They engage in capacity planning, sometimes underestimate, and are sometimes unable to actually procure as much hardware as they want to.
ahoka 3y ago
Increasing capacity would be very stupid if doing so does not also increase revenue.
secondcoming 3y ago
That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB
jeremyjh 3y ago
Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.
- 0xbadcafebee 3y ago
  ....you can autoscale database read replicas, and write nodes (master-master)....
  - jeremyjh 3y ago
    If you are bound in disk IO, adding master-master write nodes will not help you. The same number of bytes have to be written to the same volume whether they come from a replica or an application server. The only solution is partitioning/sharding, and there is no "easy button" to press and make that happen because it comes with its own limitations and trade-offs, and is something the application code will be intimately aware of.
    - lanstin 3y ago
      Auto sharding. A nice idea for a DAO layer.
  - sgarland 3y ago
    > master-master Now you have two problems.
vault_ 3y ago
Autoscaling seems like a downstream concern from the techniques being discussed here. Autoscaling tends to have a pretty high latency, so you still need a strategy for being overloaded while that extra capacity comes online. There's also a question of how the autoscaler knows what "load" is and when it's "too high." Just going off of CPU/memory usage probably means you're over-provisioning. Instead, if you have back-pressure or load-shedding built into your system you can use those as signals to the autoscaler.
- EdwardDiego 3y ago
  Autoscaling is great, if you solve the problems you rightly mention. But IMO it's best viewed not as a technique to increase capacity that risks overprovisioning, but rather it should be viewed as a technique to significantly reduce the overprovisioning you were already likely doing to provide capacity that could handle peaks in demand without blowing through delivery expectations (e.g., timeliness, data loss minimisation, etc.) At an old employer, our load was seasonal over the day. If one instance of an app could handle N req/s, and the daily peak maxed out at 100N req/s, then we had to run 100 instances as a minimum (we usually chucked some extra capacity in there for surprises) even if the mean daily peak was 75N req/s. And of course, at the times of the day when incoming reqs/s was 0.5N reqs/s, well, we still had 99 instances twiddling their thumbs. And then there were the days when suddenly we're hitting 200N req/s because Germany made the World Cup quarter-finals, and things are catching fire and services are degraded in a way that customers notice, and it becomes an official Bad Thing That Must Be Explained To The CEO. So when we reached a point in our system architecture (which took a fair bit of refactoring) where we could use autoscaling, we saved soooo much money, and had far fewer Bad Thing Explanations to do. We had always been massively overprovisioned for 20 hours of the day, and often still overprovisioned for the other 4, but we weren't overprovisioned enough for black swans, it was the worst of both worlds. (Although we kept a very close eye on Germany's progress in the football after that first World Cup experience) You're spot on that a) to autoscale up effectively we had to minimise the time an instance took to go from cold to hot, so focused a lot on shared caches being available to quickly to populate in-memory caches b) adding new hardware instances was always going to take longer than adding new app instances, so we had to find some balance in how we overprovisioned hardware capacity to give us breathing room for scaling without wasting too much money and c) we found significant efficiencies in costs and time to scale by changing the signals used to scale after starting out using CPU/mem. Also a significant learning curve for our org was realising that we needed to ensure we didn't scale down too aggressively, especially the hardware stuff that scaled down far faster than it scaled up. We hit situations where we'd scale down after a peak had ended, then shortly after along came another peak, so all the capacity we'd just dynamically removed had to be added back, with the inherent speed issues you mentioned, causing our service to be slow and annoying for customers, with minimal savings while capacity was trampolining. (This incidentally can be really problematic in systems where horizontal scaling can introduce a stop the world pause across multiple instances of an app. Anything that uses Kafka and consumer groups is particularly prone to this, as membership change in the group pauses all members of the CG while partitions are reallocated, although later versions of Kafka with sticky assignors have improved this somewhat. But yeah, very critical to stop these kinda apps from trampolining capacity if you want to keep data timeliness within acceptable bounds.) It took a lot of tuning to get all of it right, but when we did, the savings were spectacular. I think the CTO worked out that it only took six months of the reduced AWS costs to equal the cost of the two years of system refactoring needed to get to that point, and after that, it was all ongoing cream for the shareholders. And while I get the hate people have for unnecessary usage of K8s (like Kafka, it's a complex solution for complicated problems and using it unnecessarily is taking on a whole lot of complexity for no gain), it was perfect for our use case, the ability to tune how HPAs scale down, being able to scale on custom metrics, it was just brilliant. (I wish I could end with "And the company reinvested a significant proportion of the savings into growth and gave us all big fat bonuses for saving so much money", but haha, no. The CFO did try to tell us we'd been unnecessarily wasteful prior and should have just built a system that was created in 2007 like the 2019 version from the start, because apparently a lot of MBA schools have an entrance requirement of psychopathy and then to graduate you have to swear a bloodpact with the cruel and vicious God of Shareholder Value)
hinkley 3y ago
If I let my customers railroad me into running more servers to fulfill their “needs” then I may transition into losing money on my business. That’s not a solution. Needs is in scare quotes because a lot of traffic comes from misunderstanding or laziness from customers or from other divisions. Try as we might, nearly all of the improvements in capacity per customer on my project in the last six months have come from that sort of work. Other sorts of changes have unfortunately been too close to the noise floor. They add up but are difficult to demonstrate, and are very easy to undo with new feature work.
- o11c 3y ago
  Additionally, most users actually are happy with "slow me down if I'm making too many requests". This is much simpler than recovering from errors correctly.
  - lanstin 3y ago
    Which is why your test integration environment has to throw 429 and 5xx errors consistently from day one. The error handling paths are hard to retrofit but easy to do before deployment.
    - o11c 3y ago
      (What test environment?) That will make you introduce error handling, but not necessary "correct" handling. "Retry after errors" is a great way to overload a system.
      - lanstin 3y ago
        The idea behind forcing your clients to handle 429 isn't because they will do the right thing (although you'd think since ethernet won with exponential and random backup with some max timeouts that would be the default) but because it allows you the flexibility server side to start returning 429s when you need to - maybe just based on some idea of server load, maybe as some way of hard rate limits for a certain caller ID, maybe to do maintenance, whatever. It's the semantics "not processed, not fatal, try again in the future" which is hard to shoe horn into a large deployed number of callers after the fact, without angering people.
paulddraper 3y ago
While that is a "real" solution, it not necessarily a possible solution. 1. It may be cost prohibitive. 2. Some systems don't scale horizontally. (In particular, databases.) 3. Many systems do not scale linearly.
jiggawatts 3y ago
In my experience, auto scaling has not yet been applicable in the typical sense of adding more web servers. If you use an efficient language like C# or Go, the performance bottleneck moves to the database layer almost immediately. Caching can be added and can auto scale, but that has a tradeoff: stale data or eventual consistency. Databases are hard to scale because typically there has to be a single master instance responsible for write ordering and transaction consistency. Auto scaling the database layer is fraught with its own issues, and simply doesn’t work in practice for many common scenarios. For typical business apps, or typical web pages that might suddenly get a huge surge in usage (I.e.: Hug of death), a CDN can help… maybe. My customers keep asking for auto scale to solve their problems, when the root cause is invariably a missing index in their database.

secondcoming 3y ago

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.

tonyarkles 3y ago
100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If you start auto-scaling because a downstream service is struggling, you're just going to end up hitting it even harder and essentially contributing to a DoS attack against them. No silver bullets and all that. Still need to do engineering analysis to figure out what's actually happening before you start shooting.
- lanstin 3y ago
  Priority queues for your downstream calls to rate limit them. A large enough cluster can take down most any data store, except maybe spanner. Even then you could increase latency a lot while it scales up.

manuelabeledo 3y ago

> Nobody seems to talk about this, I guess because it's taken for granted? No, because it has a theoretical limit, same as queues, back pressure, etc. One cannot simply scale up indefinitely because it is not profitable.

0xbadcafebee 3y ago
There's no product in the world that would hit a limit if autoscaled globally on AWS. Sure, you could write an app whose literal sole purpose is "take up all memory, CPU and bandwidth, recursively, infinitely", but nobody is making that product. Real products built today have a finite amount of demand, and global cloud capacity is larger than that. You can't say what architecture is or isn't profitable in general, business is more complicated than that. But besides the realities of commerce, you can easily architect a system such that the autoscaling cost is a fraction of opex.
- gilbetron 3y ago
  Not really true (I work for one that can, and does, hit limits in AWS, let alone GCP/Azure), but the general point is mostly true. You just might not actually get what you want fast enough, however.
- smaudet 3y ago
  > There's no product in the world that would hit a limit if autoscaled globally on AWS > but nobody is making that product Nobody is making that product because nobody has money for it...(or the demand to supply that money). You can auto-scale without AWS, actually, its called measuring server load and auto-ordering new machines when your load increases (and keeping machines turned off on standby for spikes). Or you can go hybrid, and load-shed to a cloud...which would be the smart thing to do.
- vault_ 3y ago
  > Real products built today have a finite amount of demand, and global cloud capacity is larger than that. This isn't really true, and it's especially not true when specialized hardware comes into play. If you have a "web-scale" GPU workload, it's not unlikely that you'll hit resource availability constraints from time to time. The question isn't whether cloud capacity is larger than your demand for a particular resource, it's whether cloud capacity is larger than the aggregate peak demand for that resource. Cloud providers aren't magic. They engage in capacity planning, sometimes underestimate, and are sometimes unable to actually procure as much hardware as they want to.

ahoka 3y ago

Increasing capacity would be very stupid if doing so does not also increase revenue.

secondcoming 3y ago

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB

jeremyjh 3y ago

Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.

0xbadcafebee 3y ago
....you can autoscale database read replicas, and write nodes (master-master)....
- jeremyjh 3y ago
  If you are bound in disk IO, adding master-master write nodes will not help you. The same number of bytes have to be written to the same volume whether they come from a replica or an application server. The only solution is partitioning/sharding, and there is no "easy button" to press and make that happen because it comes with its own limitations and trade-offs, and is something the application code will be intimately aware of.
  - lanstin 3y ago
    Auto sharding. A nice idea for a DAO layer.
- sgarland 3y ago
  > master-master Now you have two problems.

vault_ 3y ago

Autoscaling seems like a downstream concern from the techniques being discussed here. Autoscaling tends to have a pretty high latency, so you still need a strategy for being overloaded while that extra capacity comes online. There's also a question of how the autoscaler knows what "load" is and when it's "too high." Just going off of CPU/memory usage probably means you're over-provisioning. Instead, if you have back-pressure or load-shedding built into your system you can use those as signals to the autoscaler.

EdwardDiego 3y ago
Autoscaling is great, if you solve the problems you rightly mention. But IMO it's best viewed not as a technique to increase capacity that risks overprovisioning, but rather it should be viewed as a technique to significantly reduce the overprovisioning you were already likely doing to provide capacity that could handle peaks in demand without blowing through delivery expectations (e.g., timeliness, data loss minimisation, etc.) At an old employer, our load was seasonal over the day. If one instance of an app could handle N req/s, and the daily peak maxed out at 100N req/s, then we had to run 100 instances as a minimum (we usually chucked some extra capacity in there for surprises) even if the mean daily peak was 75N req/s. And of course, at the times of the day when incoming reqs/s was 0.5N reqs/s, well, we still had 99 instances twiddling their thumbs. And then there were the days when suddenly we're hitting 200N req/s because Germany made the World Cup quarter-finals, and things are catching fire and services are degraded in a way that customers notice, and it becomes an official Bad Thing That Must Be Explained To The CEO. So when we reached a point in our system architecture (which took a fair bit of refactoring) where we could use autoscaling, we saved soooo much money, and had far fewer Bad Thing Explanations to do. We had always been massively overprovisioned for 20 hours of the day, and often still overprovisioned for the other 4, but we weren't overprovisioned enough for black swans, it was the worst of both worlds. (Although we kept a very close eye on Germany's progress in the football after that first World Cup experience) You're spot on that a) to autoscale up effectively we had to minimise the time an instance took to go from cold to hot, so focused a lot on shared caches being available to quickly to populate in-memory caches b) adding new hardware instances was always going to take longer than adding new app instances, so we had to find some balance in how we overprovisioned hardware capacity to give us breathing room for scaling without wasting too much money and c) we found significant efficiencies in costs and time to scale by changing the signals used to scale after starting out using CPU/mem. Also a significant learning curve for our org was realising that we needed to ensure we didn't scale down too aggressively, especially the hardware stuff that scaled down far faster than it scaled up. We hit situations where we'd scale down after a peak had ended, then shortly after along came another peak, so all the capacity we'd just dynamically removed had to be added back, with the inherent speed issues you mentioned, causing our service to be slow and annoying for customers, with minimal savings while capacity was trampolining. (This incidentally can be really problematic in systems where horizontal scaling can introduce a stop the world pause across multiple instances of an app. Anything that uses Kafka and consumer groups is particularly prone to this, as membership change in the group pauses all members of the CG while partitions are reallocated, although later versions of Kafka with sticky assignors have improved this somewhat. But yeah, very critical to stop these kinda apps from trampolining capacity if you want to keep data timeliness within acceptable bounds.) It took a lot of tuning to get all of it right, but when we did, the savings were spectacular. I think the CTO worked out that it only took six months of the reduced AWS costs to equal the cost of the two years of system refactoring needed to get to that point, and after that, it was all ongoing cream for the shareholders. And while I get the hate people have for unnecessary usage of K8s (like Kafka, it's a complex solution for complicated problems and using it unnecessarily is taking on a whole lot of complexity for no gain), it was perfect for our use case, the ability to tune how HPAs scale down, being able to scale on custom metrics, it was just brilliant. (I wish I could end with "And the company reinvested a significant proportion of the savings into growth and gave us all big fat bonuses for saving so much money", but haha, no. The CFO did try to tell us we'd been unnecessarily wasteful prior and should have just built a system that was created in 2007 like the 2019 version from the start, because apparently a lot of MBA schools have an entrance requirement of psychopathy and then to graduate you have to swear a bloodpact with the cruel and vicious God of Shareholder Value)

hinkley 3y ago

If I let my customers railroad me into running more servers to fulfill their “needs” then I may transition into losing money on my business. That’s not a solution. Needs is in scare quotes because a lot of traffic comes from misunderstanding or laziness from customers or from other divisions. Try as we might, nearly all of the improvements in capacity per customer on my project in the last six months have come from that sort of work. Other sorts of changes have unfortunately been too close to the noise floor. They add up but are difficult to demonstrate, and are very easy to undo with new feature work.

o11c 3y ago
Additionally, most users actually are happy with "slow me down if I'm making too many requests". This is much simpler than recovering from errors correctly.
- lanstin 3y ago
  Which is why your test integration environment has to throw 429 and 5xx errors consistently from day one. The error handling paths are hard to retrofit but easy to do before deployment.
  - o11c 3y ago
    (What test environment?) That will make you introduce error handling, but not necessary "correct" handling. "Retry after errors" is a great way to overload a system.
    - lanstin 3y ago
      The idea behind forcing your clients to handle 429 isn't because they will do the right thing (although you'd think since ethernet won with exponential and random backup with some max timeouts that would be the default) but because it allows you the flexibility server side to start returning 429s when you need to - maybe just based on some idea of server load, maybe as some way of hard rate limits for a certain caller ID, maybe to do maintenance, whatever. It's the semantics "not processed, not fatal, try again in the future" which is hard to shoe horn into a large deployed number of callers after the fact, without angering people.

paulddraper 3y ago

While that is a "real" solution, it not necessarily a possible solution. 1. It may be cost prohibitive. 2. Some systems don't scale horizontally. (In particular, databases.) 3. Many systems do not scale linearly.

jiggawatts 3y ago

In my experience, auto scaling has not yet been applicable in the typical sense of adding more web servers. If you use an efficient language like C# or Go, the performance bottleneck moves to the database layer almost immediately. Caching can be added and can auto scale, but that has a tradeoff: stale data or eventual consistency. Databases are hard to scale because typically there has to be a single master instance responsible for write ordering and transaction consistency. Auto scaling the database layer is fraught with its own issues, and simply doesn’t work in practice for many common scenarios. For typical business apps, or typical web pages that might suddenly get a huge surge in usage (I.e.: Hug of death), a CDN can help… maybe. My customers keep asking for auto scale to solve their problems, when the root cause is invariably a missing index in their database.