6 ms·

Graceful behavior at capacity

140 points by ingve 3y ago

rdoherty 3y ago
I learned most of this the hard way as a SRE. How systems behave at and over their limits is far more important than how they behave under them. A system that is 'forgiving' (aka resilient) is worth its weight in gold. Otherwise you get into downward spirals with systems that can't recover unless they are rebooted. Great read!
ChrisMarshallNY 3y ago
I'm writing an app that is likely to have, at most, a thousand users, for the next couple of years. I'm testing it with 12,000 fake users. It works great.
- bombcar 3y ago
  Tune down your capacities until it breaks at a thousand users (reduce disk, cpu, memory, network, etc). Determine what broke and how. Instrument that.
  - ChrisMarshallNY 3y ago
    Good idea! Thanks!
tra3 3y ago
Most of the microservice code I see is response = fetch(url, payload) if (response.error) ... but 99% of the folks I ask what is going to happen when the fetch does NOT error out but instead takes 10 seconds look at me like I'm speaking gibberish. This is the single biggest reason for cascading failures I see. Netflix has dealt with it via their Hystrix library (open source). These days it seems like a proxy like Consul is the way to go. It encapsulates all of the fancy logic (like circuit breakers and flow control) so your service doesn't have to.
- jiggawatts 3y ago
  As an ardent fan of monoliths and how they generally avoid such tar pits, I have to acknowledge that service-oriented architectures have their uses. So do we all have to keep reinventing these wheels, but only after a production outage? Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? I keep seeing the same patterns being re-engineered over and over. Maybe it’s time to refactor these out…
  - klooney 3y ago
    It's more work, is the simple answer.
    - jiggawatts 3y ago
      Once. It's more work once instead of over and over. That's the point of operating systems, standard libraries, and modules! I see this weird back-lash in modern development against having common, standard platforms. I suspect it comes from the Python and JavaScript world, where having "no batteries included" is seen as a good thing, instead of a guaranteed mess of dozens of half-complete incompatible frameworks. I'm coming from the perspective of Windows and comparing it to, say, Azure or AWS. All three have some concepts of access control, log collection, component systems, processes, etc... But all three are proprietary. Kubernetes goes a long way, but it isn't a user-mode system that can be directly accessed from code. Compare with Service Fabric, which has a substantial SDK component that integrates into the applications. As an example, here's a really basic thing that is actually absurdly difficult to solve well: web application session state. If you have sticky load balancing using cookies, then the session state is accessed on one VM something like 99.99% of the time... except for that 1% of the time when it isn't. This could be due to a restart, load rebalancing, or whatever. If you put the session state into something external like Redis, then a zone-redundant deployment will eat a ~1ms delay on every page render, every time. Service Fabric uses a model where it keeps three replicas of the state: one in the original web server, and two replicas in elsewhere. This way, reads are in process on the same VM most of the time, resulting in nanosecond latencies. Writing the state can occur asynchronously after the page response is already being sent. I'd like to see concepts like this, along with all sorts of service-to-service communication patterns, consolidated into an "operating system like platform" designed for the mid-2020s clouds instead of 1990s server farms.
thewakalix 3y ago
From my armchair, I'm not sure that "random drop" actually does decrease latency. Most clients will just repeat the request, resulting in an "effective latency" of however many times it gets randomly dropped. The queue is now implicit, and I'd guess that it's less efficient to carry out several request/drop cycles than to just leave the client in a straightforward queue.
notacoward 3y ago
FWIW, the single paragraph about "fair allocation" could be its own thesis. This gets into quality of service, active queue management, leaky buckets, deficit round robin, and so on ad infinitum. I did quite a bit of work on this on multiple projects at multiple companies, and it's still one of the very few algorithmic areas that I still think about in retirement. I highly recommend following up on some of the terms above for some interesting explorations.
- gillh 3y ago
  We built a fair scheduler for APIs and it's in the open source - https://docs.fluxninja.com/concepts/scheduler/load-scheduler https://docs.fluxninja.com/concepts/scheduler/load-scheduler I wish more people knew about this project!
0xbadcafebee 3y ago
If your clients don't implement an increasing backoff retry with jitter, you can fake it by making your server begin to randomly and increasingly time-out connections or wait to accept them. You can do it a few different ways. For already-open connections: 1) Keep the connection open and respond with a tiny amount of data every once in a while, so the connection doesn't time out, but make it long enough that a complete request will take forever.. Applications may still re-connect if a successful request-response doesn't happen within a timeout. 2) Keep the connection open but don't respond. The application may time out the connection and re-connect if it receives no data. 3) Drop the connection but don't let the tcp/ip stack send a RST, FIN, or anything else. The applications' connection will be timed out pretty soon, either by the OS stack or an application timeout. 4) Respond to requests with HTTP response codes that will make the client retry. As long as this only happens at the load-balancer level, it will still remove pressure from your application layer. For new connections: 1) Play hard to get. During the 3-way handshake, respond with weird states that won't cause the connection to drop, but will make the client keep trying to connect, like the handshake is still trying to succeed but experiencing loss. 2) Do a normal 3-way handshake, but wait forever on each step to drag it out. All of these options are a terrible idea for user-facing applications. They can cause more issues on your server side due to connections in wait state, and your clients will just see requests hanging, stalling, etc. You could just pretend the problem is with their ISP rather than your end. Or you could throw a 429 or 503 to try to stop the retries, but then they really know it's your fault. The best option is to just add capacity. Back in the day we couldn't do that; you had the servers you had. So when all 400 machines in our colo were thrashing, we just had to shed load randomly, sometimes using the tricks above. Now with the cloud you can magically add capacity anytime, automatically. Much better option. What's often much better is to add another caching layer before the application layer and just return stale responses during pressure events. But sometimes you can't, so you resort to either a 429 or 503, or the dirty tricks above.
- gillh 3y ago
  You should check out the Aperture flow control system - https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture We built a weighted fair queueing system in the Aperture flow control system to help alleviate the API load pressure - https://docs.fluxninja.com/concepts/scheduler/load-scheduler https://docs.fluxninja.com/concepts/scheduler/load-scheduler And in addition, we are investing in the graceful-js library to handle 429 and 523 codes returned by the Aperture system - https://github.com/fluxninja/graceful-js https://github.com/fluxninja/graceful-js
ISL 3y ago
Strategies like these (and kernel schedulers) are often as applicable in life as they are in computer science.
- pramodbiligiri 3y ago
  The book Algorithms to Live By tries to do exactly this: https://www.amazon.in/Algorithms-Live-Computer-Science-Decisions/dp/1627790365 https://www.amazon.in/Algorithms-Live-Computer-Science-Decis....
andrewf 3y ago
I agree with all this. "Metastable Failures in Distributed Systems" (2001) is another good read if you're facing problems in this vein. https://news.ycombinator.com/item?id=28750103 https://news.ycombinator.com/item?id=28750103
gillh 3y ago
Very interesting blog post! Our team has been working intensively in this area for the last couple of years - flow control, load shedding, controllability (PID control), and so on. We have open-sourced our work at - https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture We also did a Twitter Spaces discussion with Kelsey earlier today - https://twitter.com/kelseyhightower/status/1689355284802629633 https://twitter.com/kelseyhightower/status/16893552848026296... We would love feedback from folks reading this blog post! Disclaimer: I am one of the co-authors of the Aperture project. There are several interesting ideas we have built into this project, and I will be happy to dive into the technical details as well.
User23 3y ago
Ever since I heard of Little[1] it's been surprising to me how few working programmers know that queuing theory is basically a solved problem and has been for longer than most working programmers have been alive. [1] https://en.wikipedia.org/wiki/Little's_law https://en.wikipedia.org/wiki/Little's_law
- Animats 3y ago
  Ah. That's classic queuing theory. It has a problem. The early work on network congestion came from Kleinrock, who wrote the classic "Queuing theory". Kleinrock did his PhD thesis at MIT on Western Union Plan 55-A, a telegram switching system which can be thought of as Sendmail built out of relays and paper tape. Message switches look like a classic arrival rate / service rate problem. They have little customer-level back-pressure; you can send an email regardless of whether the transmission system is backed up. So an open-loop analysis works fine. The ARPANET had flow control on each link. Nothing could send a message until there was a buffer ready to receive it. So no packets were lost due to congestion. All overload is stopped at the sender. That approach is immune to congestion collapse, but not to lockup. Then came the pure datagram networks, and TCP/IP. Anybody can send an IP datagram any time they want to, regardless of the network status. So overloads and packet loss are possible. TCP uses retransmission to hide that, imperfectly. This introduces a new set of problems, some of which were non-obvious at the time. Classical queuing theory is open-loop. Arrival rate is considered to be independent of wait time. In the real world, it's not. Not even for store cashiers. If arrival rate exceeds service rate, the line length does not really grow without bound except in desperate situations. Customers leave without buying and take their business elsewhere. If there is no cashier idle time, the line length will increase only until the customer loss rate increases to match. Many retail managers do not get this. I coined the term "congestion collapse" in 1984.[1] In 1985, I wrote, in my "On Package Switches with Infinite Storage" RFC, "We have thus shown that a datagram network with infinite storage, first-in-first-out queuing, and a finite packet lifetime will, under overload, drop all packets."[2] Until then, people had been doing mostly classic queuing theory analysis. That's not enough. Back then, memory was very expensive, and people were obsessing over how much memory was needed in a router. It was felt that adding more memory would solve the congestion problem. I pointed out that wouldn't work. Now that memory is cheap, that problem appears as "bufferbloat". Those two RFCs started people thinking about this as a closed-loop problem. Van Jacobson later did much work in this area. I was out of it by 1986. Decades later, people are still fussing with the feedback control problems implicit in that result. As the original poster points out here, this comes up in other situations, especially chains of services. If you get congestion in the middle of the chain, things will not go well, and there's a good chance of something that looks like congestion collapse, where throughput goes to nearly zero. It's better to push congestion out towards the endpoints. We still don't have good solutions to congestion in the middle of a pure datagram network. What saved the Internet was fiber optic backbones and cheap long-haul bandwidth. There was a period in the 1990s when traffic had built up but backbone bandwidth was still expensive. The long-haul links choked and the Internet had "storms". There used to be an "Internet Weather Center", where you could check on how congested the major routers were. I also coined the term "fair queuing". That can be a useful technique for services well above the IP datagram level. Don't use a FIFO queue; queue based on who's sending. If some source is sending too much, let them compete against themselves for the service. Others can still get through. This provides resilience against denial of service attacks. I put that on a web site of mine some years ago, and for two months someone was pounding on it with useless requests without affecting response time for anybody else. (It wasn't an attack, just ineptitude using a public API.) [1] https://datatracker.ietf.org/doc/html/rfc896 https://datatracker.ietf.org/doc/html/rfc896 [2] https://datatracker.ietf.org/doc/html/rfc970 https://datatracker.ietf.org/doc/html/rfc970
DeathArrow 3y ago
>Most of the microservice code I see is Not mine. I use circuit breakers. Although, the circuit breaker will never be hit because of the high load as Kubernetes will be already firing up some new pods by that time.

140 points by ingve 3y ago

I learned most of this the hard way as a SRE. How systems behave at and over their limits is far more important than how they behave under them. A system that is 'forgiving' (aka resilient) is worth its weight in gold. Otherwise you get into downward spirals with systems that can't recover unless they are rebooted. Great read!

ChrisMarshallNY 3y ago

I'm writing an app that is likely to have, at most, a thousand users, for the next couple of years. I'm testing it with 12,000 fake users. It works great.

bombcar 3y ago
Tune down your capacities until it breaks at a thousand users (reduce disk, cpu, memory, network, etc). Determine what broke and how. Instrument that.
- ChrisMarshallNY 3y ago
  Good idea! Thanks!

tra3 3y ago

Most of the microservice code I see is response = fetch(url, payload) if (response.error) ... but 99% of the folks I ask what is going to happen when the fetch does NOT error out but instead takes 10 seconds look at me like I'm speaking gibberish. This is the single biggest reason for cascading failures I see. Netflix has dealt with it via their Hystrix library (open source). These days it seems like a proxy like Consul is the way to go. It encapsulates all of the fancy logic (like circuit breakers and flow control) so your service doesn't have to.

jiggawatts 3y ago
As an ardent fan of monoliths and how they generally avoid such tar pits, I have to acknowledge that service-oriented architectures have their uses. So do we all have to keep reinventing these wheels, but only after a production outage? Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? I keep seeing the same patterns being re-engineered over and over. Maybe it’s time to refactor these out…
- klooney 3y ago
  It's more work, is the simple answer.
  - jiggawatts 3y ago
    Once. It's more work once instead of over and over. That's the point of operating systems, standard libraries, and modules! I see this weird back-lash in modern development against having common, standard platforms. I suspect it comes from the Python and JavaScript world, where having "no batteries included" is seen as a good thing, instead of a guaranteed mess of dozens of half-complete incompatible frameworks. I'm coming from the perspective of Windows and comparing it to, say, Azure or AWS. All three have some concepts of access control, log collection, component systems, processes, etc... But all three are proprietary. Kubernetes goes a long way, but it isn't a user-mode system that can be directly accessed from code. Compare with Service Fabric, which has a substantial SDK component that integrates into the applications. As an example, here's a really basic thing that is actually absurdly difficult to solve well: web application session state. If you have sticky load balancing using cookies, then the session state is accessed on one VM something like 99.99% of the time... except for that 1% of the time when it isn't. This could be due to a restart, load rebalancing, or whatever. If you put the session state into something external like Redis, then a zone-redundant deployment will eat a ~1ms delay on every page render, every time. Service Fabric uses a model where it keeps three replicas of the state: one in the original web server, and two replicas in elsewhere. This way, reads are in process on the same VM most of the time, resulting in nanosecond latencies. Writing the state can occur asynchronously after the page response is already being sent. I'd like to see concepts like this, along with all sorts of service-to-service communication patterns, consolidated into an "operating system like platform" designed for the mid-2020s clouds instead of 1990s server farms.

thewakalix 3y ago

From my armchair, I'm not sure that "random drop" actually does decrease latency. Most clients will just repeat the request, resulting in an "effective latency" of however many times it gets randomly dropped. The queue is now implicit, and I'd guess that it's less efficient to carry out several request/drop cycles than to just leave the client in a straightforward queue.

notacoward 3y ago

FWIW, the single paragraph about "fair allocation" could be its own thesis. This gets into quality of service, active queue management, leaky buckets, deficit round robin, and so on ad infinitum. I did quite a bit of work on this on multiple projects at multiple companies, and it's still one of the very few algorithmic areas that I still think about in retirement. I highly recommend following up on some of the terms above for some interesting explorations.

gillh 3y ago
We built a fair scheduler for APIs and it's in the open source - https://docs.fluxninja.com/concepts/scheduler/load-scheduler https://docs.fluxninja.com/concepts/scheduler/load-scheduler I wish more people knew about this project!

0xbadcafebee 3y ago

If your clients don't implement an increasing backoff retry with jitter, you can fake it by making your server begin to randomly and increasingly time-out connections or wait to accept them. You can do it a few different ways. For already-open connections: 1) Keep the connection open and respond with a tiny amount of data every once in a while, so the connection doesn't time out, but make it long enough that a complete request will take forever.. Applications may still re-connect if a successful request-response doesn't happen within a timeout. 2) Keep the connection open but don't respond. The application may time out the connection and re-connect if it receives no data. 3) Drop the connection but don't let the tcp/ip stack send a RST, FIN, or anything else. The applications' connection will be timed out pretty soon, either by the OS stack or an application timeout. 4) Respond to requests with HTTP response codes that will make the client retry. As long as this only happens at the load-balancer level, it will still remove pressure from your application layer. For new connections: 1) Play hard to get. During the 3-way handshake, respond with weird states that won't cause the connection to drop, but will make the client keep trying to connect, like the handshake is still trying to succeed but experiencing loss. 2) Do a normal 3-way handshake, but wait forever on each step to drag it out. All of these options are a terrible idea for user-facing applications. They can cause more issues on your server side due to connections in wait state, and your clients will just see requests hanging, stalling, etc. You could just pretend the problem is with their ISP rather than your end. Or you could throw a 429 or 503 to try to stop the retries, but then they really know it's your fault. The best option is to just add capacity. Back in the day we couldn't do that; you had the servers you had. So when all 400 machines in our colo were thrashing, we just had to shed load randomly, sometimes using the tricks above. Now with the cloud you can magically add capacity anytime, automatically. Much better option. What's often much better is to add another caching layer before the application layer and just return stale responses during pressure events. But sometimes you can't, so you resort to either a 429 or 503, or the dirty tricks above.

gillh 3y ago
You should check out the Aperture flow control system - https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture We built a weighted fair queueing system in the Aperture flow control system to help alleviate the API load pressure - https://docs.fluxninja.com/concepts/scheduler/load-scheduler https://docs.fluxninja.com/concepts/scheduler/load-scheduler And in addition, we are investing in the graceful-js library to handle 429 and 523 codes returned by the Aperture system - https://github.com/fluxninja/graceful-js https://github.com/fluxninja/graceful-js

ISL 3y ago

Strategies like these (and kernel schedulers) are often as applicable in life as they are in computer science.

pramodbiligiri 3y ago
The book Algorithms to Live By tries to do exactly this: https://www.amazon.in/Algorithms-Live-Computer-Science-Decisions/dp/1627790365 https://www.amazon.in/Algorithms-Live-Computer-Science-Decis....

andrewf 3y ago

I agree with all this. "Metastable Failures in Distributed Systems" (2001) is another good read if you're facing problems in this vein. https://news.ycombinator.com/item?id=28750103 https://news.ycombinator.com/item?id=28750103

gillh 3y ago

Very interesting blog post! Our team has been working intensively in this area for the last couple of years - flow control, load shedding, controllability (PID control), and so on. We have open-sourced our work at - https://github.com/fluxninja/aperture https://github.com/fluxninja/aperture We also did a Twitter Spaces discussion with Kelsey earlier today - https://twitter.com/kelseyhightower/status/1689355284802629633 https://twitter.com/kelseyhightower/status/16893552848026296... We would love feedback from folks reading this blog post! Disclaimer: I am one of the co-authors of the Aperture project. There are several interesting ideas we have built into this project, and I will be happy to dive into the technical details as well.

User23 3y ago

Ever since I heard of Little[1] it's been surprising to me how few working programmers know that queuing theory is basically a solved problem and has been for longer than most working programmers have been alive. [1] https://en.wikipedia.org/wiki/Little's_law https://en.wikipedia.org/wiki/Little's_law

Animats 3y ago
Ah. That's classic queuing theory. It has a problem. The early work on network congestion came from Kleinrock, who wrote the classic "Queuing theory". Kleinrock did his PhD thesis at MIT on Western Union Plan 55-A, a telegram switching system which can be thought of as Sendmail built out of relays and paper tape. Message switches look like a classic arrival rate / service rate problem. They have little customer-level back-pressure; you can send an email regardless of whether the transmission system is backed up. So an open-loop analysis works fine. The ARPANET had flow control on each link. Nothing could send a message until there was a buffer ready to receive it. So no packets were lost due to congestion. All overload is stopped at the sender. That approach is immune to congestion collapse, but not to lockup. Then came the pure datagram networks, and TCP/IP. Anybody can send an IP datagram any time they want to, regardless of the network status. So overloads and packet loss are possible. TCP uses retransmission to hide that, imperfectly. This introduces a new set of problems, some of which were non-obvious at the time. Classical queuing theory is open-loop. Arrival rate is considered to be independent of wait time. In the real world, it's not. Not even for store cashiers. If arrival rate exceeds service rate, the line length does not really grow without bound except in desperate situations. Customers leave without buying and take their business elsewhere. If there is no cashier idle time, the line length will increase only until the customer loss rate increases to match. Many retail managers do not get this. I coined the term "congestion collapse" in 1984.[1] In 1985, I wrote, in my "On Package Switches with Infinite Storage" RFC, "We have thus shown that a datagram network with infinite storage, first-in-first-out queuing, and a finite packet lifetime will, under overload, drop all packets."[2] Until then, people had been doing mostly classic queuing theory analysis. That's not enough. Back then, memory was very expensive, and people were obsessing over how much memory was needed in a router. It was felt that adding more memory would solve the congestion problem. I pointed out that wouldn't work. Now that memory is cheap, that problem appears as "bufferbloat". Those two RFCs started people thinking about this as a closed-loop problem. Van Jacobson later did much work in this area. I was out of it by 1986. Decades later, people are still fussing with the feedback control problems implicit in that result. As the original poster points out here, this comes up in other situations, especially chains of services. If you get congestion in the middle of the chain, things will not go well, and there's a good chance of something that looks like congestion collapse, where throughput goes to nearly zero. It's better to push congestion out towards the endpoints. We still don't have good solutions to congestion in the middle of a pure datagram network. What saved the Internet was fiber optic backbones and cheap long-haul bandwidth. There was a period in the 1990s when traffic had built up but backbone bandwidth was still expensive. The long-haul links choked and the Internet had "storms". There used to be an "Internet Weather Center", where you could check on how congested the major routers were. I also coined the term "fair queuing". That can be a useful technique for services well above the IP datagram level. Don't use a FIFO queue; queue based on who's sending. If some source is sending too much, let them compete against themselves for the service. Others can still get through. This provides resilience against denial of service attacks. I put that on a web site of mine some years ago, and for two months someone was pounding on it with useless requests without affecting response time for anybody else. (It wasn't an attack, just ineptitude using a public API.) [1] https://datatracker.ietf.org/doc/html/rfc896 https://datatracker.ietf.org/doc/html/rfc896 [2] https://datatracker.ietf.org/doc/html/rfc970 https://datatracker.ietf.org/doc/html/rfc970

DeathArrow 3y ago

>Most of the microservice code I see is Not mine. I use circuit breakers. Although, the circuit breaker will never be hit because of the high load as Kubernetes will be already firing up some new pods by that time.