3 ms·

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a gr

by kyrra 9y ago

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retries), it can be a great way to handle high loads. [0] https://cloudplatform.googleblog.com/2016/12/using-load-shedding-to-survive-a-success-disaster-CRE-life-lessons.html https://cloudplatform.googleblog.com/2016/12/using-load-shed...

spyspy 9y ago
My team moved out entire platform to AppEngine to deal with the traffic spikes we get (among other reasons). It's worked out quite well I must say. It's also cool that if you know you have a spike coming that even AE can't handle, you can automatically tell the service to scale itself a few minutes beforehand in anticipation and then scale itself down later after it's passed.
jonhohle 9y ago
Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health. As a concrete example, when I worked at a large online retailer, we had to check certain customer properties during checkout. In the optimal case, we went to a service which fronted a consistent, transactional database. If that service failed or timed out, we had the option to hit a fast cache which generally was only a few milliseconds out of date. Finally, we could use less authoritative data that came from the application earlier, and if present, would always be in the customer's favor. That made it unlikely that our service ever appeared "down" regardless of the state of the dependencies.
- robbles 9y ago
  Why wouldn't you start with the cache? Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?
  - jonhohle 9y ago
    Based on previous decisions made by customers that were processed within the lifetime of the same request, there was some probability that the cache could be out of date. Typically, there was enough context within the request to make assumptions about the state of the authoritative store if data hadn't propagated to the cache in time. We eventually moved to using that initially. edit: I should add, though, the order of the datasources wasn't critical to the strategy: have a primary data source, backup data source, use any data available in the request context, and finally err in the customer's favor. That's 4 chances to get things right, none of which include retrying.
usaphp 9y ago
Are you obligated to write that top part that the opinion is your own? What is the point of it? Just wondering.
- kyrra 9y ago
  Any time I'm posting something that may endorse "the company" I tend to put that as a clear marker that I am likely biased in what I'm saying. Also, most large companies have something in the employment contract that talks about what you are allowed to post online (especially when it relates to the company).
sitkack 9y ago
Isn't the other option serving errors to all clients? What are the options for * currently serving 1k clients * can serve 10k more * see a spike in 50k clients I'd rather serve 11k clients and tell 39k of them to try again later (with enough capacity to tell them to try again).

by kyrra 9y ago

spyspy 9y ago

My team moved out entire platform to AppEngine to deal with the traffic spikes we get (among other reasons). It's worked out quite well I must say. It's also cool that if you know you have a spike coming that even AE can't handle, you can automatically tell the service to scale itself a few minutes beforehand in anticipation and then scale itself down later after it's passed.

jonhohle 9y ago

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health. As a concrete example, when I worked at a large online retailer, we had to check certain customer properties during checkout. In the optimal case, we went to a service which fronted a consistent, transactional database. If that service failed or timed out, we had the option to hit a fast cache which generally was only a few milliseconds out of date. Finally, we could use less authoritative data that came from the application earlier, and if present, would always be in the customer's favor. That made it unlikely that our service ever appeared "down" regardless of the state of the dependencies.

robbles 9y ago
Why wouldn't you start with the cache? Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?
- jonhohle 9y ago
  Based on previous decisions made by customers that were processed within the lifetime of the same request, there was some probability that the cache could be out of date. Typically, there was enough context within the request to make assumptions about the state of the authoritative store if data hadn't propagated to the cache in time. We eventually moved to using that initially. edit: I should add, though, the order of the datasources wasn't critical to the strategy: have a primary data source, backup data source, use any data available in the request context, and finally err in the customer's favor. That's 4 chances to get things right, none of which include retrying.

usaphp 9y ago

Are you obligated to write that top part that the opinion is your own? What is the point of it? Just wondering.

kyrra 9y ago
Any time I'm posting something that may endorse "the company" I tend to put that as a clear marker that I am likely biased in what I'm saying. Also, most large companies have something in the employment contract that talks about what you are allowed to post online (especially when it relates to the company).

sitkack 9y ago

Isn't the other option serving errors to all clients? What are the options for * currently serving 1k clients * can serve 10k more * see a spike in 50k clients I'd rather serve 11k clients and tell 39k of them to try again later (with enough capacity to tell them to try again).