4 ms·

"Thundering herd" problem is how I have always heard it called.

thaumasiotes 5y ago
The thundering herd problem isn't really about high levels of traffic. To the extent that that's a problem, it's just an ordinary DOS. The thundering herd problem specifically refers to what happens if you coordinate things so that all your incoming requests occur simultaneously. Imagine that over the course of a week, you tell everyone who needs something from you "I'm busy right now; please come back next Tuesday at 11:28 am". You'll be overwhelmed on Tuesday at 11:28 am regardless of whether your average weekly workload is high or low, because you concentrated your entire weekly workload into the same one minute. You solve the thundering herd problem by not giving out the same retry time to everyone who contacts you while you're busy.
- 8note 5y ago
  Hmm. I think of thundering Herd being about retries. All your failing requests batch up when your retry strategy sucks, then you end up really high traffic on every retry, and very little in between
  - thaumasiotes 5y ago
    Where does your perspective differ from what I said above?
  - thraxil 5y ago
    Retries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.
    - ithkuil 5y ago
      Jitter is one way to solve it. Request coalescing is another. It depends on the request type. Is it cacheable? Do you require a per-client side effect? ...
      - arghwhat 5y ago
        Request coalescing in a shared cache does not solve thundering herd, it just reduces propagation to backend services. Your cache is still subject to a thundering herd, and may be unable to keep up. The only way to solve thundering herd - which is that a load of all requests arrive within a short timespan - is to distribute requests over larger timespan. Reducing your herd size by having fewer requests does not solve thundering herd, but may make it bearable.
    - derefr 5y ago
      What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests? Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five minutes, you get a huge burst of traffic, from all these clients—and it’s all different traffic, with nothing coalescesable. You don’t control the client in this case, but nor can you simply ban them—they’re your paying customers! (Yes, you can “fire your customer”, but this would be most of your customers…) And certainly, you can try to teach the devs of your client how to write their own jitter logic—but that rarely works out, as often it’s junior frontend devs who wrote the client-side code, and it’s hard to have a non-intermediated conversation with them.
      - thraxil 5y ago
        If you have no control at all over the client, then ultimately, you have to just take it and build your service to handle that amount of traffic. Adding jitter is a technique that you use when writing clients. That's why I mentioned it in the context of retries. If you are writing a CDN per the article, at some point your CDN has to make requests back to the origin. If one of those requests fails and you retry, you add jitter there to avoid DoSing yourself. If you are working in a microservices architecture, you add jitter on retries between your services. The best you can do with clients that are out of your control is to publish a client library/SDK for your API that is convenient for your customers to use and implements best practices like exponential backoff, jitter, etc. If you have documentation with code snippets that junior devs are likely to copy and paste, include it in those. If you've painted yourself into a corner like you describe and are seeing extremely regular traffic patterns, you might be able to pre-cache. Ie, it's 12:01 and you know that a barrage is coming at 12:05. Start going down the list of clients/feeds that you know are likely to be requested based on recent traffic patterns and generate the response, putting it in your cache/CDN with a five minute TTL. Then at least a good portion of the requests should be served straight from there and not add load to the origin. There are obviously drawbacks/risks to that approach, but it might be all you can really do.
  - arghwhat 5y ago
    Retries tend to amplify it, but a more common cause is scheduled tasks in clients/end user devices. E.g. all clients checking for an update at 10:00 UTC every day, all clients polling for new data at fixed times, etc.
eyelidlessness 5y ago
Thundering herd is about mitigating a problem with backpressure scenarios. If you have a backoff and a delayed queue of requests, letting them all proceed at once when the backpressure scenario resolves is likely to recreate it/create a new one. Staggering them so they proceed slightly off in time avoids that.

The thundering herd problem isn't really about high levels of traffic. To the extent that that's a problem, it's just an ordinary DOS. The thundering herd problem specifically refers to what happens if you coordinate things so that all your incoming requests occur simultaneously. Imagine that over the course of a week, you tell everyone who needs something from you "I'm busy right now; please come back next Tuesday at 11:28 am". You'll be overwhelmed on Tuesday at 11:28 am regardless of whether your average weekly workload is high or low, because you concentrated your entire weekly workload into the same one minute. You solve the thundering herd problem by not giving out the same retry time to everyone who contacts you while you're busy.

8note 5y ago
Hmm. I think of thundering Herd being about retries. All your failing requests batch up when your retry strategy sucks, then you end up really high traffic on every retry, and very little in between
- thaumasiotes 5y ago
  Where does your perspective differ from what I said above?
- thraxil 5y ago
  Retries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.
  - ithkuil 5y ago
    Jitter is one way to solve it. Request coalescing is another. It depends on the request type. Is it cacheable? Do you require a per-client side effect? ...
    - arghwhat 5y ago
      Request coalescing in a shared cache does not solve thundering herd, it just reduces propagation to backend services. Your cache is still subject to a thundering herd, and may be unable to keep up. The only way to solve thundering herd - which is that a load of all requests arrive within a short timespan - is to distribute requests over larger timespan. Reducing your herd size by having fewer requests does not solve thundering herd, but may make it bearable.
  - derefr 5y ago
    What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests? Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five minutes, you get a huge burst of traffic, from all these clients—and it’s all different traffic, with nothing coalescesable. You don’t control the client in this case, but nor can you simply ban them—they’re your paying customers! (Yes, you can “fire your customer”, but this would be most of your customers…) And certainly, you can try to teach the devs of your client how to write their own jitter logic—but that rarely works out, as often it’s junior frontend devs who wrote the client-side code, and it’s hard to have a non-intermediated conversation with them.
    - thraxil 5y ago
      If you have no control at all over the client, then ultimately, you have to just take it and build your service to handle that amount of traffic. Adding jitter is a technique that you use when writing clients. That's why I mentioned it in the context of retries. If you are writing a CDN per the article, at some point your CDN has to make requests back to the origin. If one of those requests fails and you retry, you add jitter there to avoid DoSing yourself. If you are working in a microservices architecture, you add jitter on retries between your services. The best you can do with clients that are out of your control is to publish a client library/SDK for your API that is convenient for your customers to use and implements best practices like exponential backoff, jitter, etc. If you have documentation with code snippets that junior devs are likely to copy and paste, include it in those. If you've painted yourself into a corner like you describe and are seeing extremely regular traffic patterns, you might be able to pre-cache. Ie, it's 12:01 and you know that a barrage is coming at 12:05. Start going down the list of clients/feeds that you know are likely to be requested based on recent traffic patterns and generate the response, putting it in your cache/CDN with a five minute TTL. Then at least a good portion of the requests should be served straight from there and not add load to the origin. There are obviously drawbacks/risks to that approach, but it might be all you can really do.
- arghwhat 5y ago
  Retries tend to amplify it, but a more common cause is scheduled tasks in clients/end user devices. E.g. all clients checking for an update at 10:00 UTC every day, all clients polling for new data at fixed times, etc.

eyelidlessness 5y ago

Thundering herd is about mitigating a problem with backpressure scenarios. If you have a backoff and a delayed queue of requests, letting them all proceed at once when the backpressure scenario resolves is likely to recreate it/create a new one. Staggering them so they proceed slightly off in time avoids that.