4 ms·

I've worked with systems where petabytes are flowing through it every day. If your data starts backing up, it costs lots of real money in terms of hard disk spa

I've worked with systems where petabytes are flowing through it every day. If your data starts backing up, it costs lots of real money in terms of hard disk space. Backpressure crosses organizational boundaries--the team running the system which consumes data is not the same team running the system which produces data. We exposed backpressure through our APIs... if our system was overloaded, we returned a specific error code. Clients were expected to retry, sensibly. Sometimes "add more capacity" just costs too much.

tootie 5y ago
Obviously you've got to pay for something. You can either increase your compute capacity to process requests more quickly, pay for storage on a queue to regulate flow, or do some kind of internal code optimization to decrease resource demands. All of which cost money one way or the other. Alternately, you go up the food chain and determine the ROI on all the data you're receiving and maybe stop collecting some things that aren't valuable. If they are valuable, then it's a no-brainer to pay what you have to.
StillBored 5y ago
Failure codes themselves aren't back pressure though, its the client side "retry later" which creates that effect. But, if you have that behavior, its better just to signal it directly because a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. And if its gets bad enough, potentially results in livelock like behaviors. Simply slowing down ACK's or explicit "item queued, please wait before sending another" return codes are far more effective. See buffer crediting, flow control. OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).
- deleted 5y ago
  [deleted]
- klodolph 5y ago
  > ... a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. Well, two things. First, our clients didn't have a "simple" back off, retry system. They tended to have more sophisticated back-off and retry systems that didn't increase pressure on services that were overloaded. There are a number of different techniques you use to accomplish this and you can wrap up several of these techniques in a library, use the library in your client code, and have that client code (a fat client) be the official API for your service. Second, this also assumes that front-end load is significant to begin with. In our system, it definitely wasn't. Requests were super cheap. You could hammer the front-end extremely hard and it would just respond with "try again later" until there was some capacity available, and then it would let in some work. The size of an actual request was quite small, it was the size of the work represented by that request that was large. > OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency). Queues in our system were a necessary part of the design. The system would not operate efficiently without them, because work items needed to be batched in order to be processed efficiently. I don't think there's a way that you can indict a system for using queues unless you know something about the design requirements. There are absolutely other reasons to use queues other than smoothing out bursty behavior.

tootie 5y ago

Obviously you've got to pay for something. You can either increase your compute capacity to process requests more quickly, pay for storage on a queue to regulate flow, or do some kind of internal code optimization to decrease resource demands. All of which cost money one way or the other. Alternately, you go up the food chain and determine the ROI on all the data you're receiving and maybe stop collecting some things that aren't valuable. If they are valuable, then it's a no-brainer to pay what you have to.

StillBored 5y ago

Failure codes themselves aren't back pressure though, its the client side "retry later" which creates that effect. But, if you have that behavior, its better just to signal it directly because a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. And if its gets bad enough, potentially results in livelock like behaviors. Simply slowing down ACK's or explicit "item queued, please wait before sending another" return codes are far more effective. See buffer crediting, flow control. OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).

deleted 5y ago
[deleted]
klodolph 5y ago
> ... a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. Well, two things. First, our clients didn't have a "simple" back off, retry system. They tended to have more sophisticated back-off and retry systems that didn't increase pressure on services that were overloaded. There are a number of different techniques you use to accomplish this and you can wrap up several of these techniques in a library, use the library in your client code, and have that client code (a fat client) be the official API for your service. Second, this also assumes that front-end load is significant to begin with. In our system, it definitely wasn't. Requests were super cheap. You could hammer the front-end extremely hard and it would just respond with "try again later" until there was some capacity available, and then it would let in some work. The size of an actual request was quite small, it was the size of the work represented by that request that was large. > OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency). Queues in our system were a necessary part of the design. The system would not operate efficiently without them, because work items needed to be batched in order to be processed efficiently. I don't think there's a way that you can indict a system for using queues unless you know something about the design requirements. There are absolutely other reasons to use queues other than smoothing out bursty behavior.