6 ms·

Backpressure – the resisted flow of data through software

96 points by jasim 5y ago

macintux 5y ago
Backpressure was a recurring topic at Basho. I thought I might find something from one of our engineers online, but instead I found these: a look at one client’s approach for using Riak, including how they implemented their own backpressure in front of it, and Fred Hebert‘s discussion on queues and backpressure. - https://blog.booking.com/using-riak-as-events-storage-part4.html https://blog.booking.com/using-riak-as-events-storage-part4.... - https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html
- bestcoder69 5y ago
  Freds blog is great. I keep finding myself needing to go back to an article I didn’t think was applicable to me at the time. The complete opposite of medium.com, for me, which I nearly always regret clicking on.
stkni 5y ago
Interesting. For two reasons. In the many years that I developed software this was known as the producer/consumer problem [1]. A name that is somewhat (to me) more expressive than back-pressure. But that's just my opinion! What's different now, as the article points out, is the growth of micro-service architectures. Effectively creating a lot of tiny producer/consumer problems across the solution space. Sometimes, I'm glad I don't write code anymore, but only sometimes :-) [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob...
- tootie 5y ago
  Yeah right. This seems like fancy talk for the same scaling problems we've had forever. And it was 100X worse before cloud computing. System-to-system scale problems are generally a lot easier to deal with because you always have the option of resetting expectations with users. Serving web requests has to be real-time, but if a data pipeline has to be overnight due to cost constraints, you can just make the argument to management and do it.
- stonemetal12 5y ago
  Yeah, when I was in school it was called "producer/consumer problem" too, back pressure was considered one of the solutions to the problem(what he calls control). I guess someone somewhere confused the two, and it kinda stuck.
lincpa 5y ago
It works very well. The Math-based Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow
benjaminwootton 5y ago
I have worked on event driven systems which read from queues for my entire career, messaging then Kafka. Surely in most business scenarios letting data back up on queues for a while suffices? In the situations where it doesn’t, add some more capacity. Back pressure is something I’ve never had to deal with even in HFT and electronic trading situations.
- klodolph 5y ago
  I've worked with systems where petabytes are flowing through it every day. If your data starts backing up, it costs lots of real money in terms of hard disk space. Backpressure crosses organizational boundaries--the team running the system which consumes data is not the same team running the system which produces data. We exposed backpressure through our APIs... if our system was overloaded, we returned a specific error code. Clients were expected to retry, sensibly. Sometimes "add more capacity" just costs too much.
  - tootie 5y ago
    Obviously you've got to pay for something. You can either increase your compute capacity to process requests more quickly, pay for storage on a queue to regulate flow, or do some kind of internal code optimization to decrease resource demands. All of which cost money one way or the other. Alternately, you go up the food chain and determine the ROI on all the data you're receiving and maybe stop collecting some things that aren't valuable. If they are valuable, then it's a no-brainer to pay what you have to.
  - StillBored 5y ago
    Failure codes themselves aren't back pressure though, its the client side "retry later" which creates that effect. But, if you have that behavior, its better just to signal it directly because a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. And if its gets bad enough, potentially results in livelock like behaviors. Simply slowing down ACK's or explicit "item queued, please wait before sending another" return codes are far more effective. See buffer crediting, flow control. OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).
kristov 5y ago
With pipes between threads, the OS is able to suspend the reading thread when the buffer is empty, or suspend the writing thread when the buffer is full, because the OS "owns" the buffer and the threads. With distributed systems connected via Kafka for example, there is no "process" overseeing both ends that can suspend or resume hollistically, so its difficult for producers to know and respond to back pressure.
- pjc50 5y ago
  Isn't the buffer then the TCP one, which has its own backpressure system?
  - singron 5y ago
    TCP will only slow down the writer if the network or the Kafka broker can't keep up. It the Kafka consumer can't keep up, their backlog will grow unhindered until Kafka runs out of disk space. I wish there was an easier built in way to do it, but a producer could read the backlog length of it's consumers and try to throttle itself.
a-dub 5y ago
i don't see eye to eye with the author. afaik the only time a modern software developer needs to worry about backpressure is if they're using asynchronous or otherwise non-flow-controlled i/o. neither of these cases nor even a discussion of synchronous vs. asynchronous i/o were mentioned.
- bestcoder69 5y ago
  I agree it’s less of an issue in more traditional, threaded req/reply servers. But queues can appear in a bunch of places. Like your load balancer may queue up connections from clients and dole them out to app servers as they free up capacity. Normally the queue is emptyish, but app servers will do backpressure via blocking as needed, so the load balancer connection queue fills. In this case you probably don’t even want to add capacity (to an extent) because what’s the point if clients have timeouts. Also eventually you might need a worker pool pulling from a queue. Just guessing, because they seem to appear eventually in/near web servers. You can spin up a bajillion lambdas simultaneously like a madman if you want, but odds are there’s a resource constraint on the source and/or destination data store side at least. So again you’ve got async execution, a growing queue, and bottlenecks to think about still. But maybe with threaded code it’s like a “pit of success” for traffic shaping. You only have so many threads you can use, so you configure a pool size and think about what happens when you’re at capacity, and define some nice predictable failure behavior and test it. (As opposed to the node.js server that will handle more connections but surprise you with a weird bottleneck that you don’t handle gracefully). So then the trad server developer can go longer (or forever) without being bitten by the queues that remain elsewhere in their system.
- bborud 5y ago
  They key is "flow controlled". Syncronicity doesn't really enter into it, but I can see how people can get confused and think it's about that because asynchronous designs tend to introduce a greater degree of decoupling.
- colanderman 5y ago
  Synchronous systems can benefit from backpressure. Clients can and do open multiple concurrent connections to such systems (common to work around the inevitable throughput limitations of synchronous systems)... it is still helpful to tell any given connections, no, I'm at my concurrency limit, any work you give me is just going to sit in a queue and increase your service time, so please don't.
dugmartin 5y ago
Backpressure is a feature of Elixir's GenStage: https://hexdocs.pm/gen_stage/GenStage.html https://hexdocs.pm/gen_stage/GenStage.html and these additional abstactions are built on top of it: - https://github.com/dashbitco/broadway https://github.com/dashbitco/broadway - https://github.com/dashbitco/flow https://github.com/dashbitco/flow
- lastofthemojito 5y ago
  Also a concept in Apache Nifi: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Backpressure https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#...
deleted 5y ago
[deleted]
est 5y ago
> Backpressure Strategies: control, buffer, drop Or can we add autoscale? I encountered this exact problem today as I were looking for a simple to setup and maintain autoscaler solution. The programs feed in the backpressure metrics to monitor, the autoscaler would tell me how many pods/consumers there needs to be, based on naive estimation or prediction by yesterday's traffic. K8s seems to be an over kill and with stateful pods problems, KEDA looks promising, but I wish there's much simpler ones.
- freemint 5y ago
  Unless your monetary resources are infinite the bank will be your bank pressure then.
klodolph 5y ago
I think the key insight here is that just autoscaling your service to keep queue length short is not always gonna work, or not always going to be cost effective, and upstream sources should be prepared to reduce the amount of work they send (unless there is a business reason to do otherwise). Making the queue longer or just dropping data that's already accepted into the queue will sometimes break people's assumptions about how your system works. You can adjust queue size from time to time if you need it, but if your service sees increased use over time, the amount of variance in queue size can get a bit crazy. I worked on a system a while back which processed petabytes of data... autoscaling was out of the question (we had a fixed amount of hardware to process the data, and the lead time for getting new hardware was not especially short), and buffering the data would require eye-watering amounts of disk space. We just buffered as much as we could and then returned a "try again later" error code. We made sure that data accepted by the system was processed within a short enough window. We made sure that the cost of submitting a work item was so small that you could really hammer the front-end as much as you wanted, and it would just say "try again later" or accept work items. I think one of the lessons here is that you need to think long and hard about what error conditions you want to expose upstream. The farther you propagate an error condition, the more different ways you can solve it, but the more complicated your clients get. For example, disks sometimes fail. You can decide to not propagate disk failure beyond an individual machine, load up RAID 1 (mirror) in all your file servers, and back everything up. Or you can push disk failures farther up the stack, and recover from failed disks at a higher level, with lower hardware cost but higher implementation complexity. And if you build a bunch of systems assuming that each part is always working, you run very serious risks of cascading failures once one part does inevitably fail. Obviously, small enough systems running on cloud hardware can usually be autoscaled just fine.
notacoward 5y ago
Lack of backpressure is perhaps the most common newbie mistake in distributed systems. I've just seen so many failures because the servers were overwhelmed and the clients were greedy/impatient (didn't react well to anything less than an immediate response even when they were doing everything they could to overwhelm). It should be a primary design consideration for any such system being built anew. > Backpressure Strategies: control, buffer, drop One of my favorite hacks for Gluster (at Facebook) was in this area. We were using bog-standard NFS as the connection protocol, so neither explicit control nor dropping were options. (People who have always had that luxury with their own thrift/grpc/whatever protocols and libraries don't realize how much of a pain it is not to have it.) That left buffering, which led to memory pressure and latency bubbles. What I ended up implementing was implicit control. We'd just stop reading on a particular client's socket for increasing numbers of milliseconds if overall latency was too high and its queue was too long. This leveraged information we already had, and "borrowed" TCP's flow/buffer control as our own. Clients couldn't ignore it; they'd block in send() or somewhere equivalent. It worked really well to smooth out many of the worst latency spikes. If you're ever stuck in a similar situation, I highly recommend this approach.
- a11r 5y ago
  This is indeed the right way to leverage TCP flow control, and works well at the layer that knows exactly what is coming over the socket. The pitfall here is if semantically independent data is being multiplexed over the same TCP connection then suspending reads may cause collateral damage and in the worst case livelock situations. Examples of multiplexing include separate streams and control frames in HTTP/2, in-band health checks and keepalives in various protocols, etc.
- winrid 5y ago
  Wait, so for communication between services you used sockets over NFS?
  - notacoward 5y ago
    No, NFS over TCP. The service we supported was a POSIX filesystem used by many thousands of clients.
StillBored 5y ago
Back pressure, AKA flow control.
- blueflow 5y ago
  This needs to be upvoted more... younger developers don't know about the problems that already had solutions for 40 years. RTS/CTS flow control is still in use today...
eqvinox 5y ago
I'm quite befuddled by this article, because the perspective is pretty exactly 180° of what I'm used to. I don't (try to) control or manage backpressure. Backpressure controls - or rather, steers - me/my code. Getting backpressure right is massively important and frequently under-considered design aspect. The most significant thing to understand IMHO is that you need to look at backpressure in the context of the larger system. More concretely, you need to ask yourself "is this telling me that another task should run right now"? Both for synchronously driven sources (e.g. blocking write() syscall) as well as asynchronously driven sources (event loop with "can send" flag), you need to make sure you (a) don't block other progress (held locks on DB) and (b) actually pull up other tasks (particularly important for partially-asynchronous code). And this leads straight to the single most impactful design/scaling decision for these systems: is the source maintaining its outbound scheduling at a high enough level? This is best clarified with an example: say you have a key/value store that sends out change notifications per key. Does your application need every single step, or just the most recent value? And if it's the latter: you shouldn't even have a "true" queue! Instead, the code needs to track which keys are pending for which receiver, and whenever the receiver can accept data, it needs to send the most recent data for pending keys. This is the difference between an unbounded queue and a bounded by number of keys queue. The former can fail disastrously, the latter will just start lagging but once it reaches the worst case, it just stays there. > For example, if someone says: “I made this new library with built-in backpressure…” they probably mean it somehow helps you manage backpressure, not that it actually intrinsically is the cause of it. Backpressure is not “desirable” except when it’s inescapable and you want to protect something else from receiving it. … and this really encapsulates my disagreement. Backpressure is absolutely desirable. I want my libraries to provide me with backpressure when there is reason to do so, and do it in a way I can actually "consume" the backpressure! And lastly: > Backpressure Strategies: control, buffer, drop "buffer" and "drop" are in most cases just bugs. And very hard to debug & track down, due to their load dependency.
- specialist 5y ago
  My last team (3 years ago) mostly used nodejs. Our services would tip over under load. Explaining backpressure was a lot of work. With zero reward. Less so for fixing it. I was bemused that the entire nodejs ecosystem, community was oblivious to this issue. Hard to take any of it seriously.

96 points by jasim 5y ago

Backpressure was a recurring topic at Basho. I thought I might find something from one of our engineers online, but instead I found these: a look at one client’s approach for using Riak, including how they implemented their own backpressure in front of it, and Fred Hebert‘s discussion on queues and backpressure. - https://blog.booking.com/using-riak-as-events-storage-part4.html https://blog.booking.com/using-riak-as-events-storage-part4.... - https://ferd.ca/queues-don-t-fix-overload.html https://ferd.ca/queues-don-t-fix-overload.html

bestcoder69 5y ago
Freds blog is great. I keep finding myself needing to go back to an article I didn’t think was applicable to me at the time. The complete opposite of medium.com, for me, which I nearly always regret clicking on.

stkni 5y ago

Interesting. For two reasons. In the many years that I developed software this was known as the producer/consumer problem [1]. A name that is somewhat (to me) more expressive than back-pressure. But that's just my opinion! What's different now, as the article points out, is the growth of micro-service architectures. Effectively creating a lot of tiny producer/consumer problems across the solution space. Sometimes, I'm glad I don't write code anymore, but only sometimes :-) [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_prob...

tootie 5y ago
Yeah right. This seems like fancy talk for the same scaling problems we've had forever. And it was 100X worse before cloud computing. System-to-system scale problems are generally a lot easier to deal with because you always have the option of resetting expectations with users. Serving web requests has to be real-time, but if a data pipeline has to be overnight due to cost constraints, you can just make the argument to management and do it.
stonemetal12 5y ago
Yeah, when I was in school it was called "producer/consumer problem" too, back pressure was considered one of the solutions to the problem(what he calls control). I guess someone somewhere confused the two, and it kinda stuck.

lincpa 5y ago

It works very well. The Math-based Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow

benjaminwootton 5y ago

I have worked on event driven systems which read from queues for my entire career, messaging then Kafka. Surely in most business scenarios letting data back up on queues for a while suffices? In the situations where it doesn’t, add some more capacity. Back pressure is something I’ve never had to deal with even in HFT and electronic trading situations.

klodolph 5y ago
I've worked with systems where petabytes are flowing through it every day. If your data starts backing up, it costs lots of real money in terms of hard disk space. Backpressure crosses organizational boundaries--the team running the system which consumes data is not the same team running the system which produces data. We exposed backpressure through our APIs... if our system was overloaded, we returned a specific error code. Clients were expected to retry, sensibly. Sometimes "add more capacity" just costs too much.
- tootie 5y ago
  Obviously you've got to pay for something. You can either increase your compute capacity to process requests more quickly, pay for storage on a queue to regulate flow, or do some kind of internal code optimization to decrease resource demands. All of which cost money one way or the other. Alternately, you go up the food chain and determine the ROI on all the data you're receiving and maybe stop collecting some things that aren't valuable. If they are valuable, then it's a no-brainer to pay what you have to.
- StillBored 5y ago
  Failure codes themselves aren't back pressure though, its the client side "retry later" which creates that effect. But, if you have that behavior, its better just to signal it directly because a simple fail, back off, retry system actually increases overall load due to duplicate requests retrying. And if its gets bad enough, potentially results in livelock like behaviors. Simply slowing down ACK's or explicit "item queued, please wait before sending another" return codes are far more effective. See buffer crediting, flow control. OTOH, I'm sort of amazed by the people who reach for queues to solve loading issues without first assuring that the system works properly at full load without any queuing. Queue's only serve to smooth out busty behavior and come with their own problems (buffer bloat related latency).

kristov 5y ago

With pipes between threads, the OS is able to suspend the reading thread when the buffer is empty, or suspend the writing thread when the buffer is full, because the OS "owns" the buffer and the threads. With distributed systems connected via Kafka for example, there is no "process" overseeing both ends that can suspend or resume hollistically, so its difficult for producers to know and respond to back pressure.

pjc50 5y ago
Isn't the buffer then the TCP one, which has its own backpressure system?
- singron 5y ago
  TCP will only slow down the writer if the network or the Kafka broker can't keep up. It the Kafka consumer can't keep up, their backlog will grow unhindered until Kafka runs out of disk space. I wish there was an easier built in way to do it, but a producer could read the backlog length of it's consumers and try to throttle itself.

a-dub 5y ago

i don't see eye to eye with the author. afaik the only time a modern software developer needs to worry about backpressure is if they're using asynchronous or otherwise non-flow-controlled i/o. neither of these cases nor even a discussion of synchronous vs. asynchronous i/o were mentioned.

bestcoder69 5y ago
I agree it’s less of an issue in more traditional, threaded req/reply servers. But queues can appear in a bunch of places. Like your load balancer may queue up connections from clients and dole them out to app servers as they free up capacity. Normally the queue is emptyish, but app servers will do backpressure via blocking as needed, so the load balancer connection queue fills. In this case you probably don’t even want to add capacity (to an extent) because what’s the point if clients have timeouts. Also eventually you might need a worker pool pulling from a queue. Just guessing, because they seem to appear eventually in/near web servers. You can spin up a bajillion lambdas simultaneously like a madman if you want, but odds are there’s a resource constraint on the source and/or destination data store side at least. So again you’ve got async execution, a growing queue, and bottlenecks to think about still. But maybe with threaded code it’s like a “pit of success” for traffic shaping. You only have so many threads you can use, so you configure a pool size and think about what happens when you’re at capacity, and define some nice predictable failure behavior and test it. (As opposed to the node.js server that will handle more connections but surprise you with a weird bottleneck that you don’t handle gracefully). So then the trad server developer can go longer (or forever) without being bitten by the queues that remain elsewhere in their system.
bborud 5y ago
They key is "flow controlled". Syncronicity doesn't really enter into it, but I can see how people can get confused and think it's about that because asynchronous designs tend to introduce a greater degree of decoupling.
colanderman 5y ago
Synchronous systems can benefit from backpressure. Clients can and do open multiple concurrent connections to such systems (common to work around the inevitable throughput limitations of synchronous systems)... it is still helpful to tell any given connections, no, I'm at my concurrency limit, any work you give me is just going to sit in a queue and increase your service time, so please don't.

dugmartin 5y ago

Backpressure is a feature of Elixir's GenStage: https://hexdocs.pm/gen_stage/GenStage.html https://hexdocs.pm/gen_stage/GenStage.html and these additional abstactions are built on top of it: - https://github.com/dashbitco/broadway https://github.com/dashbitco/broadway - https://github.com/dashbitco/flow https://github.com/dashbitco/flow

lastofthemojito 5y ago
Also a concept in Apache Nifi: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Backpressure https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#...

deleted 5y ago

[deleted]

est 5y ago

> Backpressure Strategies: control, buffer, drop Or can we add autoscale? I encountered this exact problem today as I were looking for a simple to setup and maintain autoscaler solution. The programs feed in the backpressure metrics to monitor, the autoscaler would tell me how many pods/consumers there needs to be, based on naive estimation or prediction by yesterday's traffic. K8s seems to be an over kill and with stateful pods problems, KEDA looks promising, but I wish there's much simpler ones.

freemint 5y ago
Unless your monetary resources are infinite the bank will be your bank pressure then.

klodolph 5y ago

I think the key insight here is that just autoscaling your service to keep queue length short is not always gonna work, or not always going to be cost effective, and upstream sources should be prepared to reduce the amount of work they send (unless there is a business reason to do otherwise). Making the queue longer or just dropping data that's already accepted into the queue will sometimes break people's assumptions about how your system works. You can adjust queue size from time to time if you need it, but if your service sees increased use over time, the amount of variance in queue size can get a bit crazy. I worked on a system a while back which processed petabytes of data... autoscaling was out of the question (we had a fixed amount of hardware to process the data, and the lead time for getting new hardware was not especially short), and buffering the data would require eye-watering amounts of disk space. We just buffered as much as we could and then returned a "try again later" error code. We made sure that data accepted by the system was processed within a short enough window. We made sure that the cost of submitting a work item was so small that you could really hammer the front-end as much as you wanted, and it would just say "try again later" or accept work items. I think one of the lessons here is that you need to think long and hard about what error conditions you want to expose upstream. The farther you propagate an error condition, the more different ways you can solve it, but the more complicated your clients get. For example, disks sometimes fail. You can decide to not propagate disk failure beyond an individual machine, load up RAID 1 (mirror) in all your file servers, and back everything up. Or you can push disk failures farther up the stack, and recover from failed disks at a higher level, with lower hardware cost but higher implementation complexity. And if you build a bunch of systems assuming that each part is always working, you run very serious risks of cascading failures once one part does inevitably fail. Obviously, small enough systems running on cloud hardware can usually be autoscaled just fine.

notacoward 5y ago

Lack of backpressure is perhaps the most common newbie mistake in distributed systems. I've just seen so many failures because the servers were overwhelmed and the clients were greedy/impatient (didn't react well to anything less than an immediate response even when they were doing everything they could to overwhelm). It should be a primary design consideration for any such system being built anew. > Backpressure Strategies: control, buffer, drop One of my favorite hacks for Gluster (at Facebook) was in this area. We were using bog-standard NFS as the connection protocol, so neither explicit control nor dropping were options. (People who have always had that luxury with their own thrift/grpc/whatever protocols and libraries don't realize how much of a pain it is not to have it.) That left buffering, which led to memory pressure and latency bubbles. What I ended up implementing was implicit control. We'd just stop reading on a particular client's socket for increasing numbers of milliseconds if overall latency was too high and its queue was too long. This leveraged information we already had, and "borrowed" TCP's flow/buffer control as our own. Clients couldn't ignore it; they'd block in send() or somewhere equivalent. It worked really well to smooth out many of the worst latency spikes. If you're ever stuck in a similar situation, I highly recommend this approach.

a11r 5y ago
This is indeed the right way to leverage TCP flow control, and works well at the layer that knows exactly what is coming over the socket. The pitfall here is if semantically independent data is being multiplexed over the same TCP connection then suspending reads may cause collateral damage and in the worst case livelock situations. Examples of multiplexing include separate streams and control frames in HTTP/2, in-band health checks and keepalives in various protocols, etc.
winrid 5y ago
Wait, so for communication between services you used sockets over NFS?
- notacoward 5y ago
  No, NFS over TCP. The service we supported was a POSIX filesystem used by many thousands of clients.

StillBored 5y ago

Back pressure, AKA flow control.

blueflow 5y ago
This needs to be upvoted more... younger developers don't know about the problems that already had solutions for 40 years. RTS/CTS flow control is still in use today...

eqvinox 5y ago

I'm quite befuddled by this article, because the perspective is pretty exactly 180° of what I'm used to. I don't (try to) control or manage backpressure. Backpressure controls - or rather, steers - me/my code. Getting backpressure right is massively important and frequently under-considered design aspect. The most significant thing to understand IMHO is that you need to look at backpressure in the context of the larger system. More concretely, you need to ask yourself "is this telling me that another task should run right now"? Both for synchronously driven sources (e.g. blocking write() syscall) as well as asynchronously driven sources (event loop with "can send" flag), you need to make sure you (a) don't block other progress (held locks on DB) and (b) actually pull up other tasks (particularly important for partially-asynchronous code). And this leads straight to the single most impactful design/scaling decision for these systems: is the source maintaining its outbound scheduling at a high enough level? This is best clarified with an example: say you have a key/value store that sends out change notifications per key. Does your application need every single step, or just the most recent value? And if it's the latter: you shouldn't even have a "true" queue! Instead, the code needs to track which keys are pending for which receiver, and whenever the receiver can accept data, it needs to send the most recent data for pending keys. This is the difference between an unbounded queue and a bounded by number of keys queue. The former can fail disastrously, the latter will just start lagging but once it reaches the worst case, it just stays there. > For example, if someone says: “I made this new library with built-in backpressure…” they probably mean it somehow helps you manage backpressure, not that it actually intrinsically is the cause of it. Backpressure is not “desirable” except when it’s inescapable and you want to protect something else from receiving it. … and this really encapsulates my disagreement. Backpressure is absolutely desirable. I want my libraries to provide me with backpressure when there is reason to do so, and do it in a way I can actually "consume" the backpressure! And lastly: > Backpressure Strategies: control, buffer, drop "buffer" and "drop" are in most cases just bugs. And very hard to debug & track down, due to their load dependency.

specialist 5y ago
My last team (3 years ago) mostly used nodejs. Our services would tip over under load. Explaining backpressure was a lot of work. With zero reward. Less so for fixing it. I was bemused that the entire nodejs ecosystem, community was oblivious to this issue. Hard to take any of it seriously.