3 ms·
>> To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. Maybe the ki
by applepple 5y ago
>> To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow.
Maybe the kinds of streams you're thinking of are more like 'jobs' (e.g. task queues).
Streams in SC are focused around the pub/sub use case. They are end-to-end (user to user). So if new data is being output by a consumer stream faster than the user's front end can process it, what other option is there but to stop consuming (kill) the stream on the receiver's side?
The front end application could process messages in parallel, but if it did, the backpressure would not build up in the first place. Backpressure only builds up if you make the stream await the processing of each message in a series. If there is no explicit await, then the backpressure is always going to be 0.
End-to-end streams are far easier to manage than the backend-only 'task queues' advocated by Rabbit MQ, Kafka, NSQ, etc. Backend streams are unweildy because you can't give feedback to the user on the front end if something goes wrong, the stream has to process the message no matter what... This is because message queues are completely disconnected from front end applications. Not sure why someone would architect a system like this in the first place. That seems Kafkaesque (pun intended). Isn't it better if the system can catch an issue as soon as it happens and give the originator of the action an opportunity to resolve the issue as soon as possible? The originator of an action is usually best placed to figure out how to handle issues related to the action that they've just performed (e.g. retry, show an error...).
Dealing with realtime data end-to-end is far more straight forward.