4 ms·

A lot of the ideas mentioned in this article have been incorporated into SocketCluster https://socketcluster.io/ https://socketcluster.io/ - It's an end-to-end

A lot of the ideas mentioned in this article have been incorporated into SocketCluster https://socketcluster.io/ https://socketcluster.io/ - It's an end-to-end RPC + pub/sub client/server library and it provides a versatile API to manage backpressure. This point is particularly relevant: > Exerting backpressure is why we don’t retry except at the ends of a system. If we’re going to fail, we want to propagate that failure information to where it can be used. This is an excellent observation and why SC is an end-to-end system which extends all the way to the end user (as opposed to Redis, Kafka, RabbitMQ, STOMP, NSQ and MQTT... which are all designed to be used on the backend in a permissioned environment and do not extend to the end user; they require an additional communication and authentication layer for last-mile delivery). If a message fails to be delivered, then the end user's client is the one which should re-submit the message. The principle of putting most of the control in the hands of end users has been at the core of SC's design since the beginning.

twic 5y ago
> it provides a versatile API to manage backpressure I think whoever wrote this library has a very different idea of what backpressure is to me: > For these reasons, SocketCluster exposes a simple API for tracking stream backpressure and it lets you immediately kill streams or groups of streams which are becoming overly congested. SocketCluster lets you measure the backpressure of individual streams within a socket and also the aggregate backpressure of all streams within the socket. https://socketcluster.io/docs/streams-and-backpressure/ https://socketcluster.io/docs/streams-and-backpressure/ To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. It is an application-level concern in the upstream stages how that slowing-down can be achieved; it might involve coalescing updates, dropping updates that are made irrelevant, reducing sample frequency, etc. "This got slow so we killed it lol" does not sound useful.
- applepple 5y ago
  >> To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. Maybe the kinds of streams you're thinking of are more like 'jobs' (e.g. task queues). Streams in SC are focused around the pub/sub use case. They are end-to-end (user to user). So if new data is being output by a consumer stream faster than the user's front end can process it, what other option is there but to stop consuming (kill) the stream on the receiver's side? The front end application could process messages in parallel, but if it did, the backpressure would not build up in the first place. Backpressure only builds up if you make the stream await the processing of each message in a series. If there is no explicit await, then the backpressure is always going to be 0. End-to-end streams are far easier to manage than the backend-only 'task queues' advocated by Rabbit MQ, Kafka, NSQ, etc. Backend streams are unweildy because you can't give feedback to the user on the front end if something goes wrong, the stream has to process the message no matter what... This is because message queues are completely disconnected from front end applications. Not sure why someone would architect a system like this in the first place. That seems Kafkaesque (pun intended). Isn't it better if the system can catch an issue as soon as it happens and give the originator of the action an opportunity to resolve the issue as soon as possible? The originator of an action is usually best placed to figure out how to handle issues related to the action that they've just performed (e.g. retry, show an error...). Dealing with realtime data end-to-end is far more straight forward.

twic 5y ago

> it provides a versatile API to manage backpressure I think whoever wrote this library has a very different idea of what backpressure is to me: > For these reasons, SocketCluster exposes a simple API for tracking stream backpressure and it lets you immediately kill streams or groups of streams which are becoming overly congested. SocketCluster lets you measure the backpressure of individual streams within a socket and also the aggregate backpressure of all streams within the socket. https://socketcluster.io/docs/streams-and-backpressure/ https://socketcluster.io/docs/streams-and-backpressure/ To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. It is an application-level concern in the upstream stages how that slowing-down can be achieved; it might involve coalescing updates, dropping updates that are made irrelevant, reducing sample frequency, etc. "This got slow so we killed it lol" does not sound useful.

applepple 5y ago
>> To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. Maybe the kinds of streams you're thinking of are more like 'jobs' (e.g. task queues). Streams in SC are focused around the pub/sub use case. They are end-to-end (user to user). So if new data is being output by a consumer stream faster than the user's front end can process it, what other option is there but to stop consuming (kill) the stream on the receiver's side? The front end application could process messages in parallel, but if it did, the backpressure would not build up in the first place. Backpressure only builds up if you make the stream await the processing of each message in a series. If there is no explicit await, then the backpressure is always going to be 0. End-to-end streams are far easier to manage than the backend-only 'task queues' advocated by Rabbit MQ, Kafka, NSQ, etc. Backend streams are unweildy because you can't give feedback to the user on the front end if something goes wrong, the stream has to process the message no matter what... This is because message queues are completely disconnected from front end applications. Not sure why someone would architect a system like this in the first place. That seems Kafkaesque (pun intended). Isn't it better if the system can catch an issue as soon as it happens and give the originator of the action an opportunity to resolve the issue as soon as possible? The originator of an action is usually best placed to figure out how to handle issues related to the action that they've just performed (e.g. retry, show an error...). Dealing with realtime data end-to-end is far more straight forward.