3 ms·

The biggest pain point I've experienced with these pubsub services is how to negotiate client connections so that all the subscribers to a topic are directly co

The biggest pain point I've experienced with these pubsub services is how to negotiate client connections so that all the subscribers to a topic are directly connected to the same machine. In other words, clients' messages do not have to ever be relayed at the networking level. This means providing some reconnecting or connection-info data for the client, and it's why I think most of these frameworks fail to improve on home cooked solutions: you have to touch the message to help squeeze out efficiency. A non-IP based routing mechanism requires some kind of header/parsing/deserialization of the message being passed. In application level code this is quite slow, you have to query some kind of map synchronized about cluster nodes. This is especially shitty if message addresses are ephemeral, especially per RPC call (which is the default behaviour in most cluster-based RPC frameworks), because then your near cache of the cluster map is never populated. Alternatively you can just broadcast to everyone (skip the cluster map) but then why cluster? It would be great if we just had IPv6 adoption though. Then you could just uh, route. The DNS based routing in stuff like Kubernetes is close but not for external Internet facing clients. The whole point is to avoid a bastion host.

threeseed 7y ago
Wouldn't the ingress help here with sticky sessions e.g. cookies or a header. I am assuming the RPC is happening over WebSockets though.
yasserf 7y ago
I totally agree. Having a single point of entry gives us some small benefits like a single connection/simplified deployments but it isn't efficient to have a provider connected on one node in a cluster and have to forward data to subscribers on another just because of how they were distributed by a load-balancer. Theres actually an action in the handshake protocol which allows a node in a cluster to redirect a connection to a different/more optimal node. It was used for multi-tenancy clusters previously. In theory you can run multiple isolated clusters (or individual nodes) and have multiple connections from the client, routed based on which subscription lives where. It's a pattern I have been talking about with a couple of users but it hasn't been required by anyone yet. It isn't as efficient as doing it on a network layer but it at least reduced intercluster traffic dramatically.
- saurik 7y ago
  I used to be really concerned about this, but one day I sat down and did the math and realized that the worst case scenario is that I'd have no more than two times as many messages, which means this isn't a scalability issue: if a topic has N subscribers, at bare minimum you are going to have N+1 messages to send a given event (one to receive it into the cluster, and N to dispatch it to the subscribers); if everyone is connected to some random machine, then you need to take the incoming event and send it to the right internal machine that is storing that topic (which is one extra message) and then that machine will need to indirectly send the message to the connecting computers of the subscribers (which is one extra message per subscriber), so you end up with 2N+2 messages. The alternative, where users cluster themselves based on topic with their entry connections, is that these individual end users have to maintain a ton of separate connections for inbound messages for each of the topics they are interested in, which is often going to work out to be much worse for everyone involved.

threeseed 7y ago

Wouldn't the ingress help here with sticky sessions e.g. cookies or a header. I am assuming the RPC is happening over WebSockets though.

yasserf 7y ago

I totally agree. Having a single point of entry gives us some small benefits like a single connection/simplified deployments but it isn't efficient to have a provider connected on one node in a cluster and have to forward data to subscribers on another just because of how they were distributed by a load-balancer. Theres actually an action in the handshake protocol which allows a node in a cluster to redirect a connection to a different/more optimal node. It was used for multi-tenancy clusters previously. In theory you can run multiple isolated clusters (or individual nodes) and have multiple connections from the client, routed based on which subscription lives where. It's a pattern I have been talking about with a couple of users but it hasn't been required by anyone yet. It isn't as efficient as doing it on a network layer but it at least reduced intercluster traffic dramatically.

saurik 7y ago
I used to be really concerned about this, but one day I sat down and did the math and realized that the worst case scenario is that I'd have no more than two times as many messages, which means this isn't a scalability issue: if a topic has N subscribers, at bare minimum you are going to have N+1 messages to send a given event (one to receive it into the cluster, and N to dispatch it to the subscribers); if everyone is connected to some random machine, then you need to take the incoming event and send it to the right internal machine that is storing that topic (which is one extra message) and then that machine will need to indirectly send the message to the connecting computers of the subscribers (which is one extra message per subscriber), so you end up with 2N+2 messages. The alternative, where users cluster themselves based on topic with their entry connections, is that these individual end users have to maintain a ton of separate connections for inbound messages for each of the topics they are interested in, which is often going to work out to be much worse for everyone involved.