4 ms·

> Running your own pub/sub service is easy Yep, creating a barebones pub/sub service is pretty trivial (and fun!). But then you may find yourself slowly addin

by SEMW 10y ago

> Running your own pub/sub service is easy Yep, creating a barebones pub/sub service is pretty trivial (and fun!). But then you may find yourself slowly adding things that a decent pub/sub service should have. E.g. connection state recovery, so that a client on a phone driving through a tunnel gets all their messages (in the right order) once they reconnect. Presence, and presence data. Stored history. A permissions system. Encryption. Smart push notifications. Decent multi-region federation, so a channel that happens to only have subscribers in ap-southeast doesn't roundtrip messages to us-east. Guaranteed message ordering preservation. And so on. And eventually, you end up spending what adds up to quite a lot of effort, which could have been spent in ways other than slowly recreating the SAAS. I think that's the case for a lot of SAAS products (that creating a barebones version of the product yourself is pretty trivial, but the SAAS is still a useful product, since often people don't just want a barebones version). (Disclaimer: I work at Ably)

jondubois 10y ago
There are a quite a few open source pub/sub solutions which handle connection state recovery, authentication, encryption, in-order-and-exactly-once delivery, auto-scalability (on Kubertetes or Swarm)... And they give you even more flexibility than what you can get from a third-party service (which give you limited or no ability to customize back-end logic). There might be a few minor features which the third-party service might have which the open source solution doesn't have; but you can always extend the open source solution to have those features; this not the case with the third-party service; in that case you're stuck with whatever features are being offered by the service... For example, if your product requires that you have longer channel names, a more lightweight protocol, ability to batch messages together in a particular way, transform or filter or analyse data streams on the backend as they arrive, apply advanced authorization rules on the backend or to integrate directly with data from your own database... you need an open source solution. I would say that the argument for using third-party pub/sub services is the opposite of the one you made; it's good for simple projects which have limited requirements, but not good for larger companies with more advanced requirements.
- SEMW 10y ago
  Thanks for the well-thought-out response, I appreciate it. Of course you're right that for some usecases, rolling your own infrastructure on Kubertetes or Swarm with an open source solution rather than using a SAAS is the right way to go. (But then, the same thing is true of most SAAS products). BTW, I'd be interested to know if you had a particular open source product in mind in your first para? > For example, if your product requires that you have longer channel names, a more lightweight protocol, ability to batch messages together in a particular way, transform or filter or analyse data streams on the backend as they arrive, apply advanced authorization rules on the backend or to integrate directly with data from your own database... you need an open source solution. I'd query a few of these - - Re channel name length: we have no particular limit on that...? - Re lightweight protocols: The article we're commenting on is actually about us adding support for more protocols :) (including MQTT, which is pretty lightweight - though that one's not in general availability yet). We also support queues as an alternate way to consume data, which gives you AMQP and STOMP too. - Re batching: Clients can batch messages together into a single protocol messages as they like, just by passing an array of messages to publish(), and subscribers will receive them all together. (Which assumes they're all for the same channel, but if not, not sure what it'd mean to batch them). - Re analysing data streams on the backend as they arrive: you can subscribe to messages on your server, and then do what you like with them. (Or alternatively use our queueing feature, which lets you have arbitrarily many independent backend workers consuming messages from the queues). It is true that you can't filter messages before they're broadcast to other clients, though. I guess our answer to that would be to have the client publish to a channel that only the server has permissions to subscribe to, set then have server filter those and rebroadcast on a different channel. (Probably in combination with the queues, so you can have multiple server workers doing this). It's true that that will add a bit more latency, though. - Re advanced authorization rules -- our token capabilities system allows the server pretty fine-grained control over the permissions it grants each client.[0] But sure, it's never going to be as flexible as a completely custom solution. [0] https://www.ably.io/documentation/general/authentication https://www.ably.io/documentation/general/authentication

by SEMW 10y ago

jondubois 10y ago

There are a quite a few open source pub/sub solutions which handle connection state recovery, authentication, encryption, in-order-and-exactly-once delivery, auto-scalability (on Kubertetes or Swarm)... And they give you even more flexibility than what you can get from a third-party service (which give you limited or no ability to customize back-end logic). There might be a few minor features which the third-party service might have which the open source solution doesn't have; but you can always extend the open source solution to have those features; this not the case with the third-party service; in that case you're stuck with whatever features are being offered by the service... For example, if your product requires that you have longer channel names, a more lightweight protocol, ability to batch messages together in a particular way, transform or filter or analyse data streams on the backend as they arrive, apply advanced authorization rules on the backend or to integrate directly with data from your own database... you need an open source solution. I would say that the argument for using third-party pub/sub services is the opposite of the one you made; it's good for simple projects which have limited requirements, but not good for larger companies with more advanced requirements.

SEMW 10y ago
Thanks for the well-thought-out response, I appreciate it. Of course you're right that for some usecases, rolling your own infrastructure on Kubertetes or Swarm with an open source solution rather than using a SAAS is the right way to go. (But then, the same thing is true of most SAAS products). BTW, I'd be interested to know if you had a particular open source product in mind in your first para? > For example, if your product requires that you have longer channel names, a more lightweight protocol, ability to batch messages together in a particular way, transform or filter or analyse data streams on the backend as they arrive, apply advanced authorization rules on the backend or to integrate directly with data from your own database... you need an open source solution. I'd query a few of these - - Re channel name length: we have no particular limit on that...? - Re lightweight protocols: The article we're commenting on is actually about us adding support for more protocols :) (including MQTT, which is pretty lightweight - though that one's not in general availability yet). We also support queues as an alternate way to consume data, which gives you AMQP and STOMP too. - Re batching: Clients can batch messages together into a single protocol messages as they like, just by passing an array of messages to publish(), and subscribers will receive them all together. (Which assumes they're all for the same channel, but if not, not sure what it'd mean to batch them). - Re analysing data streams on the backend as they arrive: you can subscribe to messages on your server, and then do what you like with them. (Or alternatively use our queueing feature, which lets you have arbitrarily many independent backend workers consuming messages from the queues). It is true that you can't filter messages before they're broadcast to other clients, though. I guess our answer to that would be to have the client publish to a channel that only the server has permissions to subscribe to, set then have server filter those and rebroadcast on a different channel. (Probably in combination with the queues, so you can have multiple server workers doing this). It's true that that will add a bit more latency, though. - Re advanced authorization rules -- our token capabilities system allows the server pretty fine-grained control over the permissions it grants each client.[0] But sure, it's never going to be as flexible as a completely custom solution. [0] https://www.ably.io/documentation/general/authentication https://www.ably.io/documentation/general/authentication