4 ms·

Got a question for you - I'm making a chat bot thing (https://www.brian.bot/ https://www.brian.bot/) and have added socket-based web chat... but I don't even k

Got a question for you - I'm making a chat bot thing (https://www.brian.bot/ https://www.brian.bot/) and have added socket-based web chat... but I don't even know how to phrase the question of "how do I horizontally scale the socket connections". Webchat <-> Server <-> Slack. Once I have more than one server dyno (heroku), then I can't ensure that the Slack event hits the dyno with the corresponding socket connection. Thoughts / tips?

mathgladiator 6y ago
I'll be completely honest... you're mostly screwed. I say this with full love in my heart. Your best bet is to use Amazon and then leverage their ELB offering with sticky routing. HOWEVER, now you have a new problem as capacity dies, gets removed, cycled, added, etc. Sticky routing was designed with the assumption that it was an efficiency play as opposed to a deterministic play. What you will see happen is that your internal state will be split brain, and people joining the chat will not be globally consistent. Some people will be on host X while some will be on host Y. So, you have two choices. You could use a database to provide the global state, at which point now you have a whole bunch of problems. You can start with polling the database, and the key advantage of the websocket is moving the polling from client to server, so that will be a win for users. However, it will cost you. Second, you could introduce a message broker, but then you have the problem of how to initialize the state. Ultimately, everyone ends up with a hybrid database for storage and message broker for real-time. In your model, the database becomes like Slack, and your Server becomes a proxy. This could be super neat, but what is the value-add of the Server? The interesting thing to observe is that the solution is how to turn your front-end app into a database like thing. If the load balancer could route traffic like a database traffic controller thing, then you can put all your state in the front-end and move crazy fast. However, you will move so fast that you will fuck up your state. State is VERY HARD and exceptionally unforgiving to manage yourself. There is great wisdom in using a database with versioning and all that. It’s a great abstraction having lasted so long. I’m working on the state bits and trying to figure out the people side of discipline with my silly language, and I have some novel contributions to provide over the coming years. There are some linked pdfs on http://www.adama-lang.org/blog/some-thinky-thoughts-2021 http://www.adama-lang.org/blog/some-thinky-thoughts-2021 which may provide some more insight.
- naugtur 6y ago
  Just a random thought - wonder if anyone tried a p2p protocol instead of a database or message broker for spreading information. Wonder if it'd be any better than broadcast to every stateful instance.
  - zomglings 6y ago
    Bittorrent, blockchains.
- RangerScience 6y ago
  That's super helpfully, actually - tells me about some problems that I wouldn't have anticipated, and tells me NOT to go looking for certain other solutions. Short term, I think I can get by with switching from Slack's events API to their real-time widget - aka, the server gets a socket to both the web chat, and the slack. Long term - probably the Redis pub/sub with the socket-receiving server split out as a microservice (which normally I'd be "eh" about, but this seems perfect). I currently don't have to worry about multiple people on the webchat (outside of needed features) and even once that's an important thing, I doubt I'll need to worry about exactness. As long as each message get delivered reasonably quickly, and to everyone, should always be good. hmm, hmm hmm - great stuff, thank you!
Deadron 6y ago
You can use a redis to distribute messages via PUB/SUB. The sockets subscribe to the events that are relevant to them. It can handle thousands of messages a second in a local environment and probably more in a dedicated hosting environment.
- stevewodil 6y ago
  Seconding this approach. I've used PUB/SUB for this exact purpose. When you need to send something through the websocket then publish a message via redis and have handlers in the code that are subscribed to that channel so every server can check if they have the websocket connection and whichever one has it will send the data through
  - RangerScience 6y ago
    That mostly sounds great, although TBH my first instinct would be to stick all the socket stuff in a microservice. I don't normally advocate for them, but this seems like a perfect situation: very well defined layer that contains no business logic.
abrookewood 6y ago
Pretty sure that Erlang or Elixir would work there - you can have clusters of stateful servers.

mathgladiator 6y ago

I'll be completely honest... you're mostly screwed. I say this with full love in my heart. Your best bet is to use Amazon and then leverage their ELB offering with sticky routing. HOWEVER, now you have a new problem as capacity dies, gets removed, cycled, added, etc. Sticky routing was designed with the assumption that it was an efficiency play as opposed to a deterministic play. What you will see happen is that your internal state will be split brain, and people joining the chat will not be globally consistent. Some people will be on host X while some will be on host Y. So, you have two choices. You could use a database to provide the global state, at which point now you have a whole bunch of problems. You can start with polling the database, and the key advantage of the websocket is moving the polling from client to server, so that will be a win for users. However, it will cost you. Second, you could introduce a message broker, but then you have the problem of how to initialize the state. Ultimately, everyone ends up with a hybrid database for storage and message broker for real-time. In your model, the database becomes like Slack, and your Server becomes a proxy. This could be super neat, but what is the value-add of the Server? The interesting thing to observe is that the solution is how to turn your front-end app into a database like thing. If the load balancer could route traffic like a database traffic controller thing, then you can put all your state in the front-end and move crazy fast. However, you will move so fast that you will fuck up your state. State is VERY HARD and exceptionally unforgiving to manage yourself. There is great wisdom in using a database with versioning and all that. It’s a great abstraction having lasted so long. I’m working on the state bits and trying to figure out the people side of discipline with my silly language, and I have some novel contributions to provide over the coming years. There are some linked pdfs on http://www.adama-lang.org/blog/some-thinky-thoughts-2021 http://www.adama-lang.org/blog/some-thinky-thoughts-2021 which may provide some more insight.

naugtur 6y ago
Just a random thought - wonder if anyone tried a p2p protocol instead of a database or message broker for spreading information. Wonder if it'd be any better than broadcast to every stateful instance.
- zomglings 6y ago
  Bittorrent, blockchains.
RangerScience 6y ago
That's super helpfully, actually - tells me about some problems that I wouldn't have anticipated, and tells me NOT to go looking for certain other solutions. Short term, I think I can get by with switching from Slack's events API to their real-time widget - aka, the server gets a socket to both the web chat, and the slack. Long term - probably the Redis pub/sub with the socket-receiving server split out as a microservice (which normally I'd be "eh" about, but this seems perfect). I currently don't have to worry about multiple people on the webchat (outside of needed features) and even once that's an important thing, I doubt I'll need to worry about exactness. As long as each message get delivered reasonably quickly, and to everyone, should always be good. hmm, hmm hmm - great stuff, thank you!

Deadron 6y ago

You can use a redis to distribute messages via PUB/SUB. The sockets subscribe to the events that are relevant to them. It can handle thousands of messages a second in a local environment and probably more in a dedicated hosting environment.

stevewodil 6y ago
Seconding this approach. I've used PUB/SUB for this exact purpose. When you need to send something through the websocket then publish a message via redis and have handlers in the code that are subscribed to that channel so every server can check if they have the websocket connection and whichever one has it will send the data through
- RangerScience 6y ago
  That mostly sounds great, although TBH my first instinct would be to stick all the socket stuff in a microservice. I don't normally advocate for them, but this seems like a perfect situation: very well defined layer that contains no business logic.

abrookewood 6y ago

Pretty sure that Erlang or Elixir would work there - you can have clusters of stateful servers.