7 ms·

Migrating Millions of Concurrent WebSockets to Envoy

188 points by jbredeche 6y ago

AlchemistCamp 6y ago
I remember reading about Discord hitting 5 million concurrent users back in 2017. It would be really interesting to compare the current scale of the two offerings, in terms of peak concurrent users, hardware needed and engineering team size. The user base is very different (business for Slack and gaming for Discord), but has started overlapping more and more in recent years.
ColeByrne 6y ago
This is big news. Simultaneous migration of web sockets will require a lot of power and strong control over the process. I see that you did it, how did you do it? Share your experience, thank you in advance. Specialists are always appreciated (https://www.ramotion.com/ https://www.ramotion.com/)
jeffbee 6y ago
When you control both client and server it seems like hot restart is just a complicated stunt you don’t need. Isn’t it fine to just stop accepting connections, tell all your clients to reconnect, and do a normal restart? The frontend load balancer that stands between you and Gmail doesn’t know how to restart hot but you probably never noticed.
- jayd16 6y ago
  The Gmail load balancer has to do a cold restart to add or remove an instance? That's the requirement they placed on themselves because they do not trust the runtime HAProxy api.
  - theflyinghorse 6y ago
    If they do not trust HAProxy runtime API then why are they using HAProxy at all?
    - deleted 6y ago
      [deleted]
    - vad_ 6y ago
      Haproxy itself is a solid piece of software. The runtime API is something they added on top of it because of competitors (envoy).
  - jfrunyon 6y ago
    Are you saying that Slack don't trust it? Or Google? If the former, it's actually because they don't trust their code that used the HAProxy API. https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ https://slack.engineering/a-terrible-horrible-no-good-very-b... > Consul-template renders a new version of the host list, and a separate program developed at Slack, haproxy-server-state-management, reads that host list and uses the HAProxy Runtime API to update the HAProxy state. ... However, over the course of the day, a problem developed. The program which synced the host list generated by consul template with the HAProxy server state had a bug.
- hermanradtke 6y ago
  > stop accepting connections, tell all your clients to reconnect This "drain" pattern is great for maintenance, upgrades, etc too. The only caveat is that the clients need to be given time to migrate. How long that is depends on how well the clients behave. A hot restart may be much faster.
- jsiepkes 6y ago
  > Isn’t it fine to just stop accepting connections, tell all your clients to reconnect, and do a normal restart? Dependents on how many config changes you need (per day). Besides Envoy supports it and I would call it a bonus if you can reload your configuration without client interruption. As for complicating things the implementation for hot reload isn't terribly complicated in Envoy.
  - jeffbee 6y ago
    I'm mentally separating the hot restart part from the reloadable configs part, even though they are together in the article. To me, not having reloadable configs is too crazy to even imagine.
- mbyio 6y ago
  I think Slack is different than Gmail because people are actively having conversations, so if you disconnect, it is much more likely to be noticeable and annoying. Reading between the lines, I think what they would need is a way to tell clients to move to a new websocket connection at the proxy layer. I don't think there is an easy built-in way to do this in the websocket protocol, so they would have to implement something custom in their application layer. This would also require triggering custom code in the client to make a new websocket connection, start using it, and then close the old connection. I feel like it would have been simpler to just have the client do a graceful reconnect every 5 minutes. But they probably decided to use envoy so they could have the other advantages too.
- zemo 6y ago
  maybe it could work, but in practice it's often not as easy as you'd like it to be. disconnecting everyone at or around the same time simultaneously can easily create a thundering herd or a TCP global synchronization problem, so "just ask everyone to reconnect" has its own set of complications.
- toast0 6y ago
  Depending on where the load balancer is in the protocol stack, it might be hard to signal the client to please reconnect. I've tended to run/use load balancers in tcp mode or direct server return mode; in both cases, the load balancer can't signal the client or server (except by killing connections, which isn't ideal). If you do restart with reconnect, many clients will need to reconnect multiple times, as it they get kicked off early, they'll tend to connect to another old server, and have to get kicked off again later. This isn't great because connection startup is usually expensive; also a (hopefully) small minority of clients may have difficculty connecting because of network issues and it may take them a substantial time to get reconnected. If you have a working hotload path, you get clients to new server server code faster using less resources. Load balancing software is also often behind some other layer of load balancing/availability (DNS or ECMP or BGP or CARP), and a hot load means you don't need to churn that layer, which can avoid any issues with changes in that layer.
- paxys 6y ago
  It is normal to have production deploys continuously throughout the day. If your client disconnected and reconnected every time a Slack engineer decided to push some code, it would get annoying pretty quick.
  - jeffbee 6y ago
    I don't see why even an endless series of connection interruptions would ever need to come to my attention.
  - deleted 6y ago
    [deleted]
endisneigh 6y ago
I wonder if Slack has considered using webrtc to do peer to peer chats on the client side and then gathering up the chat metadata and having each client periodically send their version of the history and reconciling it server side. This would also have the effect of allowing slack to peer more or less normally even if Slack was down (of course bots, search, etc wouldn’t work).
- ryanianian 6y ago
  I suspect there may be regulatory restrictions about allowing text-based communications that aren't available during an audit.
  - ssss11 6y ago
    What regulation do you think would apply? And how/why would this regulation differ for e2e encrypted chat products like Signal, Telegram, WhatsApp etc that can’t access text based chat messages?
    - detaro 6y ago
      Companies that have such audit requirements do not use Signal et al either for internal comms.
    - zonotope 6y ago
      IANAL, but the enterprise companies that make up Slack's customer base are often under regulations to preserve their employees' official communications in case they are needed for future investigations. Those same regulations prevent them from using the products you listed as official communication channels.
    - deleted 6y ago
      [deleted]
    - toomuchtodo 6y ago
      FINRA recordkeeping and retention requirements, as well as SEC statute around records and reporting requirements (finance industry specific).
ex3ndr 6y ago
I am curious about backend part - do ws is still ws on services? Why? For example, why to have thousands of connections instead of a single one (or a bunch) that simply forwards websocket packets with some "connection id" with them. This way you could restart service without killing ws connection, move all overhead of handling millions of connections to the lb.
- Matthias247 6y ago
  You could do that. But it would introduce a lot of head of line blocking on that connection. In return you Save a TLS handshake which might not be too important for long lived connections. Plus it requires the backend to support that custom multiplexing layer.
  - muxator 6y ago
    How so? Why there should be a head of line blocking? This would be a simple multiplexing of the payloads.
    - Matthias247 6y ago
      If there on the other connection a lot of messages are exchanged (and a websocket frame could be gigabytes long according to the spec) the other stream would be blocked behind that.Of course that can be mitigated by using a non websocket framing, but that in the end reinvents yet another transport layer.
- bpicolo 6y ago
  If I remember right, Slack's main application was PHP, now Hack, based, so I wouldn't be surprised to hear that app doesn't hold onto websocket connections itself - long-lived connections aren't historically PHP's wheelhouse
  - 1f60c 6y ago
    I didn't know anyone outside of Facebook used Hack! TIL.
forgotmypw17 6y ago
Is there a plan to migrate to an open protocol to allow accessible clients?
kureikain 6y ago
This is great but why they have to do that? Slack uptime isnt great. They might be up globally but they definerely degrade performance all the time. I got issue like fail to send message very often. Plus, it is just chat. We re really ok with it being down
- Aeolun 6y ago
  You may be ok with your chat being broken at random times, but I think it’s fairly annoying.
Aeolun 6y ago
I’m really hppy to see that there’s a whole new generation of services being released that finally feels like they cover all the modern use cases. I had that feeling with nginx vs Apache too of course, but this is another level of that same thing.
- ofrzeta 6y ago
  I share that sentiment but also want to point out that HAProxy has gained a lot of support for "modern use cases" over the years, for instance the "Data Plane API" that lets you configure HAProxy dynamically.
yawaramin 6y ago
They handled the complex config in an interesting way. I'm taking a look at Dhall for a real use case at work recently, and thinking it might be able to handle a lot of complex config scenarios. I guess that remains to be proven out.
- dilyevsky 6y ago
  If you’re looking at other solutions check out https://github.com/stripe/skycfg https://github.com/stripe/skycfg It works with Envoy and lots of other things that support protobuf configs
  - yawaramin 6y ago
    For the time being I'm targeting YAML and JSON, so satisfied with the feature set that Dhall is offering.
    - dilyevsky 6y ago
      Easy to render output as either and bonus is if you know Python, Starlark is very similar
      - yawaramin 6y ago
        I checked it out, but it's not obvious to me how to target YAML or JSON using it. In contrast, the first use case the Dhall homepage is how to convert to YAML. There are other things Dhall makes obvious, like how to create a record type to target specific YAML/JSON constructs, with type-checking. Also I prefer Dhall's evaluation model (all values are at collapsed at compile time into a single output value with no side effects) to Starlark's full-fledged 'runnable' model, e.g. with the 'print' function I can print out anything I like, including possibly secrets and passwords. This is impossible with Dhall. That said, I do wish that Dhall had something closer to Starlark's syntax and better type inference :-)

188 points by jbredeche 6y ago

I remember reading about Discord hitting 5 million concurrent users back in 2017. It would be really interesting to compare the current scale of the two offerings, in terms of peak concurrent users, hardware needed and engineering team size. The user base is very different (business for Slack and gaming for Discord), but has started overlapping more and more in recent years.

ColeByrne 6y ago

This is big news. Simultaneous migration of web sockets will require a lot of power and strong control over the process. I see that you did it, how did you do it? Share your experience, thank you in advance. Specialists are always appreciated (https://www.ramotion.com/ https://www.ramotion.com/)

jeffbee 6y ago

When you control both client and server it seems like hot restart is just a complicated stunt you don’t need. Isn’t it fine to just stop accepting connections, tell all your clients to reconnect, and do a normal restart? The frontend load balancer that stands between you and Gmail doesn’t know how to restart hot but you probably never noticed.

jayd16 6y ago
The Gmail load balancer has to do a cold restart to add or remove an instance? That's the requirement they placed on themselves because they do not trust the runtime HAProxy api.
- theflyinghorse 6y ago
  If they do not trust HAProxy runtime API then why are they using HAProxy at all?
  - deleted 6y ago
    [deleted]
  - vad_ 6y ago
    Haproxy itself is a solid piece of software. The runtime API is something they added on top of it because of competitors (envoy).
- jfrunyon 6y ago
  Are you saying that Slack don't trust it? Or Google? If the former, it's actually because they don't trust their code that used the HAProxy API. https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ https://slack.engineering/a-terrible-horrible-no-good-very-b... > Consul-template renders a new version of the host list, and a separate program developed at Slack, haproxy-server-state-management, reads that host list and uses the HAProxy Runtime API to update the HAProxy state. ... However, over the course of the day, a problem developed. The program which synced the host list generated by consul template with the HAProxy server state had a bug.
hermanradtke 6y ago
> stop accepting connections, tell all your clients to reconnect This "drain" pattern is great for maintenance, upgrades, etc too. The only caveat is that the clients need to be given time to migrate. How long that is depends on how well the clients behave. A hot restart may be much faster.
jsiepkes 6y ago
> Isn’t it fine to just stop accepting connections, tell all your clients to reconnect, and do a normal restart? Dependents on how many config changes you need (per day). Besides Envoy supports it and I would call it a bonus if you can reload your configuration without client interruption. As for complicating things the implementation for hot reload isn't terribly complicated in Envoy.
- jeffbee 6y ago
  I'm mentally separating the hot restart part from the reloadable configs part, even though they are together in the article. To me, not having reloadable configs is too crazy to even imagine.
mbyio 6y ago
I think Slack is different than Gmail because people are actively having conversations, so if you disconnect, it is much more likely to be noticeable and annoying. Reading between the lines, I think what they would need is a way to tell clients to move to a new websocket connection at the proxy layer. I don't think there is an easy built-in way to do this in the websocket protocol, so they would have to implement something custom in their application layer. This would also require triggering custom code in the client to make a new websocket connection, start using it, and then close the old connection. I feel like it would have been simpler to just have the client do a graceful reconnect every 5 minutes. But they probably decided to use envoy so they could have the other advantages too.
zemo 6y ago
maybe it could work, but in practice it's often not as easy as you'd like it to be. disconnecting everyone at or around the same time simultaneously can easily create a thundering herd or a TCP global synchronization problem, so "just ask everyone to reconnect" has its own set of complications.
toast0 6y ago
Depending on where the load balancer is in the protocol stack, it might be hard to signal the client to please reconnect. I've tended to run/use load balancers in tcp mode or direct server return mode; in both cases, the load balancer can't signal the client or server (except by killing connections, which isn't ideal). If you do restart with reconnect, many clients will need to reconnect multiple times, as it they get kicked off early, they'll tend to connect to another old server, and have to get kicked off again later. This isn't great because connection startup is usually expensive; also a (hopefully) small minority of clients may have difficculty connecting because of network issues and it may take them a substantial time to get reconnected. If you have a working hotload path, you get clients to new server server code faster using less resources. Load balancing software is also often behind some other layer of load balancing/availability (DNS or ECMP or BGP or CARP), and a hot load means you don't need to churn that layer, which can avoid any issues with changes in that layer.
paxys 6y ago
It is normal to have production deploys continuously throughout the day. If your client disconnected and reconnected every time a Slack engineer decided to push some code, it would get annoying pretty quick.
- jeffbee 6y ago
  I don't see why even an endless series of connection interruptions would ever need to come to my attention.
- deleted 6y ago
  [deleted]

endisneigh 6y ago

I wonder if Slack has considered using webrtc to do peer to peer chats on the client side and then gathering up the chat metadata and having each client periodically send their version of the history and reconciling it server side. This would also have the effect of allowing slack to peer more or less normally even if Slack was down (of course bots, search, etc wouldn’t work).

ryanianian 6y ago
I suspect there may be regulatory restrictions about allowing text-based communications that aren't available during an audit.
- ssss11 6y ago
  What regulation do you think would apply? And how/why would this regulation differ for e2e encrypted chat products like Signal, Telegram, WhatsApp etc that can’t access text based chat messages?
  - detaro 6y ago
    Companies that have such audit requirements do not use Signal et al either for internal comms.
  - zonotope 6y ago
    IANAL, but the enterprise companies that make up Slack's customer base are often under regulations to preserve their employees' official communications in case they are needed for future investigations. Those same regulations prevent them from using the products you listed as official communication channels.
  - deleted 6y ago
    [deleted]
  - toomuchtodo 6y ago
    FINRA recordkeeping and retention requirements, as well as SEC statute around records and reporting requirements (finance industry specific).

ex3ndr 6y ago

I am curious about backend part - do ws is still ws on services? Why? For example, why to have thousands of connections instead of a single one (or a bunch) that simply forwards websocket packets with some "connection id" with them. This way you could restart service without killing ws connection, move all overhead of handling millions of connections to the lb.

Matthias247 6y ago
You could do that. But it would introduce a lot of head of line blocking on that connection. In return you Save a TLS handshake which might not be too important for long lived connections. Plus it requires the backend to support that custom multiplexing layer.
- muxator 6y ago
  How so? Why there should be a head of line blocking? This would be a simple multiplexing of the payloads.
  - Matthias247 6y ago
    If there on the other connection a lot of messages are exchanged (and a websocket frame could be gigabytes long according to the spec) the other stream would be blocked behind that.Of course that can be mitigated by using a non websocket framing, but that in the end reinvents yet another transport layer.
bpicolo 6y ago
If I remember right, Slack's main application was PHP, now Hack, based, so I wouldn't be surprised to hear that app doesn't hold onto websocket connections itself - long-lived connections aren't historically PHP's wheelhouse
- 1f60c 6y ago
  I didn't know anyone outside of Facebook used Hack! TIL.

forgotmypw17 6y ago

Is there a plan to migrate to an open protocol to allow accessible clients?

kureikain 6y ago

This is great but why they have to do that? Slack uptime isnt great. They might be up globally but they definerely degrade performance all the time. I got issue like fail to send message very often. Plus, it is just chat. We re really ok with it being down

Aeolun 6y ago
You may be ok with your chat being broken at random times, but I think it’s fairly annoying.

Aeolun 6y ago

I’m really hppy to see that there’s a whole new generation of services being released that finally feels like they cover all the modern use cases. I had that feeling with nginx vs Apache too of course, but this is another level of that same thing.

ofrzeta 6y ago
I share that sentiment but also want to point out that HAProxy has gained a lot of support for "modern use cases" over the years, for instance the "Data Plane API" that lets you configure HAProxy dynamically.

yawaramin 6y ago

They handled the complex config in an interesting way. I'm taking a look at Dhall for a real use case at work recently, and thinking it might be able to handle a lot of complex config scenarios. I guess that remains to be proven out.

dilyevsky 6y ago
If you’re looking at other solutions check out https://github.com/stripe/skycfg https://github.com/stripe/skycfg It works with Envoy and lots of other things that support protobuf configs
- yawaramin 6y ago
  For the time being I'm targeting YAML and JSON, so satisfied with the feature set that Dhall is offering.
  - dilyevsky 6y ago
    Easy to render output as either and bonus is if you know Python, Starlark is very similar
    - yawaramin 6y ago
      I checked it out, but it's not obvious to me how to target YAML or JSON using it. In contrast, the first use case the Dhall homepage is how to convert to YAML. There are other things Dhall makes obvious, like how to create a record type to target specific YAML/JSON constructs, with type-checking. Also I prefer Dhall's evaluation model (all values are at collapsed at compile time into a single output value with no side effects) to Starlark's full-fledged 'runnable' model, e.g. with the 'print' function I can print out anything I like, including possibly secrets and passwords. This is impossible with Dhall. That said, I do wish that Dhall had something closer to Starlark's syntax and better type inference :-)