8 ms·
Discord Reduced WebSocket Traffic by 40%
- paxys 2y agoReading through the post they seem to have been hyper focused on compression ratios and reducing the payload size/network bandwidth as much as possible, but I don't see a single mention of CPU time or evidence of any actual measureable improvement for the end user. I have been involved with a few such efforts at my own company, and the conclusion always was that the added compression/decompression overhead on both sides resulted in worse performance. Especially considering we are talking about packets at the scale of bytes or a few kilobytes at most.
- xnx 2y agoDo the packets transmit through Discord's servers? Reducing their bill may be more important to them than user performance.
- bri3d 2y agoThey're going from 2+MB (for some reason) to 300KB - even if decompression is "slow," that's going to be a win for their bandwidth costs and for perceived speed for _most_ users. I was surprised to see little server-side CPU benchmarking too, though. While I'd expect overall client timing for (transfer + decompress) to be improved dramatically unless the user was on a ridiculously fast network connection, I can't imagine server load not being affected in a meaningful way.
- usernamear 2y agoThere already was compression before, through zlib. The findings, as showed in the post, was that Zstandard was also a lot more efficient than zlib from a cpu time standpoint.
- paxys 2y agoBandwidth costs for text messages, maybe, but how much data is that really compared to images, audio, video or even just the app's JS bundle?
- bri3d 2y agoPresumably that's all CDNed and therefore a lot cheaper to serve.
- bguebert 2y agoProbably not for live shared audio/video
- toast0 2y agoThe bandwidth probably doesn't really matter, but a 2MB must have blob vs a 300kB must have blob at the start of a connection is a big difference. The start of a tcp connection is limited by round trip times more than bandwidth. Especially for mobile, optimizing to reduce the number of round trips required is pretty handy.
- jhgg 2y agoThe 2mb case is pathological - an account on MANY servers with no local cache state (the READY payload works to only send data that's changed between when you've reconnected by having the client send hashes of data it knows.)
- BoorishBears 2y agoThey might have been getting murdered by egress fees in which case they'd be willing to make that sacrifice
- usernamear 2y agoSome of those payloads are much larger than a few kilobytes (READY, MESSAGE_CREATE etc.) There is a section and data on "time to compress". No time to decompress though.
- hiddencost 2y agoIt's almost certainly about hosting costs, not user facing value.
- ihumanable 2y ago> zstandard streaming significantly outperforms zlib both in time to compress and compression ratio. Time to compress is a measure of how long the CPU spends compressing. So this is in the blogpost
- koito17 2y agoI think the person is concerned with client-side compute, not just server-side compute. The article does not mention whether zstd has additional decompression overhead compared to zlib. Client-side compute may sound like a contrived issue, but Discord runs on a wide variety of devices. Many of these devices are not necessarily the latest flagship smartphones, or a computer with a recent CPU. I am going to guess that zstd decompression is roughly as expensive as zlib, since (de)compression time was a motivating factor in the development of zstd. Also the reason to prefer zstd over xz, despite the latter providing better compression efficiency.
- colechristensen 2y agozstd has faster decompression though I always thought lz4 to be the sweet spot for anything requiring speed, somewhat less compression ratio in exchange for very fast compression and decompression
- TulliusCicero 2y ago> I don't see a single mention of CPU time > Looking once again at MESSAGE_CREATE, the compression time per byte of data is significantly lower for zstandard streaming than zlib, with zlib taking around 100 microseconds per byte and zstandard taking 45 microseconds.
- lilyball 2y agoThat's compression time, the parent is talking about the end user so we want decompression time instead.
- monocasa 2y agoZstd is generally known for markedly faster decompression (as well as compression) than zlib. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/ https://gregoryszorc.com/blog/2017/03/07/better-compression-...
- dgfitz 2y agoI imagine it correlates with the end-user device specs, no?
- jhgg 2y agoI think one thing this blog post did not mention was the historical context of moving from uncompressed to compressed traffic (using zlib), something I worked on in 2017. IIRC, the bandwidth savings were massive (75%). It did use more server side CPU, and negligible client side CPU, so we went for it anyways as bandwidth is a very precious thing to optimize for especially with cloud bandwidth costs. Either way the incremental improvements here are great - and it's important to consider optimization both from transport level (compression, encoding) and also from a protocol level (the messages actually sent over the wire.) Also one thing not mentioned is client side decompression on desktop used to use a JS implementation of zlib (pako) to a native implementation, that's exposed to the client via napi bindings.
- Moru 2y agoCan't remember last time I had to worry about bandwidth for the servers. It only came up when talking about iPhones in the start because everyone was on realy slow mobile networks. Our company is usually very cost sensitive but all our server hotels so far has had unmetered bandwidth connected to a 100 Mbps interface. Has had zero complaints during the last 20 years even though we fill that one now and then. But we don't use any of the usual cloud offerings, only smaller local companies.
- dumbo-octopus 2y agoThe explicitly mention compression time. It’s actually lower in the new approach. > the compression time per byte of data is significantly lower for zstandard streaming than zlib, with zlib taking around 100 microseconds per byte and zstandard taking 45 microseconds
- yuliyp 2y agoI am fairly sure those should have been per kilobyte, not per byte.
- dvh 2y agoThose are atrocious numbers, that's only 23kB/s for the faster variant. It should have been GB/s not kB.
- refulgentis 2y agoFirst, let's establish a cheery mood: Happy Friday!!!! Second, I noticed we're extrapolating from a tossed out measurement in "microseconds per byte" here, of extremely small payloads, probably included fixed-cost overhead of doing anything at all. All leading up to: Is "atrocious" the right word choice here? :) More directly: do you really think Discord rolled out a compression algorithm that does 23 KB/s for payloads in the megabytes? Even more directly, avoiding being passive and just adopting your tone: this is atrocious analysis that glibly chooses to create obviously wrong numbers, then criticizes them as if they are real.
- Starlevel004 2y ago> More directly: do you really think Discord rolled out a compression algorithm that does 23 KB/s for payloads in the megabytes? yes, actually
- refulgentis 2y agothat's "yngmi" bait; you're suggesting it takes 2 minutes per intro payload. (2 MB / 20 KB/s ≈ 100 seconds = 1m40s)
- zarzavat 2y agoPerformance is probably the wrong lens. Mobile data is often expensive in terms of money, whereas compression is cheap in terms of CPU time. More compression is almost always the right answer for users of mobile apps.
- bri3d 2y agoInteresting way to approach this (dictionary based compression over JSON and Erlang ETF) vs. moving to a schema-based system like Cap'n Proto or Protobufs where the repeated keys and enumeration values would be encoded in the schema explicitly. Also would be interested in benchmarks between Zstandard vs. LZ4 for this use case - for a very different use case (streaming overlay/HUD data for drones), I ended up using LZ4 with dictionaries produced by the Zstd dictionary tool. LZ4 produced similar compression at substantially higher speed, at least on the old ARM-with-NEON processor I was targeting. I guess it's not totally wild but it's a bit surprising that common bootstrapping responses (READY) were 2+MB, as well.
- echelon 2y agoThey use JSON over the wire and not a binary protocol? That's madness and reminds me of XML / Jabber. Protos or a custom wire protocol would be far better suited to the task.
- treyd 2y agoI don't understand why so many protocols that expect to handle large amounts of data don't default to a binary schema. JSON is fine on the edges, but the wire format between nodes is not the edge.
- nojvek 2y agoI assume it’s mostly because it’s way easier to debug json over websockets and http with browser devtools instead of custom protocols. Custom protocol would be binary. They could make a custom extension but it wouldn’t be that easy. I worked on browser devtools for IE and edge. Even chrome/vscode use jsonrpc over websockets for ease of development.
- Spivak 2y agoHard disagree given the constraints. Every bot is also consuming the Discord API and forcing 3rd-party devs, many whom aren't particularly advanced coders to suddenly deal a binary wire format would be painful especially if you needed to constantly update a proto file. Their API is also part WebSocket part HTTP and many methods doing double-duty.
- mevv 2y ago[dead]
- acer4666 2y agoAnytime I have a discord tab open it noticeably grinds my computer to a halt
- sfn42 2y agoSounds like a problem with your computer. I can have discord, 50 browser tabs, two different games, a JetBrains IDE and various other stuff open at the same time without any trouble at all. And my computer isn't particularly crazy. Maybe like $1500.
- toastercat 2y ago"Works on my machine"
- sfn42 2y agoLol, in fact it works on all my machines. And lots of other peoples machines.
- FactKnower69 2y agoworthless post
- PhilipRoman 2y agoSame here, but the computer cost $200 But on a more serious note, I still hate bloated software. Money can't buy latency and the sluggishness gets really annoying.
- acer4666 2y agoMy computer isn't great, I'll admit, but I'm making a relative comparison. I visit many different websites on my low spec computer but discord is a noticeable outlier on how it affects the performance.
- creesch 2y agoHow many servers have you joined and how many of those are large and active? Also relevant, do you need to be in all of them? Most of the time I have seen people complain about this it is because they have joined a ton of hyperactive servers. You could argue it shouldn't be an issue and more dynamically load things like messages on servers. But then you'd have people complaining that switching servers takes so long.
- transcriptase 2y agoWhen can we expect discord not to take 20-30 seconds to launch on a $5000 PC? What exactly is it doing, recompiling the client from source using a single core each time it opens?
- dumbo-octopus 2y agoIt needs to download the 20 distinct update patches that they added in the past 4 hours, in series, all of which combine to together change the actual product in precisely no way whatsoever.
- HeXetic 2y ago> change the actual product in precisely no way whatsoever How dare you belittle the new Super Ultra Nitro Deluxe Gold Platinum emoji, stickers, and playable sound effects.
- fearthetelomere 2y ago>Diving into the actual contents of one of those PASSIVE_UPDATE_V1 dispatches, we would send all of the channels, members, or members in voice, even if only a single element changed. > the metrics that guided us during the [zstd experiment] revealed a surprising behavior This feels so backwards. I'm glad that they addressed this low-hanging fruit, but I wonder why they didn't do this metrics analysis from the start, instead of during the zstd experiment. I also wonder why they didn't just send deltas from the get-go. If PASSIVE_UPDATE_V1 was initially implemented "as a means to scale Discord servers to hundreds of thousands of users", why was this obvious optimization missed?
- jhgg 2y agoIt was a bug
- fearthetelomere 2y agoThanks, explains a lot. Wish the article did a better job explaining that instead of framing it as a version upgrade to V2.
- nickphx 2y agomIRC did it better.
- mrinfinitiesx 2y agoTruth.
- jimmyl02 2y agoOne thing I appreciate very much about this article is that they describe things they tried and didn't work as well. It's becoming increasingly rare (and understandably why) for articles to describe failed attempts but it's very interesting and helpful as someone unfamiliar with the space!
- RainyDayTmrw 2y agoSomething important that didn't get mentioned, neither in the post nor in the comments, is whether this is safe in the face of compression oracle attacks[1] like BREACH[2]. Given how much effort it seems Discord put into the compression rollout, I would be inclined to believe that they surely must have considered this, and I wish that they had written something more specific. [1]: https://en.wikipedia.org/wiki/Oracle_attack https://en.wikipedia.org/wiki/Oracle_attack [2]: https://en.wikipedia.org/wiki/BREACH https://en.wikipedia.org/wiki/BREACH