8 ms·
Discord outage postmortem
- jchw 7y agoIn my opinion, the HTTP server’s Request.Body’s io.Reader should return an error on read when the request body ends unexpectedly, rather than having every use of the request body try to sanity check the length which seems cumbersome and error prone.
- Jasper_ 7y agoI believe it should, but it's likely a userspace proxy like HAProxy is in the way causing errors to be swallowed, as there's no way with the current sockets API to intentionally send an error over the wire.
- jchw 7y agoHi Jasper :) Do you think so? I checked the Go http server code and my not-so-careful reading seems to confirm their suspicions: it just reads from the body io.Reader, which doesn't seem to validate length. (They DO use a limit reader to prevent DoSing with huge bodies, at least.) Now if they were phoning out to a remote server, I’d assume some kind of fabric or load balancer. But I assume they’ve got a local etcd on each node or something along those lines so it is very possible that it’s all local networking. Though in most cases, outside of this use case, I’d agree that this is not particularly useful because it would never be terribly reliable. I know for a fact Amazon ELB rewrites the entire request, as an example...
- Jasper_ 7y agoI was talking about getting an error when reading from io.Reader, like a TCP timeout. I was quite sure the error would bubble up, but it of course wouldn't if there was no error to begin with, with something like HAProxy in the way.
- Rapzid 7y agoI haven't found the code yet but it should throw a line too long error for malformed chunked encoding. The implementer can always compare the read bytes to the content-length themselves. I have a hard time believing without evidence that etcd's implementation is wrong at this stage..
- intsunny 7y ago"All times are PDT." A daily reminder that one can do anything and everything in JavaScript, except display information in UTC.
- dlbucci 7y agoThis is just incorrect. I mean, Date.toISOString() will do that. Plus, it's very likely that this is just what's written in the post by the author and may have nothing to do with JavaScript. Or it could be JS that converts times to local time (I'm in PDT, so I can't tell), which is what a lot of sites do.
- jchw 7y agoDuring the handling of an incident, the last thing you want to think twice about is timezones, so it is understandable for Discord who started in PT to just use PT universally. (Eventually, or perhaps already, it will just become a weird quirk of history.)
- sbr464 7y agoThe reason (it seems) they stick to one timezone, are comments /updates can be added to status page incidents. It can be conflicting to talk about event times in a comment, and the post has different times. Here's an example from slack, where it's mixed, comments in PDT and posts in the local timezone. https://status.slack.com/2019-06-28 https://status.slack.com/2019-06-28. I think it's a setting in statuspage, but it's tough decide which is best. I definitely wouldn't prefer looking at it in UTC.
- coder543 7y ago> Our service discovery system is resilient to various failure modes within etcd, however, we did not anticipate a client being able to write a corrupt record due to improper HTTP handling. golang's net/http module — which is in violation of the HTTP 1.1 specification in this specific case — introduced an additional failure case which we did not anticipate. It is expected that a Content-Length header and an incomplete body should be rejected by a server as being incomplete. Unfortunately net/http does not check that the bytes read from the body portion of the request match the Content-Length header. Whichever engineer wrote puts too much weight on the theoretical aspects of HTTP. The reality is that any of Discord's software could have had erroneous code that announced an empty service list. This would have crashed all of Discord regardless -- transient network errors and mismatched HTTP Content-Length headers need not apply. A resilient HTTP server will try to handle whatever the HTTP client sends it, and then the specific endpoint can enforce whatever levels of strictness are needed by the application. It's nice to be able to point a finger at "not my code", but how many popular web server libraries (not standalone web servers like nginx or apache) out there will actually flat out reject the entire request because of a simple Content-Length mismatch, rather than handing that decision off to the endpoint? If an empty string is not a valid value for your service discovery system, you should reject those from the database with a CHECK CONSTRAINT if possible, or with the code that sits in front of the database if CONSTRAINTS don't exist (like I believe is the case for etcd). It's also not a bad idea for the clients to validate what they receive and filter out values they can't handle to enhance robustness. You definitely shouldn't rely on nothing ever attempting to put a bad value into the database. The web server's willingness to accept "strictly" malformed requests when the engineers were unaware of this fact was certainly a contributing factor, but I don't think it was the root cause in this case, as they're heavily trying to suggest. (I think it would be neat to have a "strict" mode that you could turn on for every web server, especially when being used as an internally-facing service, but most companies want to make it as easy as possible for their customers/partners to integrate with their service... and that means accepting requests that aren't always academically perfect.)
- jchw 7y agoI think it would be nice if the HTTP server could at least optionally validate that the length is expected. For example I would really not expect a malformed chunked encoding lacking a terminating zero length chunk to silently succeed. However, I think you are dead on the money. Etcd is not validating the values but Discord’s software seems to assume the etcd values must be valid. That seems like a recipe for future outages if the mentality were kept this way.
- ddevault 7y agoIt's a good thing that everyone makes their own discord "server", which naturally is resilient to any failures upstream by virtue of being its own fully contained, self-hosted software stack. Wait, no, Discord just aped that word to gaslight users into behaving in a way which makes them more money.
- foota 7y agoDiscord servers are free. I fail to see how this was a move to make more money.
- ReidZB 7y agoThat does not seem like a very generous interpretation. I'd wager that most of Discord's clientele don't have any specific expectations for the word "server". Regardless, I think "server" was carried over from the Mumble / Ventrilo / Teamspeak gaming community.
- Slartie 7y agoThat's where it came from. Before discord, gamers set up actual servers with Teamspeak or Mumble and handed the IPs and ports to their guildmates. The "Teamspeak server" was where you met for voice chatting, and since Discord just wanted to supplant Teamspeak with their own product, they kept the "server" part in the name of the thing that gamers were supposed to meet for voice chatting. It is a pure marketing ploy. Internally they call the "server" thing a "guild" AFAIK, which is technically just as much BS as the "server" moniker, but for other reasons (not every community using Discord is a "guild" in the commonly accepted meaning of that word in the gaming world, which is a long-term organizational group doing stuff together in an MMORPG).
- csears 7y agoAs part of the fix, they mention changing the Python etcd client code to send the request in a single packet. How would one force that behavior? And is that even a good idea? It seems like TCP fragmentation would be handled at a much lower level in the Python networking stack.
- q3k 7y agoNo idea. This 'fix' doesn't make any sense.
- Rapzid 7y agoI'm skeptical and would like to see some etcd/golang github issues referenced on this.
- mst 7y agoIt makes perfect sense as "makes it 95% less likely to hit this bug." Notice from the write up that they already also modified their code to be resilient against the bug being hit at all, so "reducing the number of times the resilience code needs to be invoked by 95%" on top of that is a net win.
- de_watcher 7y agoThis fix is supper common. There are tonnes of software that freak out when you don't send a message in single call hopefully resulting in one packet. People don't understand that TCP alone doesn't really handle application-level messages.
- MarkSweep 7y agoI’m guessing they switched from making two system calls to send data on the socket to a single system call.
- jchw 7y agoThis is my guess - two write calls to one write call. I think this will happen to work even in the presence of Nagle’s algorithm, because there is initially not going to be an already sent packet waiting acknowledgement, so at least the first write call will probably be unbuffered in that sense. Assuming an mtu of 1500, ~40 byte TCP header, and a little bit of HTTP/1.1 headers, that leaves probably plenty of room for the etcd PUT to not get fragmented, I think.
- ayorosmage2 7y agohttps://jepsen.io/analyses/etcd-3.4.3 https://jepsen.io/analyses/etcd-3.4.3
- ravedave5 7y agoFYI for anyone using discord - dont use the web client. Often users with web client can only be heard by certain people and vice versa.
- DingleDook101 7y agoDiscord is hot garbagio.
- kylek 7y agoThe level of detail in this is pretty nice. But I also feel like it's too much- some of the comments here are poking holes in the write-up. Is there a mis-translation from geek-speak to lay-speak (maybe due to the short turnaround)? Is it better to not mention some of these details at all?
- deleted 7y ago[deleted]
- dirtydroog 7y ago> unexpected deserialization error I've had this with Boost.Serialization. Not fun. No exception, just a crash.
- chasers 7y agoIt sounds like they were maybe pattern matching to strictly. If the value is always a string just save the empty string and try to discover that service. If it doesn't exist, oh well, it doesn't crash the process.
- k__ 7y agoWe used Discord for the "We vs Virus" hackathon last weekend and I was baffled how much better than Slack it is. Everything runs much smoother, voice chat is like with TeamSpeak back in the days. Solid product.