15 ms·

Discord Postmortem from Friday

96 points by b1naryth1ef 9y ago

lwansbrough 9y ago
Funny definition of HA. :)
atomical 9y ago
Does anyone use discord for work?
- katastic 9y ago
  I have informally with co-workers. But not in any official capacity. It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used. My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the frame rate up and looked presentable at least to 720p.
  - avree 9y ago
    Weird, I find Discord's audio quality especially to be terrible. And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.
    - b1naryth1ef 9y ago
      Can you give more explicit examples of the bad audio quality you experience? I'd be happy to forward this onto our native team to look into if there are some concrete things they can look at. Generally 99% of the audio issues we see people experience are due to ISP/peering/DDoS/etc issues, most of which are handled automatically by our servers within a few minutes.
      - mooman219 9y ago
        Anecdotal, but direct calls are pretty unusable for me. I'm on the west coast, and when I attempt calls to the east cost, it frequently cuts out. The workaround was just creating a server and using a voice room in it. This had drastically higher call quality. I assume this is ISP/Peering related, but to see such a night and day difference between voice channels and direct calls leads me to believe that there's something that can be done on your end. Complaints aside, I love the service. Echoing what avree said, long term monetization worries me. I would like to see discord survive, but its story looks similar to Trello right now in terms of monetization.
        b1naryth1ef 9y ago
        Ahhhh, thats actually something we're aware of. Currently direct calls run on an entirely separate set of metals vs. everything else (this was mostly to help us test/measure video & screenshare rollout). Unfortunately some providers seem to be having issues with DDoS filtering over-triggering when they see video traffic, which negatively impacts the whole server. Something we're hopefully fixing in the short term!
      - asddddd 9y ago
        I've been using voice calls regularly, and it's occasionally been problematic. Yesterday it was behaving as though it had absolutely insane packet loss (occasional robotic sounding fragments would make it through, otherwise the line was dead) without any indication of high jitter or packet loss - I had to resort to stupid workarounds to get it working. Move convo to a server -> change server to Central from West (why can't this be done for regular calls!) -> instantly working perfectly. FWIW, it generally works pretty well, and overall, Discord is a fantastic product that I'm happy to be using.
      - eropple 9y ago
        FWIW, a serious feature that I would pay you money for right-here-right-now is the ability to multitrack audio. Let me give you many dollars to route each voice to a separate Soundflower channel, so that I can mix them outside of Discord, and I will give you said monies. I'm pretty sure you're even sending unmixed streams down? But I can't get at them! This probably involves not being Electron (from my own adventures in the area), so I don't hold out much hope, but it keeps Discord from replacing Extremely Expensive And Bad solutions like SkypeTX for me.
    - gizmo385 9y ago
      One thing that I've noticed helps is swapping the region for the server. It doesn't result in any noticeable outage on your server and has resolved any voice quality issues that we've run into most of the time. Usually just swapping between US-Central and US-West in our case.
- sbarre 9y ago
  We've debated it internally, but currently have decided to stick with Slack (free version)
- andyfleming 9y ago
  I briefly thought about whether it would be a better option than Slack. However, Slack seems a little more polished in the areas that we actually use. The subtle, better design in places like the core chat experience aren't worth giving up. I have used Discord quite a bit for gaming, but it hasn't proved a better option than Slack to me (at least in a work context).
- zanny 9y ago
  I would strongly recommend against it. Discord has no privacy guarantees in their TOS and actively says they will sell your data. Its how they offer their service "for free". Workplace conversations should be private, and that almost exclusively means either a siloed self hosted solution or a service with a contract that guarantees data integrity and privacy. If you are playing video games using Discord is fine, at least up until you start talking about personal details of your life in their chat rooms or over their voice services. If I'm having conversations I don't want Discord's data customers having, I'll use Signal / Matrix (self hosted or encrypted channels) / Mumble.
  - eslachance 9y ago
    That's nice misinformation, mate. Discord makes money from Nitro, and they started off with a fairly large investor cushion which grew since launch. There is nothing in the ToS that states in any way, shape, or form, that they can sell their data. I challenge you to quote the exact place where it says that.
    - lightedman 9y ago
      You don't look in the ToS. You look in the company articles of incorporation or bylaws which will explain things like how money is allowed to be made.
      - exikyut 9y ago
        How would that work? If a company drafts ToS and then plumb violates them, well, it'll be found out eventually and then you get things like the recent debacle with telco data being sold (https://news.ycombinator.com/item?id=15477286 https://news.ycombinator.com/item?id=15477286), and then the company(s) in question will either get completely shredded or their reputation will at least be seriously besmirched. I can see the logic of what you're saying, but at the same time I can't see how it would hold up. If you can show me how it would work I'm listening though.
        9y ago
- synicalx 9y ago
  Unofficially, for "out of band" discussions and also for on-call events where we need to collab on stuff. We're stuck with Skype for official stuff and telephony though.
- jakebasile 9y ago
  We did, but due to some people having image issues about using a “gamer tool”, we were forced to switch back to the inferior Slack.
- eslachance 9y ago
  I'll need to echo some of the other comments here - at work we're pretty much stuck with Skype for Business (aka "lync") because the voice system ties into it. When I suggested Discord, it was shrugged off because it's "Chat for Gamers" and clearly they're not going to budge from that niche anytime soon, from all the recent features I've seen.
ZeroCool2u 9y ago
I'm really impressed. I was using Discord for most of this weekend, specifically Friday and Saturday. Never noticed any issues.
- gizmo385 9y ago
  They don't have a posted outage for Saturday, but I noticed issues with it on Saturday evening/night I believe. I'm wondering if it was related to the issues that they included in the post-mortem.
  - b1naryth1ef 9y ago
    Very possible you saw a slight interruption around 11:30PST for around 10 minutes until we found and decommissioned the host that experienced this problem. We generally don't update status until we can verify impact/source, we see tons of limited outages from ISPs misbehaving.
- s_kilk 9y ago
  I was recording a podcast through Discord, and got hit by this particular outage. To be fair, it’s the first I’ve seen first hand so
- aefx 9y ago
  I was using it on Friday. My friend couldn't connect and I had trouble jumping into voice chat. I closed the client and was able to log in 10 minutes later. Overall I only noticed an issue for about 15-20 minutes. Having just read the post mortem I'm pretty impressed with their service and operations.
jhgg 9y ago
It's worth noting that the instance migration basically null-routed the redis VM for a good 30 minutes, until we manually intervened and restarted it. The instance was completely disconnected from the internal network immediately following the migration. From what we could gather from instance logs, the routing table on the VM was completely dropped and it could not even connect to the magic metadata service (metadata.internal - we saw "no route to host" errors for that). This is a pretty serious bug within GCP and we've already opened a case with them hoping they can get a fix. I think this is the 4th or 5th major bug we've encountered with their live migration system that could have, or has led to an outage or internal service degradation. GCP team has seriously investigated and fixed every bug we've reported to them so far, so props to them for that! Live migration is incredibly difficult to get right. We believe this triggered a bug in the redis-py python driver we use (specifically this one: https://github.com/andymccurdy/redis-py/pull/886 https://github.com/andymccurdy/redis-py/pull/886) that made us have to rolling restart our API cluster in the first place, to get the connection pools back into a working state. redis-sentinel had appropriately detected the instance going away, and initiated a fail-over almost immediately following the instance going offline, but due to the odd network situation that was caused by the migration (absolute packet loss instead of connections being reset) - the client driver was unable to properly fail-over to the new master. We already have work planned for our own connection pooling logic for redis-py - as right now the state of the drive in HA redis is actually pretty awful, and the maintainer doesn't appear to have the time to close or look at PRs that address these issues (we opened one that fixes a pretty serious bug during fail-over in march https://github.com/andymccurdy/redis-py/pull/847 https://github.com/andymccurdy/redis-py/pull/847 that has yet to be addressed).
- fulafel 9y ago
  For those of us unfamiliar with GCP, do you mean that the default-route of your VM was unable to route its traffic? Or is there a routing config running on customer VMs that GCP live-manages?
  - b1naryth1ef 9y ago
    GCP has a virtual networking stack to support a bunch of crazy (and awesome) features Google has built. Unfortunately the complexity here seems to hurt power-users like us. In this case it appears that for some unknown reason the node failed to program its network stack when coming up, meaning it was completely unavailable (even the metadata service used internally by google failed).
phreack 9y ago
Ever since they launched screen sharing, I've uninstalled both Skype and Hangouts and relied entirely on it for pair programming sessions. The smoothness of the reproduction is just incredible, and I don't see myself going back soon.
cordite 9y ago
The level of detail and linearity is impressive. At this scale, it seems like it may be warranted to start using reliability testing in production in like with Netflix. At the end I see mention of a library with flaws. I am curious as to which library that is, given I develop some projects in Elixir.
- b1naryth1ef 9y ago
  Thanks, we try our best with these. Past experience has shown they can be very valuable, and help everyone at the company get context on the system and how we handle failures. Reliability testing is definitely something we're interested in as we spin up more SRE/reliability focused individuals, but also has probably the least amount of cost-benefit for us (compared to engineering effort on improving the things we know need work). Some of the failure in the system we experienced is related to issues we know about, but haven't prioritized (read; had time for) yet. For the library, we believe the bug is related to hackney and the fact it uses the high priority setting for its pool process. For some reason (this is the part we're not entirely sure on, and still spending some time investigating) this high priority process got stuck and consumed all of the scheduler time (presumably related to the earlier API degradation), breaking the distribution port and the application in a weird way. Oddly enough the systems we run on are SMP, so in theory one rogue process should not be able to have this effect.
  - cordite 9y ago
    That is indeed very odd! Thank you for sharing. Hackney, through another library, is used in a telegram api wrapper that I wrote up. Though my stuff usually runs on a $5 vps, nothing with multiple cores.
humanfromearth 9y ago
We had the exact same issue with RMQ (HA setup) on GCP (running on GKE) a few weeks ago. Tried contacting support about this, it's paid - no customer support for their own bugs. The solution we came up so far is to disable automatic migrations. Not sure if that option actually does anything.
- sleepydog 9y ago
  You can't disable automatic migrations in GCP. You can choose between allowing live migrations (move the instance while it's still running) or (hard) instance reboots.
  - humanfromearth 9y ago
    You're right. I meant hard reboots.

96 points by b1naryth1ef 9y ago

lwansbrough 9y ago

Funny definition of HA. :)

atomical 9y ago

Does anyone use discord for work?

katastic 9y ago
I have informally with co-workers. But not in any official capacity. It's like 20x better than every other product out there though. And their new video chat + screen sharing is pretty great. The bandwidth is far higher than any other competitors I've used. My brother and I were playing 1080p videos on each of our screens and watching the other's, just to test it out. Obviously it wasn't full quality, but it kept the frame rate up and looked presentable at least to 720p.
- avree 9y ago
  Weird, I find Discord's audio quality especially to be terrible. And their lack of scalable monetization leads me worried about its longterm success as a platform - they are adding more cost-intensive features and continuing to try to support it with what is essentially a $5 monthly donation model.
  - b1naryth1ef 9y ago
    Can you give more explicit examples of the bad audio quality you experience? I'd be happy to forward this onto our native team to look into if there are some concrete things they can look at. Generally 99% of the audio issues we see people experience are due to ISP/peering/DDoS/etc issues, most of which are handled automatically by our servers within a few minutes.
    - mooman219 9y ago
      Anecdotal, but direct calls are pretty unusable for me. I'm on the west coast, and when I attempt calls to the east cost, it frequently cuts out. The workaround was just creating a server and using a voice room in it. This had drastically higher call quality. I assume this is ISP/Peering related, but to see such a night and day difference between voice channels and direct calls leads me to believe that there's something that can be done on your end. Complaints aside, I love the service. Echoing what avree said, long term monetization worries me. I would like to see discord survive, but its story looks similar to Trello right now in terms of monetization.
      - b1naryth1ef 9y ago
        Ahhhh, thats actually something we're aware of. Currently direct calls run on an entirely separate set of metals vs. everything else (this was mostly to help us test/measure video & screenshare rollout). Unfortunately some providers seem to be having issues with DDoS filtering over-triggering when they see video traffic, which negatively impacts the whole server. Something we're hopefully fixing in the short term!
    - asddddd 9y ago
      I've been using voice calls regularly, and it's occasionally been problematic. Yesterday it was behaving as though it had absolutely insane packet loss (occasional robotic sounding fragments would make it through, otherwise the line was dead) without any indication of high jitter or packet loss - I had to resort to stupid workarounds to get it working. Move convo to a server -> change server to Central from West (why can't this be done for regular calls!) -> instantly working perfectly. FWIW, it generally works pretty well, and overall, Discord is a fantastic product that I'm happy to be using.
    - eropple 9y ago
      FWIW, a serious feature that I would pay you money for right-here-right-now is the ability to multitrack audio. Let me give you many dollars to route each voice to a separate Soundflower channel, so that I can mix them outside of Discord, and I will give you said monies. I'm pretty sure you're even sending unmixed streams down? But I can't get at them! This probably involves not being Electron (from my own adventures in the area), so I don't hold out much hope, but it keeps Discord from replacing Extremely Expensive And Bad solutions like SkypeTX for me.
  - gizmo385 9y ago
    One thing that I've noticed helps is swapping the region for the server. It doesn't result in any noticeable outage on your server and has resolved any voice quality issues that we've run into most of the time. Usually just swapping between US-Central and US-West in our case.
sbarre 9y ago
We've debated it internally, but currently have decided to stick with Slack (free version)
andyfleming 9y ago
I briefly thought about whether it would be a better option than Slack. However, Slack seems a little more polished in the areas that we actually use. The subtle, better design in places like the core chat experience aren't worth giving up. I have used Discord quite a bit for gaming, but it hasn't proved a better option than Slack to me (at least in a work context).
zanny 9y ago
I would strongly recommend against it. Discord has no privacy guarantees in their TOS and actively says they will sell your data. Its how they offer their service "for free". Workplace conversations should be private, and that almost exclusively means either a siloed self hosted solution or a service with a contract that guarantees data integrity and privacy. If you are playing video games using Discord is fine, at least up until you start talking about personal details of your life in their chat rooms or over their voice services. If I'm having conversations I don't want Discord's data customers having, I'll use Signal / Matrix (self hosted or encrypted channels) / Mumble.
- eslachance 9y ago
  That's nice misinformation, mate. Discord makes money from Nitro, and they started off with a fairly large investor cushion which grew since launch. There is nothing in the ToS that states in any way, shape, or form, that they can sell their data. I challenge you to quote the exact place where it says that.
  - lightedman 9y ago
    You don't look in the ToS. You look in the company articles of incorporation or bylaws which will explain things like how money is allowed to be made.
    - exikyut 9y ago
      How would that work? If a company drafts ToS and then plumb violates them, well, it'll be found out eventually and then you get things like the recent debacle with telco data being sold (https://news.ycombinator.com/item?id=15477286 https://news.ycombinator.com/item?id=15477286), and then the company(s) in question will either get completely shredded or their reputation will at least be seriously besmirched. I can see the logic of what you're saying, but at the same time I can't see how it would hold up. If you can show me how it would work I'm listening though.
      - 9y ago
synicalx 9y ago
Unofficially, for "out of band" discussions and also for on-call events where we need to collab on stuff. We're stuck with Skype for official stuff and telephony though.
jakebasile 9y ago
We did, but due to some people having image issues about using a “gamer tool”, we were forced to switch back to the inferior Slack.
eslachance 9y ago
I'll need to echo some of the other comments here - at work we're pretty much stuck with Skype for Business (aka "lync") because the voice system ties into it. When I suggested Discord, it was shrugged off because it's "Chat for Gamers" and clearly they're not going to budge from that niche anytime soon, from all the recent features I've seen.

ZeroCool2u 9y ago

I'm really impressed. I was using Discord for most of this weekend, specifically Friday and Saturday. Never noticed any issues.

gizmo385 9y ago
They don't have a posted outage for Saturday, but I noticed issues with it on Saturday evening/night I believe. I'm wondering if it was related to the issues that they included in the post-mortem.
- b1naryth1ef 9y ago
  Very possible you saw a slight interruption around 11:30PST for around 10 minutes until we found and decommissioned the host that experienced this problem. We generally don't update status until we can verify impact/source, we see tons of limited outages from ISPs misbehaving.
s_kilk 9y ago
I was recording a podcast through Discord, and got hit by this particular outage. To be fair, it’s the first I’ve seen first hand so
aefx 9y ago
I was using it on Friday. My friend couldn't connect and I had trouble jumping into voice chat. I closed the client and was able to log in 10 minutes later. Overall I only noticed an issue for about 15-20 minutes. Having just read the post mortem I'm pretty impressed with their service and operations.

jhgg 9y ago

It's worth noting that the instance migration basically null-routed the redis VM for a good 30 minutes, until we manually intervened and restarted it. The instance was completely disconnected from the internal network immediately following the migration. From what we could gather from instance logs, the routing table on the VM was completely dropped and it could not even connect to the magic metadata service (metadata.internal - we saw "no route to host" errors for that). This is a pretty serious bug within GCP and we've already opened a case with them hoping they can get a fix. I think this is the 4th or 5th major bug we've encountered with their live migration system that could have, or has led to an outage or internal service degradation. GCP team has seriously investigated and fixed every bug we've reported to them so far, so props to them for that! Live migration is incredibly difficult to get right. We believe this triggered a bug in the redis-py python driver we use (specifically this one: https://github.com/andymccurdy/redis-py/pull/886 https://github.com/andymccurdy/redis-py/pull/886) that made us have to rolling restart our API cluster in the first place, to get the connection pools back into a working state. redis-sentinel had appropriately detected the instance going away, and initiated a fail-over almost immediately following the instance going offline, but due to the odd network situation that was caused by the migration (absolute packet loss instead of connections being reset) - the client driver was unable to properly fail-over to the new master. We already have work planned for our own connection pooling logic for redis-py - as right now the state of the drive in HA redis is actually pretty awful, and the maintainer doesn't appear to have the time to close or look at PRs that address these issues (we opened one that fixes a pretty serious bug during fail-over in march https://github.com/andymccurdy/redis-py/pull/847 https://github.com/andymccurdy/redis-py/pull/847 that has yet to be addressed).

fulafel 9y ago
For those of us unfamiliar with GCP, do you mean that the default-route of your VM was unable to route its traffic? Or is there a routing config running on customer VMs that GCP live-manages?
- b1naryth1ef 9y ago
  GCP has a virtual networking stack to support a bunch of crazy (and awesome) features Google has built. Unfortunately the complexity here seems to hurt power-users like us. In this case it appears that for some unknown reason the node failed to program its network stack when coming up, meaning it was completely unavailable (even the metadata service used internally by google failed).

phreack 9y ago

Ever since they launched screen sharing, I've uninstalled both Skype and Hangouts and relied entirely on it for pair programming sessions. The smoothness of the reproduction is just incredible, and I don't see myself going back soon.

cordite 9y ago

The level of detail and linearity is impressive. At this scale, it seems like it may be warranted to start using reliability testing in production in like with Netflix. At the end I see mention of a library with flaws. I am curious as to which library that is, given I develop some projects in Elixir.

b1naryth1ef 9y ago
Thanks, we try our best with these. Past experience has shown they can be very valuable, and help everyone at the company get context on the system and how we handle failures. Reliability testing is definitely something we're interested in as we spin up more SRE/reliability focused individuals, but also has probably the least amount of cost-benefit for us (compared to engineering effort on improving the things we know need work). Some of the failure in the system we experienced is related to issues we know about, but haven't prioritized (read; had time for) yet. For the library, we believe the bug is related to hackney and the fact it uses the high priority setting for its pool process. For some reason (this is the part we're not entirely sure on, and still spending some time investigating) this high priority process got stuck and consumed all of the scheduler time (presumably related to the earlier API degradation), breaking the distribution port and the application in a weird way. Oddly enough the systems we run on are SMP, so in theory one rogue process should not be able to have this effect.
- cordite 9y ago
  That is indeed very odd! Thank you for sharing. Hackney, through another library, is used in a telegram api wrapper that I wrote up. Though my stuff usually runs on a $5 vps, nothing with multiple cores.

humanfromearth 9y ago

We had the exact same issue with RMQ (HA setup) on GCP (running on GKE) a few weeks ago. Tried contacting support about this, it's paid - no customer support for their own bugs. The solution we came up so far is to disable automatic migrations. Not sure if that option actually does anything.

sleepydog 9y ago
You can't disable automatic migrations in GCP. You can choose between allowing live migrations (move the instance while it's still running) or (hard) instance reboots.
- humanfromearth 9y ago
  You're right. I meant hard reboots.