15 ms·

How Discord Scaled Elixir to 5M Concurrent Users (2017)

528 points by lelf 8y ago

denart2203 8y ago
Really neat post. Did y’all consider GenStage or another demand based approach for your overflow problem? I’d be interested in hearing about the tradeoffs between demand vs semaphore, it seems like the two have some similarities.
- cuddlecake 8y ago
  I think the main difference is that "demand" in GenStage tells the producer how much work the producer should do to satisfy the consumer. The consumer sets the demand, the producer does not limit it. With how discord uses semaphores, the Consumer will not even make a demand for more events from the producer. Other than that, I think in the case of Discord, it's more of an RPC thing, and GenStage is rather made for concurrent streaming and processing of data.
aboutruby 8y ago
I'm continuously impressed at the quality of the software coming from Discord. By far the best messaging user experience on desktop there is. Discord already uses Cloudflare, I'm curious what they think of Workers. edit: I'm on macOS, not an heavy user, I used Slack / Skype / Hangouts / WhatsApp / Messenger / etc. before
- tomc1985 8y ago
  Discord lets me pick sound output and input devices... Slack doesn't even do that! Never in the world would I have thought a gaming chat client had more robust features than the pricey enterprise chat client
  - tenryuu 8y ago
    If you're on windows, the 'App Volume device preferences' settings pane has you covered It's the replacement for the vintage volume control, while a little sub-par by any modern standard, you can pick the input/output device per application
- deleted 8y ago
  [deleted]
- qpery 8y ago
  >By far the best messaging user experience on desktop there is. No offence but that's like your opinion. Coming from IRC and TeamSpeak, to me Discord is unnecessarily bloated and full of analytics. >I used Slack / Skype / Hangouts / WhatsApp / Messenger / etc. before Yeah, explains it. :P
  - tapland 8y ago
    I'm still on IRC. It is much quicker and it is much easier to be connected and active in multiple servers/channels, whereas the overview in Discord is terrible unless you use third party clients, which can get you banned from the service. The voice service isn't very high quality and there are frequent server outages. I use Telegram, Facebook, whatever except Discord for instant messaging, and wish Discord wasn't the default for non-professional groups. People my age (upper 20s) praising it are always those who used Skype for gaming and never touched Teamspeak/Ventrilo/Mumble.
    - fastball 8y ago
      I used Teamspeak and Ventrilo for gaming back in the day and much prefer Discord.
    - devonkim 8y ago
      I was a user of all three of the gaming-centric chat / voice services and the issue always has been about centralizing different communities together rather than quality of service / technology itself. One group plays on a random guy’s Teamspeak server that’s been running in a VPS or colo for 6 years, another splinter group moved to Vent, etc. Then there were the fun times dealing with various authentication and rotating certificates (if you practiced decent security at all you had to have one) for each of these disparate communities. As gaming itself got broader reach a lot of casual folks were left to their own devices and underserved, so that’s the big gap being seen. I know plenty of 20-somethings that know about and used the predecessors to Discord but they’re all hardcore gamers compared to even the folks closer to mid/late 30s that may have been hardcore before but don’t have the time to fiddle with these systems anymore. It seems with Discord quality was tough to maintain at scale and we’ve got the inverse problem set.
- cameronhowe 8y ago
  I have the opposite experience. Frequent outages/voice connectivity degradation. Font rendering was broken on many linux distros for months. And now there has been some kind of crash for about a month and still ongoing where discord becomes sluggish and eventually closes over time. (If you have a game open it can crash within one hour) Not to mention where an update randomly the noise suppression would completely mute your mic.
- rumblefrog 8y ago
  Unfortunately I have the opposite experience - Abusive trust and safety team - Constant outages for both users and bots - Frontend is heavily bloated - UI designed for money grab instead for the users
  - deleted 8y ago
    [deleted]
  - qpery 8y ago
    By judging all the strings in the client frontend, their help website, and my interactions with the support team, Discord is being run by some rather immature people.
  - metildaa 8y ago
    Don't forget the wierd dependency hell when it comes to writing Discord bots in Python. They require you use an older version of Python that is "fun" to work with.
    - jhgg 8y ago
      All the Diacord API libraries are community built and maintained.
    - tomc1985 8y ago
      Why do you have to use Python?
  - busterarm 8y ago
    Discord's trust and safety team has come under fire lately for their (until a week ago) lax policy on "cub content" (for those unfamiliar, it's the furry equivalent of child porn). Apparently some of the people on the team may have been participants in sharing it; there are number of furries on that team (at least: Tinyfeex, Allthefoxes). While I appreciate their technical accomplishments, this is one of those things that gives me chills about the company.
  - rumblefrog 8y ago
    As much as I want to support Discord, their goal from when they initially started as a company has shifted in an entirely new direction. I feel betrayed as a 2015 user.
- snek 8y ago
  I can definitely agree that Discord is a fantastic chat experience. I use both IRC and Discord, and I think Discord (if only slightly) takes the cake. Discord has actually been using workers since before they were available to the general public. They've been using them mainly for edge caching games for the store, A/B, and promotion of clients to different testing environments.
- nikentic 8y ago
  I fully agree, compared to other widely used software in the gaming space (Teamspeak, Mumble, Ventrilo). Experience to join new servers, create new content on existing one and chat+voice is amazing!
- zinclozenge 8y ago
  Having gone through some of their Elixir libraries, I concluded the opposite. They have some very inefficient and ugly code in a lot of places.
- ziont 8y ago
  wow...thats a first
- gsich 8y ago
  Not self-hostable.
- jhgg 8y ago
  >I'm curious what they think of Workers. We are a very heavy user of cloudflare workers. Our marketing/developer/web app are entirely served from the edge using workers - and we use workers for request authentication for downloading game chunks on our store.
  - aboutruby 8y ago
    Thanks a lot, I should have googled it, I would have seen your previous comment mentioning it which has complementary information[1]. [1]: https://news.ycombinator.com/item?id=17447355 https://news.ycombinator.com/item?id=17447355
juhatl 8y ago
Earlier HN discussion from 2017 on the same post: https://news.ycombinator.com/item?id=14748028 https://news.ycombinator.com/item?id=14748028
- deleted 8y ago
  [deleted]
eric_khun 8y ago
Is the ETS cache system can be used by something else than elixir?
- jimmy_ruska 8y ago
  When you have an erlang distributed app and need caching without having the overhead of converting erlang terms then ETS is the best solution. That's not to say you would want to use ETS instead of redis from a java app.
- toast0 8y ago
  ETS is basically a key value store with optional sorting, multiple values, limited atomic updates, and a weird query language. I'm not sure how well it compares to other key value stores, but it's main differentiator is that it stores Erlang native types without a significant marshalling burden (at least for the developer -- if you store complex values, it's still plenty of work for the runtime). It wouldn't be a lot of work to build an Erlang service to expose ETS to something else, but I don't know that the data marshalling required would be worth it. More interesting might be to expose mnesia, maybe. But even then, it might be most useful to build out your data handling in Erlang and expose a higher level API for clients in other languages.
est31 8y ago
From time to time my ISP changes the IP address and discord totally freaks out about that. I have to fill out a captcha and have to receive a verification E-Mail. They are treating me as if I used asdf1234 as my password.
- ubercow13 8y ago
  Yes, and if they don't like you enough they lock your account until you provide a phone number (no thanks)
spork12 8y ago
Is that why discord uses more memory than my chrome browser? ;)
nottorp 8y ago
Nice. How about fixing the client side by, you know, doing a native client?
rcarmo 8y ago
I'm a bit curious as to the message processing overhead. BEAM and HIPE are relatively fast, but string processing of any kind was kind of slow back when I messed around with Elixir...
- im_down_w_otp 8y ago
  Can you be more specific in what you mean by "processing"? - Reading from the mailbox? - Destructuring and binding the content? - Operating on the data by transforming, modifying, or filtering it? etc.
- siscia 8y ago
  String manipulation is notoriously slow in the BEAM, mostly implementation reasons. Message passing is somehow slow as well, slower than go(Lang) of you want a comparison. But on those systems, the one that you design on the BEAM, velocity is usually not a problem. What you usually try to do is to reach a design that does not have a single bottle neck or failure point so that pretty much whatever happen you can just add machines. This turn out to be a great way to design multi{processes, cores} and multi nodes system. What you usually get on a BEAM under load is an 100% of CPU usage, even on multi core, but to be honest those core are not used as efficiently as you could. It is a matter of tradeoffs, I can quickly write multi process software that can easily scale, but I will leave on the table some raw performance number. Again it turns out that just raw numbers are not as important as they are simple to measure.
  - brightball 8y ago
    FWIW, that’s also why use of IO Lists are strongly encouraged rather than string manipulation. It’s also why HTML with Phoenix is so fast. https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/ https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2...
- brightball 8y ago
  Use IO List rather than strings. Same result, but much much faster. https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/ https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2...
sametmax 8y ago
While 5M concurrent chat users is definitly a massive feat, it seems to me that it's kinda this tech (Erlang) very sweet spot and not being able to do so would have been disapointing. Or am I missing something ?
- HelloFellowDevs 8y ago
  They are attempting to expand to being a springboard for games and apps within their app. Sort of like Steam, but since you're already chatting with friends why not start up a game of League?
  - sametmax 8y ago
    No I mean Erlang.
- cthalupa 8y ago
  Sure. This is basically the exact sort of problem Erlang is designed to solve. But it's still cool to see it done, and a lot of people are unfamiliar with Erlang or Elixir, so the discussion is interesting.
- pdimitar 8y ago
  My takeaway is that Erlang/Elixir aren't designed to squeeze that much performance with that setup and that the Discord team went out of their way to optimize hot-spots to make it possible. Erlang/Elixir are still quite impressive out of the box, even without these Discord-specific optimizations. But they probably wouldn't scale to 5M right away.
  - sametmax 8y ago
    Nothing does, 5 millions __concurrent__ user creating and sending new data in real time is huge. Not to mention video and sound.
    - pdimitar 8y ago
      Yes. In my eyes the win here isn't concurrent performance that cannot get beaten by anything else; I am pretty sure a carefully crafted Golang or Rust stack can beat Elixir any day. But they won't have the fault tolerance guarantees of the BEAM VM. The real win IMO is the good reliability:performance ratio that Discord achieved. I feel Erlang/Elixir are excellent in optimizing this exact metric.
dev_dull 8y ago
> Sending messages between Erlang processes was not as cheap as we expected, and the reduction cost — Erlang unit of work used for process scheduling — was also quite high. This is really surprising to me, and definitely something the elixir team should look at optimizing. Sending messages should be extremely fast.
- zinclozenge 8y ago
  That's something that the BEAM group would work on, although jose has contributed some patches to the compiler.
- jadbox 8y ago
  I have no insight into how Erlang performs this process, but I'd assume Erlang is performing sending messages almost as fast as it's possible, as it's a key part of the platform and has had two decades to perfect it. Most likely this is a limitation of the paradigm (of the Actor model) and not the implementation.
  - larryweya 8y ago
    I believe the bottleneck the author was trying to overcome here is how the Erlang VM moves processes to the back of the run queue once they hit a predetermined number of operations (preemptive scheduling) and not of the Actor Model.
- keithnoizu 8y ago
  There are ways to speed up message passing. It's not slow by any means but weird things become your bottle neck when you have 5m concurrent users. For instance you have to build you OTP tree extra layers deep or spawn hundreds of supervisors because just routine calls to the master process to add / remove children begins to become bottle neck. In my implementation, only 1/2 a million concurrent IOT devices, I use routing tables that narrow down a process to a node and registry cluster (https://hexdocs.pm/elixir/master/Registry.html https://hexdocs.pm/elixir/master/Registry.html) , and from that registry it fans out to 1 of 100 supervisors for that worker type per node.
- 8y ago
pault 8y ago
It was really interesting to read how each problem they solved uncovered another problem further down the pipeline. It also reveals how much is going on behind the scenes when a service goes down and everybody starts accusing the developers of incompetence. :)
- deleted 8y ago
  [deleted]
- deleted 8y ago
  [deleted]
- jchw 8y ago
  It also helps you learn to appreciate software that very rarely goes down at all. It's no simple accomplishment.
DevX101 8y ago
Have any Node.js/websocket implementations scaled on this order of magnitude? Would like to do a read-up of any challenges faced.
- wrong_variable 8y ago
  I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me of PG's having a higher bird eye view. NodeJS, Python, Haskell (!), PHP, C - all belong to the same class of coding style with the same type of problems. You use Erlang not because of playing the testosterone game of nominal performance - but because you really want some guarantee. ( Money should be not a problem since the world seems flush with cash - if your manager is complaining its because he wants his bonus to be higher. )
  - tehsauce 8y ago
    Care to elaborate what these types of problems are and how erland fixes them?
    - wrong_variable 8y ago
      Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
      - stingraycharles 8y ago
        Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).
xb95 8y ago
Discord infra engineer here -- this blog post needs an update! Since then we've scaled this system much more. :) The Fortnite Official server has exceeded 100,000 concurrent users, Discord itself is way past that 5M concurrent number, we're now using Rust in certain places to make Elixir go faster, we've built a general purpose replacement to Process.monitor that scales a whole truckload more that we're open sourcing next week at Code BEAM SF... the list goes on. There's a lot of fun stuff going on to try to make this system even more efficient and reliable, there's a lot to do still. We run everything on a very small engineering team (there are 4 fulltime engineers on the core infrastructure, only about 40 engineers in the whole company) and we're always looking for a few more. Feel free to reach out to me (zorkian#0001 on Discord) if this blog post sounds up your alley!
- devwastaken 8y ago
  How is webrtc handled at scale? Seems most webrtc server software doesn't perform that well, and discord I believe is using turn to proxy it all, so that's gotta be a lot of data flowing.
  - jhgg 8y ago
    The tldr is we wrote our own highly efficient SFU - and we used our own transport layer that is not dtls, but xsalsa poly over udp.
  - vhiremath4 8y ago
    https://blog.discordapp.com/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc-ce01c3187429 https://blog.discordapp.com/how-discord-handles-two-and-half...
- shostack 8y ago
  Congrats! You say only about 40 engineers in the company... How big is the company and how big are your other departments relative to engineering?
  - dajohnson89 8y ago
jhgg 8y ago
We've since ditched fastglobal actually, as we found that it'd take too long to recompile code as the number of processes on the BEAM VM grew. As it turns out, the cost to recompile is linear scaling up with the # of processes on the node. Once we reached the million processes per node threshold it ended up being too slow, and resource intensive to do fastglobal dynamic module recompilation. We've since been able to squeeze out the performance from ETS we needed to be able to run our hashring. More on that in this PR here: https://github.com/discordapp/ex_hash_ring/pull/1 https://github.com/discordapp/ex_hash_ring/pull/1 Also one amazing feat we've accomplished thanks to ZenMonitor (which we will open source soon as my colleague mentioned), is that we can now tolerate a guilds node failure and recover from it in ~40 seconds. We had a node fail on Friday actually due to the underlying host rebooting (gcp calls this a 'host error'), and didn't even notice until after the system had already recovered and the on-call got alerted after the fact. Back in the day this lead to a cascading failure throughout the system. A guild node runs between 600k-700k concurrent discord guilds, or servers as it's known in userland. And although we haven't done it recently, we clock a full restart of our distributed system at roughly 17 minutes (from shutdown to service fully restored).
- pm90 8y ago
  Interesting. Do you run on top of bare gce vms? What machine types do you use? Didn't know they were being automatically rebooted under the hood.
  - jhgg 8y ago
    Yes. We run bare VMs for this workload (n1-standard-X). They do in the event of hardware failure or other unexpected fault. Generally it does not get to that though, as google is fairly good at migrating your VMs off of a host if it detects it will fail soon.
    - boulos 8y ago
      Disclosure: I work on Google Cloud (and have emailed jh about Discord). Yeah, the most straightforward failure that we can’t migrate away from is the NIC (or rack switch) failing. Obviously, if your path to get off the box is dead, that’s not going to happen :). Others though, like the hypervisor crashing or even host kernel are also possible, but much less frequent than “Hmm, I think the network is dead, we should start a replacement VM”. One of the many motivations for (now) only having Persistent Disk for boot disks, is that it lets us avoid a whole class of truly unrecoverable errors. There are still bad DIMMs, but monitoring for ECC failures often lets us mark the host for replacement in time for the VMs to migrate off before they actually fail.
nicodjimenez 8y ago
Question for the Discord team.. if you started this project today in 2019, would you still build a homegrown event sourcing system? Or would you use Kafka?
- jhgg 8y ago
  We would still use our own homegrown system. And we do use kafka internally already for a plethora of other things. Just not on the real-time chat side of things.
  - nicodjimenez 8y ago
    Why not on the real time chat stuff?
    - jhgg 8y ago
      Because Kafka is definitely not a tool suited for real time chat and event distribution to millions of clients. Once a message hits our distribution it’s fanned our to clients on average 5-10ms later.
JulianMorrison 8y ago
Needs some attention to the scaling down, IMO. Discord is annoying on intermittent connections. Why do sent messages not have identity such that they can go through twice?
- jhgg 8y ago
  We actually added this 2nd half of last year. Message send operations are now deduped and idempotent given a client-generated nonce. Are you still experiencing this issue?
  - ubercow13 8y ago
    Yes I experience this constantly, very recently, on poor internet connections
  - JulianMorrison 8y ago
    Your nonce might not survive the "message -> disconnect -> red message -> re-send" flow. Could you just get rid of red messages and make it "pending" until acked as "sent"?
jplayer01 8y ago
So why can't they add basic features like organizing servers?
pdimitar 8y ago
A corollary: As a mostly Elixir programmer for 2 years now, I find it quite amusing how the Kubernetes community tries their damnest to emulate Erlang's OTP system -- going as far as to bolt stricter typing system on Golang even -- and try and reinvent Erlang's "let it crash and get rebooted" idea. I am however not at all convinced that "let it crash" is a good mantra when applied to entire containers. One such container can take 20+ seconds to restart and you can lose a lot more compared to the mini-processes Erlang/Elixir have which are happily left to crash and [semi-]auto-recover. Imagine if that container was ingesting events and crashed when it had 5_000 in its in-memory queue and only managed to process 50-100 of them. Or imagine 100 transactions in progress being cancelled. I applaud the hard work of the Kubernetes team, I am just not very sure they invest their energy in the right tech stack. Sure Golang is faster than Erlang/Elixir -- by a lot, too. But is it not K8s idea to be fault-tolerant and not immensely quick? What does a Golang's speed matter when a container needs seconds to reboot? In my eyes, K8s could have been written in Bash shell scripts... but I am probably missing something important.
- Thaxll 8y ago
  Kubernetes is based on Google Borg / Omega not OTP very different things. As for your queue / DB question there is something called transactions for that kind of problem, I hope you don't use OTP features replacing transactions or you're looking for very serious issues. OTP doesn't do 5% of what Kubernetes is able to provide ( cpu / io / memory quota, live / readiness probes, rolling deployments / canary / blue green.
  - pdimitar 8y ago
    I knew somebody will get my random examples and turn them against me. :D Serves me right for not researching several hours to find the perfect example, I guess! I am well aware of the differences in scope between K8s and Erlang/OTP. My point was that they kind of try to do the same only on different levels. And I am still not sold on the idea of throwaway containers. That works well if you have a hyper-network of microservices that are able to discover each other and self-heal a bigger graph of services but if you have any sort of a more classic coherent whole app... then not really. Throwing containers away and rebooting new copies isn't something that looks viable for many projects. But hell, who knows. K8s team and their audience are really dedicated. They might change every single tool of the trade just to make K8s work well. I was mostly saying that I don't feel it brings something really radical to the table.

528 points by lelf 8y ago

Really neat post. Did y’all consider GenStage or another demand based approach for your overflow problem? I’d be interested in hearing about the tradeoffs between demand vs semaphore, it seems like the two have some similarities.

cuddlecake 8y ago
I think the main difference is that "demand" in GenStage tells the producer how much work the producer should do to satisfy the consumer. The consumer sets the demand, the producer does not limit it. With how discord uses semaphores, the Consumer will not even make a demand for more events from the producer. Other than that, I think in the case of Discord, it's more of an RPC thing, and GenStage is rather made for concurrent streaming and processing of data.

aboutruby 8y ago

I'm continuously impressed at the quality of the software coming from Discord. By far the best messaging user experience on desktop there is. Discord already uses Cloudflare, I'm curious what they think of Workers. edit: I'm on macOS, not an heavy user, I used Slack / Skype / Hangouts / WhatsApp / Messenger / etc. before

tomc1985 8y ago
Discord lets me pick sound output and input devices... Slack doesn't even do that! Never in the world would I have thought a gaming chat client had more robust features than the pricey enterprise chat client
- tenryuu 8y ago
  If you're on windows, the 'App Volume device preferences' settings pane has you covered It's the replacement for the vintage volume control, while a little sub-par by any modern standard, you can pick the input/output device per application
deleted 8y ago
[deleted]
qpery 8y ago
>By far the best messaging user experience on desktop there is. No offence but that's like your opinion. Coming from IRC and TeamSpeak, to me Discord is unnecessarily bloated and full of analytics. >I used Slack / Skype / Hangouts / WhatsApp / Messenger / etc. before Yeah, explains it. :P
- tapland 8y ago
  I'm still on IRC. It is much quicker and it is much easier to be connected and active in multiple servers/channels, whereas the overview in Discord is terrible unless you use third party clients, which can get you banned from the service. The voice service isn't very high quality and there are frequent server outages. I use Telegram, Facebook, whatever except Discord for instant messaging, and wish Discord wasn't the default for non-professional groups. People my age (upper 20s) praising it are always those who used Skype for gaming and never touched Teamspeak/Ventrilo/Mumble.
  - fastball 8y ago
    I used Teamspeak and Ventrilo for gaming back in the day and much prefer Discord.
  - devonkim 8y ago
    I was a user of all three of the gaming-centric chat / voice services and the issue always has been about centralizing different communities together rather than quality of service / technology itself. One group plays on a random guy’s Teamspeak server that’s been running in a VPS or colo for 6 years, another splinter group moved to Vent, etc. Then there were the fun times dealing with various authentication and rotating certificates (if you practiced decent security at all you had to have one) for each of these disparate communities. As gaming itself got broader reach a lot of casual folks were left to their own devices and underserved, so that’s the big gap being seen. I know plenty of 20-somethings that know about and used the predecessors to Discord but they’re all hardcore gamers compared to even the folks closer to mid/late 30s that may have been hardcore before but don’t have the time to fiddle with these systems anymore. It seems with Discord quality was tough to maintain at scale and we’ve got the inverse problem set.
cameronhowe 8y ago
I have the opposite experience. Frequent outages/voice connectivity degradation. Font rendering was broken on many linux distros for months. And now there has been some kind of crash for about a month and still ongoing where discord becomes sluggish and eventually closes over time. (If you have a game open it can crash within one hour) Not to mention where an update randomly the noise suppression would completely mute your mic.
rumblefrog 8y ago
Unfortunately I have the opposite experience - Abusive trust and safety team - Constant outages for both users and bots - Frontend is heavily bloated - UI designed for money grab instead for the users
- deleted 8y ago
  [deleted]
- qpery 8y ago
  By judging all the strings in the client frontend, their help website, and my interactions with the support team, Discord is being run by some rather immature people.
- metildaa 8y ago
  Don't forget the wierd dependency hell when it comes to writing Discord bots in Python. They require you use an older version of Python that is "fun" to work with.
  - jhgg 8y ago
    All the Diacord API libraries are community built and maintained.
  - tomc1985 8y ago
    Why do you have to use Python?
- busterarm 8y ago
  Discord's trust and safety team has come under fire lately for their (until a week ago) lax policy on "cub content" (for those unfamiliar, it's the furry equivalent of child porn). Apparently some of the people on the team may have been participants in sharing it; there are number of furries on that team (at least: Tinyfeex, Allthefoxes). While I appreciate their technical accomplishments, this is one of those things that gives me chills about the company.
- rumblefrog 8y ago
  As much as I want to support Discord, their goal from when they initially started as a company has shifted in an entirely new direction. I feel betrayed as a 2015 user.
snek 8y ago
I can definitely agree that Discord is a fantastic chat experience. I use both IRC and Discord, and I think Discord (if only slightly) takes the cake. Discord has actually been using workers since before they were available to the general public. They've been using them mainly for edge caching games for the store, A/B, and promotion of clients to different testing environments.
nikentic 8y ago
I fully agree, compared to other widely used software in the gaming space (Teamspeak, Mumble, Ventrilo). Experience to join new servers, create new content on existing one and chat+voice is amazing!
zinclozenge 8y ago
Having gone through some of their Elixir libraries, I concluded the opposite. They have some very inefficient and ugly code in a lot of places.
ziont 8y ago
wow...thats a first
gsich 8y ago
Not self-hostable.
jhgg 8y ago
>I'm curious what they think of Workers. We are a very heavy user of cloudflare workers. Our marketing/developer/web app are entirely served from the edge using workers - and we use workers for request authentication for downloading game chunks on our store.
- aboutruby 8y ago
  Thanks a lot, I should have googled it, I would have seen your previous comment mentioning it which has complementary information[1]. [1]: https://news.ycombinator.com/item?id=17447355 https://news.ycombinator.com/item?id=17447355

juhatl 8y ago

Earlier HN discussion from 2017 on the same post: https://news.ycombinator.com/item?id=14748028 https://news.ycombinator.com/item?id=14748028

deleted 8y ago
[deleted]

eric_khun 8y ago

Is the ETS cache system can be used by something else than elixir?

jimmy_ruska 8y ago
When you have an erlang distributed app and need caching without having the overhead of converting erlang terms then ETS is the best solution. That's not to say you would want to use ETS instead of redis from a java app.
toast0 8y ago
ETS is basically a key value store with optional sorting, multiple values, limited atomic updates, and a weird query language. I'm not sure how well it compares to other key value stores, but it's main differentiator is that it stores Erlang native types without a significant marshalling burden (at least for the developer -- if you store complex values, it's still plenty of work for the runtime). It wouldn't be a lot of work to build an Erlang service to expose ETS to something else, but I don't know that the data marshalling required would be worth it. More interesting might be to expose mnesia, maybe. But even then, it might be most useful to build out your data handling in Erlang and expose a higher level API for clients in other languages.

est31 8y ago

From time to time my ISP changes the IP address and discord totally freaks out about that. I have to fill out a captcha and have to receive a verification E-Mail. They are treating me as if I used asdf1234 as my password.

ubercow13 8y ago
Yes, and if they don't like you enough they lock your account until you provide a phone number (no thanks)

spork12 8y ago

Is that why discord uses more memory than my chrome browser? ;)

nottorp 8y ago

Nice. How about fixing the client side by, you know, doing a native client?

rcarmo 8y ago

I'm a bit curious as to the message processing overhead. BEAM and HIPE are relatively fast, but string processing of any kind was kind of slow back when I messed around with Elixir...

im_down_w_otp 8y ago
Can you be more specific in what you mean by "processing"? - Reading from the mailbox? - Destructuring and binding the content? - Operating on the data by transforming, modifying, or filtering it? etc.
siscia 8y ago
String manipulation is notoriously slow in the BEAM, mostly implementation reasons. Message passing is somehow slow as well, slower than go(Lang) of you want a comparison. But on those systems, the one that you design on the BEAM, velocity is usually not a problem. What you usually try to do is to reach a design that does not have a single bottle neck or failure point so that pretty much whatever happen you can just add machines. This turn out to be a great way to design multi{processes, cores} and multi nodes system. What you usually get on a BEAM under load is an 100% of CPU usage, even on multi core, but to be honest those core are not used as efficiently as you could. It is a matter of tradeoffs, I can quickly write multi process software that can easily scale, but I will leave on the table some raw performance number. Again it turns out that just raw numbers are not as important as they are simple to measure.
- brightball 8y ago
  FWIW, that’s also why use of IO Lists are strongly encouraged rather than string manipulation. It’s also why HTML with Phoenix is so fast. https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/ https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2...
brightball 8y ago
Use IO List rather than strings. Same result, but much much faster. https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2-io-lists-in-phoenix/ https://www.bignerdranch.com/blog/elixir-and-io-lists-part-2...

sametmax 8y ago

While 5M concurrent chat users is definitly a massive feat, it seems to me that it's kinda this tech (Erlang) very sweet spot and not being able to do so would have been disapointing. Or am I missing something ?

HelloFellowDevs 8y ago
They are attempting to expand to being a springboard for games and apps within their app. Sort of like Steam, but since you're already chatting with friends why not start up a game of League?
- sametmax 8y ago
  No I mean Erlang.
cthalupa 8y ago
Sure. This is basically the exact sort of problem Erlang is designed to solve. But it's still cool to see it done, and a lot of people are unfamiliar with Erlang or Elixir, so the discussion is interesting.
pdimitar 8y ago
My takeaway is that Erlang/Elixir aren't designed to squeeze that much performance with that setup and that the Discord team went out of their way to optimize hot-spots to make it possible. Erlang/Elixir are still quite impressive out of the box, even without these Discord-specific optimizations. But they probably wouldn't scale to 5M right away.
- sametmax 8y ago
  Nothing does, 5 millions __concurrent__ user creating and sending new data in real time is huge. Not to mention video and sound.
  - pdimitar 8y ago
    Yes. In my eyes the win here isn't concurrent performance that cannot get beaten by anything else; I am pretty sure a carefully crafted Golang or Rust stack can beat Elixir any day. But they won't have the fault tolerance guarantees of the BEAM VM. The real win IMO is the good reliability:performance ratio that Discord achieved. I feel Erlang/Elixir are excellent in optimizing this exact metric.

dev_dull 8y ago

> Sending messages between Erlang processes was not as cheap as we expected, and the reduction cost — Erlang unit of work used for process scheduling — was also quite high. This is really surprising to me, and definitely something the elixir team should look at optimizing. Sending messages should be extremely fast.

zinclozenge 8y ago
That's something that the BEAM group would work on, although jose has contributed some patches to the compiler.
jadbox 8y ago
I have no insight into how Erlang performs this process, but I'd assume Erlang is performing sending messages almost as fast as it's possible, as it's a key part of the platform and has had two decades to perfect it. Most likely this is a limitation of the paradigm (of the Actor model) and not the implementation.
- larryweya 8y ago
  I believe the bottleneck the author was trying to overcome here is how the Erlang VM moves processes to the back of the run queue once they hit a predetermined number of operations (preemptive scheduling) and not of the Actor Model.
keithnoizu 8y ago
There are ways to speed up message passing. It's not slow by any means but weird things become your bottle neck when you have 5m concurrent users. For instance you have to build you OTP tree extra layers deep or spawn hundreds of supervisors because just routine calls to the master process to add / remove children begins to become bottle neck. In my implementation, only 1/2 a million concurrent IOT devices, I use routing tables that narrow down a process to a node and registry cluster (https://hexdocs.pm/elixir/master/Registry.html https://hexdocs.pm/elixir/master/Registry.html) , and from that registry it fans out to 1 of 100 supervisors for that worker type per node.
8y ago

pault 8y ago

It was really interesting to read how each problem they solved uncovered another problem further down the pipeline. It also reveals how much is going on behind the scenes when a service goes down and everybody starts accusing the developers of incompetence. :)

deleted 8y ago
[deleted]
deleted 8y ago
[deleted]
jchw 8y ago
It also helps you learn to appreciate software that very rarely goes down at all. It's no simple accomplishment.

DevX101 8y ago

Have any Node.js/websocket implementations scaled on this order of magnitude? Would like to do a read-up of any challenges faced.

wrong_variable 8y ago
I have given up on NodeJS and moved solely to Erlang for long term projects. I still love NodeJS ! but once you learn Erlang its hard to go back, it reminds me of PG's having a higher bird eye view. NodeJS, Python, Haskell (!), PHP, C - all belong to the same class of coding style with the same type of problems. You use Erlang not because of playing the testosterone game of nominal performance - but because you really want some guarantee. ( Money should be not a problem since the world seems flush with cash - if your manager is complaining its because he wants his bonus to be higher. )
- tehsauce 8y ago
  Care to elaborate what these types of problems are and how erland fixes them?
  - wrong_variable 8y ago
    Sure ! Think about how prinf / console.log / print .... works in traditional settings. - How would you make it so that printf doesn't crash your entire program if the console hangs. - how would you isolate a single codebase's IO operations ? - How would writing to console work in a multi threaded environment ? multi server ? 100 servers ? Haskell sort of tries to answer these questions but I am not sure how its going go work out for them, in Erlang's process based universe all those questions have been answered already !
    - stingraycharles 8y ago
      Sounds like reasons to prefer FP over OOP, not necessarily Erlang. I do really appreciate Erlang / Elixir (have contributed several libraries), but the problems you describe are not uniquely solved by Erlang. Akka / Scala is another take on the whole actor based architecture, and seems to have considerably more traction (hiring talent will be easier for your manager).

xb95 8y ago

Discord infra engineer here -- this blog post needs an update! Since then we've scaled this system much more. :) The Fortnite Official server has exceeded 100,000 concurrent users, Discord itself is way past that 5M concurrent number, we're now using Rust in certain places to make Elixir go faster, we've built a general purpose replacement to Process.monitor that scales a whole truckload more that we're open sourcing next week at Code BEAM SF... the list goes on. There's a lot of fun stuff going on to try to make this system even more efficient and reliable, there's a lot to do still. We run everything on a very small engineering team (there are 4 fulltime engineers on the core infrastructure, only about 40 engineers in the whole company) and we're always looking for a few more. Feel free to reach out to me (zorkian#0001 on Discord) if this blog post sounds up your alley!

devwastaken 8y ago
How is webrtc handled at scale? Seems most webrtc server software doesn't perform that well, and discord I believe is using turn to proxy it all, so that's gotta be a lot of data flowing.
- jhgg 8y ago
  The tldr is we wrote our own highly efficient SFU - and we used our own transport layer that is not dtls, but xsalsa poly over udp.
- vhiremath4 8y ago
  https://blog.discordapp.com/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc-ce01c3187429 https://blog.discordapp.com/how-discord-handles-two-and-half...
shostack 8y ago
Congrats! You say only about 40 engineers in the company... How big is the company and how big are your other departments relative to engineering?
- dajohnson89 8y ago

jhgg 8y ago

We've since ditched fastglobal actually, as we found that it'd take too long to recompile code as the number of processes on the BEAM VM grew. As it turns out, the cost to recompile is linear scaling up with the # of processes on the node. Once we reached the million processes per node threshold it ended up being too slow, and resource intensive to do fastglobal dynamic module recompilation. We've since been able to squeeze out the performance from ETS we needed to be able to run our hashring. More on that in this PR here: https://github.com/discordapp/ex_hash_ring/pull/1 https://github.com/discordapp/ex_hash_ring/pull/1 Also one amazing feat we've accomplished thanks to ZenMonitor (which we will open source soon as my colleague mentioned), is that we can now tolerate a guilds node failure and recover from it in ~40 seconds. We had a node fail on Friday actually due to the underlying host rebooting (gcp calls this a 'host error'), and didn't even notice until after the system had already recovered and the on-call got alerted after the fact. Back in the day this lead to a cascading failure throughout the system. A guild node runs between 600k-700k concurrent discord guilds, or servers as it's known in userland. And although we haven't done it recently, we clock a full restart of our distributed system at roughly 17 minutes (from shutdown to service fully restored).

pm90 8y ago
Interesting. Do you run on top of bare gce vms? What machine types do you use? Didn't know they were being automatically rebooted under the hood.
- jhgg 8y ago
  Yes. We run bare VMs for this workload (n1-standard-X). They do in the event of hardware failure or other unexpected fault. Generally it does not get to that though, as google is fairly good at migrating your VMs off of a host if it detects it will fail soon.
  - boulos 8y ago
    Disclosure: I work on Google Cloud (and have emailed jh about Discord). Yeah, the most straightforward failure that we can’t migrate away from is the NIC (or rack switch) failing. Obviously, if your path to get off the box is dead, that’s not going to happen :). Others though, like the hypervisor crashing or even host kernel are also possible, but much less frequent than “Hmm, I think the network is dead, we should start a replacement VM”. One of the many motivations for (now) only having Persistent Disk for boot disks, is that it lets us avoid a whole class of truly unrecoverable errors. There are still bad DIMMs, but monitoring for ECC failures often lets us mark the host for replacement in time for the VMs to migrate off before they actually fail.

nicodjimenez 8y ago

Question for the Discord team.. if you started this project today in 2019, would you still build a homegrown event sourcing system? Or would you use Kafka?

jhgg 8y ago
We would still use our own homegrown system. And we do use kafka internally already for a plethora of other things. Just not on the real-time chat side of things.
- nicodjimenez 8y ago
  Why not on the real time chat stuff?
  - jhgg 8y ago
    Because Kafka is definitely not a tool suited for real time chat and event distribution to millions of clients. Once a message hits our distribution it’s fanned our to clients on average 5-10ms later.

JulianMorrison 8y ago

Needs some attention to the scaling down, IMO. Discord is annoying on intermittent connections. Why do sent messages not have identity such that they can go through twice?

jhgg 8y ago
We actually added this 2nd half of last year. Message send operations are now deduped and idempotent given a client-generated nonce. Are you still experiencing this issue?
- ubercow13 8y ago
  Yes I experience this constantly, very recently, on poor internet connections
- JulianMorrison 8y ago
  Your nonce might not survive the "message -> disconnect -> red message -> re-send" flow. Could you just get rid of red messages and make it "pending" until acked as "sent"?

jplayer01 8y ago

So why can't they add basic features like organizing servers?

pdimitar 8y ago

A corollary: As a mostly Elixir programmer for 2 years now, I find it quite amusing how the Kubernetes community tries their damnest to emulate Erlang's OTP system -- going as far as to bolt stricter typing system on Golang even -- and try and reinvent Erlang's "let it crash and get rebooted" idea. I am however not at all convinced that "let it crash" is a good mantra when applied to entire containers. One such container can take 20+ seconds to restart and you can lose a lot more compared to the mini-processes Erlang/Elixir have which are happily left to crash and [semi-]auto-recover. Imagine if that container was ingesting events and crashed when it had 5_000 in its in-memory queue and only managed to process 50-100 of them. Or imagine 100 transactions in progress being cancelled. I applaud the hard work of the Kubernetes team, I am just not very sure they invest their energy in the right tech stack. Sure Golang is faster than Erlang/Elixir -- by a lot, too. But is it not K8s idea to be fault-tolerant and not immensely quick? What does a Golang's speed matter when a container needs seconds to reboot? In my eyes, K8s could have been written in Bash shell scripts... but I am probably missing something important.

Thaxll 8y ago
Kubernetes is based on Google Borg / Omega not OTP very different things. As for your queue / DB question there is something called transactions for that kind of problem, I hope you don't use OTP features replacing transactions or you're looking for very serious issues. OTP doesn't do 5% of what Kubernetes is able to provide ( cpu / io / memory quota, live / readiness probes, rolling deployments / canary / blue green.
- pdimitar 8y ago
  I knew somebody will get my random examples and turn them against me. :D Serves me right for not researching several hours to find the perfect example, I guess! I am well aware of the differences in scope between K8s and Erlang/OTP. My point was that they kind of try to do the same only on different levels. And I am still not sold on the idea of throwaway containers. That works well if you have a hyper-network of microservices that are able to discover each other and self-heal a bigger graph of services but if you have any sort of a more classic coherent whole app... then not really. Throwing containers away and rebooting new copies isn't something that looks viable for many projects. But hell, who knows. K8s team and their audience are really dedicated. They might change every single tool of the trade just to make K8s work well. I was mostly saying that I don't feel it brings something really radical to the table.