20 ms·

How Discord Scaled Elixir to 5M Concurrent Users

802 points by b1naryth1ef 9y ago

brian_herman 9y ago
I love discord's posts they are very informative and easy to read.
_ar7 9y ago
Really liked the blog post. Elixir and the capabilities of the BEAM VM seems really awesome, but I can't really find an excuse to really use them in my day to day anywhere.
brightball 9y ago
I so appreciate write ups that get into details of microsecond size performance gains at that scale. It's a huge help for the community.
- _asummers 9y ago
  Erlang and Elixir measure in microseconds, so it'd be them throwing away information if they did otherwise!
rdtsc 9y ago
Good stuff. Erlang VM FTW! > mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node. Jesper L. Andersen (jlouis) talked about it in his blog: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimization-efde8b20cba7 https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio... > After some research we stumbled upon :ets.update_counter/4 Might not help in this case but 20.0 adds select_replace so can do a full on CAS (compare and exchange) pattern http://erlang.org/doc/man/ets.html#select_replace-2 http://erlang.org/doc/man/ets.html#select_replace-2 . So something like acquiring a lock would be much easier to do. > We found that the wall clock time of a single send/2 call could range from 30μs to 70us due to Erlang de-scheduling the calling process. There are few tricks the VM uses there and it's pretty configurable. For example sending to a process with a long message queue will add a bit of a backpressure to the sender and un-schedule them. There are tons of configuration settings for the scheduler. There is to bind scheduler to physical cores to reduce the chance of scheduler threads jumping around between cores: http://erlang.org/doc/man/erl.html#+sbt http://erlang.org/doc/man/erl.html#+sbt Sometimes it helps sometimes it doesn't. Another general trick is to build the VM with the lcnt feature. This will add performance counters for locks / semaphores in the VM. So then can check for the hotspots and know where to optimize: http://erlang.org/doc/man/lcnt.html http://erlang.org/doc/man/lcnt.html
- cschneid 9y ago
  Are there good resources / books on advanced BEAM? I'd love to know more of the nitty-gritty details of the VM, and how to make the best use of it.
  - mononcqc 9y ago
    https://github.com/happi/theBeamBook https://github.com/happi/theBeamBook for the VM http://erlang-in-anger.com/ http://erlang-in-anger.com/ for introspecting it in production
- pmarreck 9y ago
  I would like to subscribe to your newsletter Seriously, though, I might need your services in the future, if you're available. Also, I'm pretty sure the security of your career is pretty guaranteed at this point lol (looking at Indeed data, interest in Elixir has risen 20fold in the past 3 years and the slope of that line is far steeper than all other languages in that space)
  - rdtsc 9y ago
    I don't have a newsletter. I thought of having a blog for a bunch of stuff like this but I like writing software more than blog posts and I knew it'd abandon it after a while. Maybe there is "turn all your hn posts higher than X upvotes into a blog post series" script somewhere. However you can subscribe to the Erlang mailing list. That's often a good place to start for tricky or interesting questions you might have: http://erlang.org/mailman/listinfo/erlang-questions http://erlang.org/mailman/listinfo/erlang-questions To contact me directly check my profile. Cheers!
- jlouis 9y ago
  It isn't that likely the OTP20 optimization helps here. If the process never sends a message containing the literal value, then there is no benefit in the optimization. What `mochiglobal` and friends are good at is when you have a large set of data (A ring, say) which update rarely, so you can treat it as semi-static data in the system. But then you shouldn't really send that ring data around in the system too much, although it will now be free. [There is a nice subscription-based approach to updates which are now feasible in OTP20, but that is more for convenience] if send/2 takes 30us to 70us, I'm guessing blocking as well, either on distributed communication or something else along those lines. For local message passes to take that long, my something-is-amiss-sixth-sense is tingling.
myth_drannon 9y ago
It's interesting how on StackOverflow Jobs Elixir knowledge is required more often than Erlang. http://www.reallyhyped.com/?keywords=erlang%2Celixir http://www.reallyhyped.com/?keywords=erlang%2Celixir
- cutler 9y ago
  Despite its appeal to HN geeks I doubt if Elixir will ever achieve mainstream adoption. Searching Indeed.co.uk's API by title, there are only 5 Elixir jobs in London, compared with 445 Python and 171 Ruby. I also attended a Silicon Roundabout jobs fair recently and was disappointed to find Elixir wasn't even listed in the literature.
  - itbeho 9y ago
    It's still early. I've bet on the wrong horse (stack) before, so take my comments with a grain of salt, but I know several companies that are currently adopting Elixir/Phoenix with the same level of excitement that I recall from the early days of Ruby/Rails. It may never be "Rails big", but there is definitely some momentum building.
    - bigtunacan 9y ago
      When Rails arrived on the scene it was very different from everything else, but also there weren't 1000 new languages/ frameworks popping up all at once. I'm not implying in anyway that Elixir is bad. I just think there are too many horses these days to know which to bet on. Elixir? Node? Go? Rust? Something else?
      - cutler 9y ago
        Exactly. If Elixir didn't have to compete with Clojure, Rust, Go, Node, Scala and Kotlin it might have had a chance to become mainstream. Beyond the hype job stats are the ultimate indicator.
      - vertex-four 9y ago
mbesto 9y ago
This is one of those few instances where getting the technology choice right actually has an impact on cost of operations, service reliability, and overall experience of a product. For like 80% of all the other cases, it doesn't matter what you use as long as your devs are comfortable with it.
- kornish 9y ago
  Not sure why this comment saw a couple downvotes earlier. mbesto is correct: for most startups, most of the time, competitive advantage doesn't come from the underlying tech stack. To make a general statement, most things could be done similarly on any of several platforms. However, when product requirements match exceptionally well with a specialized technology, you can see things that would simply be infeasible or extremely tough using a different stack. WhatsApp + Erlang was one of those cases (watch this talk and imagine trying to recreate that system with only a handful of server engineers using any other tech: https://www.youtube.com/watch?v=c12cYAUTXXs https://www.youtube.com/watch?v=c12cYAUTXXs). Discord + Elixir appears to be another. Curious if anyone has any examples that spring to mind from outside the highly concurrent messaging space.
ShaneWilton 9y ago
Thanks for putting this writeup together! I use Elixir and Erlang every day at work, and the Discord blog has been incredibly useful in terms of pointing me towards the right tooling when I run into a weird performance bottleneck. FastGlobal in particular looks like it nicely solves a problem I've manually had to work around in the past. I'll probably be pulling that into our codebase soon.
- pmarreck 9y ago
  Note that Erlang 20 may have solved the problem that FastGlobal tries to fix (that of not copying large amounts of data unnecessarily)
  - deleted 9y ago
    [deleted]
  - deleted 9y ago
    [deleted]
  - ShaneWilton 9y ago
    Erlang 20 fixes the case where you're copying a constant literal, but unfortunately won't help if you're sharing a dynamically generated, but infrequently modified, term; like Discord does in this post.
danso 9y ago
According to Wikipedia, Discord's initial release was March 2015. Elixir hit 1.0 in September 2014 [0]. That's impressively early for adoption of a language for prototyping and for production. [0] https://github.com/elixir-lang/elixir/releases/tag/v1.0.0 https://github.com/elixir-lang/elixir/releases/tag/v1.0.0
- quaunaut 9y ago
  If I remember right, they were using Erlang at the beginning, and moved slowly to Elixir as they got more comfortable, and the ecosystem built up around them.
  - Vishnevskiy 9y ago
    We started with Elixir, some of our engineers have used Erlang prior though.
jmcgough 9y ago
Great to see more posts like this promoting Elixir. I've been really enjoying the language and how much power it gets from BEAM. Hopefully more companies see success stories like this and take the plunge - I'm working on an Elixir project right now at my startup and am loving it.
orliesaurus 9y ago
Unlike Discord's design team who seem to just copy all of Slack's designs and assets, the Engineering team seems to have their shit together, it is delightful to read your Elixir blogposts. Good job!
- orliesaurus 9y ago
  yeah go on with the downvotes, you know everyone's thinking what I said! Copying your way to success is a strategy and it's not the first time either way :)
joonoro 9y ago
Elixir was one of the reasons I started using Discord in the first place. I figured if they were smart enough to use Elixir for a program like this then they would probably have a bright future ahead of them. In practice, Discord hasn't been completely reliable for my group. Lately messages have been dropping out or being sent multiple times. Voice gets messed up (robot voice) at least a couple times per week and we have to switch servers to make it work again. A few times a person's voice connection has stopped working completely for several minutes and there's nothing we can do about it. I don't know if these problems have anything to do with the Elixir backend or the server. EDIT: Grammar
- Vishnevskiy 9y ago
  The messages struggles have been sadly due to issues with Cassandra and GC pauses caused by bugs within it. We have been trying to work with the Cassandra developers to resolve these. Voice issues should not be happening. Please contact our support with more information and we will gladly investigate.
  - joonoro 9y ago
    Thanks for the response, it's good to know what's causing the problems with messages and that it's being worked on. I'll try to contact support next time I have voice issues with my group.
    - Vishnevskiy 9y ago
      :) np We love Cassandra, and hate it at the same time. Check out this nasty bug we got. https://issues.apache.org/jira/browse/CASSANDRA-13004 https://issues.apache.org/jira/browse/CASSANDRA-13004
      - jjirsa 9y ago
        Two things are true about Cassandra: It is by far the best at doing what it does It has plenty of room for improvement
zitterbewegung 9y ago
Compared to slack discord is a much better service for large groups . Facebook uses them for react.
- btown 9y ago
  I just wish they were more professional in their communications. Service went down hard this weekend and their response was a cat GIF. Hard to justify to anyone in business that they should take it seriously. A shame when their tech is such a great match for professional collaboration, and all it would take to grow it organically would be less marketing copy not more.
  - notnight 9y ago
    We take outages seriously internally, but users generally don't care about the nitty-gritty. For a full, detailed postmortem on that outage you can check out our status site: https://status.discordapp.com/incidents/ywdwttd6b0hg https://status.discordapp.com/incidents/ywdwttd6b0hg
jakebasile 9y ago
I'm continually impressed with Discord and their technical blogs contribute to my respect for them. I use it in both my personal life (I run a small server for online friends, plus large game centric servers) and my professional life (instead of Slack). It's a delight to use, the voice chat is extremely high quality, text chat is fast and searchable, and notifications actually work. Discord has become the de facto place for many gaming communities to organize which is a big deal considering how discriminating and exacting PC gamers can be. My only concern is their long term viability and I don't just mean money wise. I'm concerned they'll have to sacrifice the user experience to either achieve sustainability or consent to a buyout by a larger company that only wants the users and brand. I hope I'm wrong, and I bought a year of Nitro to do my part.
- lordCarbonFiber 9y ago
  A closed source walled garden chat service that survives purely on the free flow of VC capital 100% has no long term viability. No federation means as soon as the "next thing" shows up and wins way the VC dollars they'll disappear. It's so incredibly frusterating that so many companies and users make/support these closed environments even as we enter a new golden age of open sourced and federated technologies.
  - jakebasile 9y ago
    If there was an open source and federated equivalent to the features Discord provides I'd use it. There is no such product. Matrix is interesting, but the experience is no where near as polished as Discord and friends and that matters for mass adoption.
    - aphextron 9y ago
      >If there was an open source and federated equivalent to the features Discord provides I'd use it. But will you pay for Discord though? The features and quality Discord is able to provide are artificially propped up by VC funding. When it runs dry, we will be left with open source offerings, or the next product to take it's place and repeat the cycle.
jaequery 9y ago
Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror gem? I see Phoenix has a built-in error stack trace page which looks like a clone of bettererror but it doesn't have the real-time console inside of it. Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?
- themgt 9y ago
  Just use IEx.pry if you want a debug console.
- kornish 9y ago
  Curious about your thoughts on Ecto: https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto. Seems to be the most popular solution for database access.
- lojack 9y ago
  > Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this? May be a matter of semantics, but the concept of ORM simply can't exist in a functional language as there aren't objects. That said, Ecto is pretty much the go-to in Elixir. It's extremely powerful, and in my opinion provides the right amount of abstraction without going too far.
- cschneid 9y ago
  > like Sequel Ecto can do a reasonable impression with its DSL & direct use of tables. You don't need to go through the whole stack of schemas and such. What in particular do you like about Sequel (over what Ecto is providing)?
- _asummers 9y ago
  Ecto is hands down the best "ORM" I've used. At no point has it ever gotten in the way of us dynamically creating queries on the fly, just following the Ecto internals (not recommended, but totally doable) and has an extremely expressive syntax. The team behind it has been rolling out new features and are extremely responsive to requests on the mailing list. I encourage you to give it a chance. The mental shift is that "schemas" are not objects in an ORM sense, but instead are just data, or views over data. The functions come from taking in data or a Changeset and manipulating those, versus calling a function on a class. As far as errors, Elixir 1.4.5 has much better error messages, specifically printing the args to crashes and such, and OTP 20/ Elixir 1.5 should drastically improve Dialyzer error messages. I am not a Ruby guy, so perhaps you can say what you feel is missing from Elixir's messages and how the Ruby gem improves it. Also you can literally inspect running state on the fly with the BEAM tooling, so the need for things like debuggers goes down.
iagooar 9y ago
This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites. Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level. Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions. I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!
- toolz 9y ago
  I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.
  - rubyn00bie 9y ago
    Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
    - _asummers 9y ago
      We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).
Cieplak 9y ago
I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point, but perhaps there's an element of "less is more" going on here. Also, we're looking for Erlang folks with payments experience. cGF0cmljaytobkBmaW5peHBheW1lbnRzLmNvbQ==
- kevincennis 9y ago
  Off topic, but what's the story behind base-64 encoding your email? Is that just a spam prevention measure?
  - Cieplak 9y ago
    Yes
    - Corrado 9y ago
      I actually really like this idea and am going to steal it. :)
  - Deinumite 9y ago
    I think it is also sometimes a simple barrier to entry to weed out some candidates (think of it as a super simple fizzbuzz). I also see a lot of job postings asking people to decode some string for bonus "points"... 90% of the time its just rot13.
- pencilhappen 9y ago
  > Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point It's not really a superset. The JVM has a single global heap, whereas BEAM has per-process heaps. Global state is the enemy of concurrency, and the best GC algorithm is one you don't need to run at all. The "several hundred millisecond HTTP response latency and horde (sic) memory" problems you note are in part a result of having a global heap.
ConanRus 9y ago
I do not see there any Elixir specific, it is all basically Erlang/Erlang VM/OTP stuff. When you using Erlang, you think in terms of actors/processes and message passing, and this is (IMHO) a natural way of thinking about distributed systems. So this article is a perfect example how simple solutions can solve scalability issues if you're using right platform for that.
- nichochar 9y ago
  You're right. Elixir doesn't pretend to do anything except make using the wonderful erlang VM/OTP stuff easier. The VM is an absolute marvel of engineering, and it's insane to me that it doesn't have more adoption yet in big tech companies. My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again.
  - pmarreck 9y ago
    > and it's insane to me that it doesn't have more adoption yet in big tech companies. elixir (and winter) is coming > My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again. you conflated "top" with "large".
  - Anderkent 9y ago
    Coordination problems get more difficult in larger teams/companies. Getting everyone to use a particular non-standard language is a coordination problem. Thus large companies are unlikely to experiment with languages. It makes sense too - say it's 10x easier to write something in language X than Y. If there's only 10 other people that might interact with the thing / have to read the sources, that's a great tradeoff. If there's a thousand other people that might have to at some point understand how some part of your code works, suddenly all of them have to learn the new language X.
marlokk 9y ago
"How Discord Scaled Elixir to 5M Concurrent Users" click link [Error 504 Gateway time-out] only on Hacker News
didibus 9y ago
So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only. That's why I'd like to hear more about productivity and ease now. Is it faster and more fun to scale things in certain languages then others. Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and procudctive way to scale, that doesn't matter. Anyways, learning how team scaled is interesting, but it's clear to me now languages aren't limiting factors to scale.
- YorickPeterse 9y ago
  While you don't need language X to scale, certain languages can definitely make it _easier_ and more cost-effective to scale. So it can matter depending on what you're trying to achieve.
- nateberkopec 9y ago
  If you have a share-nothing architecture, then yes, any language can scale to any load, some with more hardware than others.
- dmix 9y ago
  > So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Just like any language vs language debate each one has benefits for various particular use-cases. Any meaningful comparison of languages must be prefaced with the use-case scenario. One of the strongest use-cases of Erlang/Elixir has always been building large distributed apps that need to scale (async web apps, telecom, chat servers, messaging mobile apps, etc). The ability to build these large distributed systems are baked into the very primitive parts of the language and standard library - to a degree that few other languages can compare to it, if any. With Erlang/Elixir you design ALL applications in a way where scaling is rarely an after thought but rather a natural extension of the program. > Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and productive way to scale, that doesn't matter. People often make the mistake of trivializing Erlang/Elixirs as merely programming with actors. It's development not only predated the actor model but it also goes well beyond that to being the standard programming style you use when developing any program when using the language - the same way Rails embraces MVP. When this is fundamental part of every Erlang application then the means of scaling to a large distributed system are also a fundamental part of each program. This built-in scaling is gained without any significant costs in terms of development time but also provides many benefits beyond scaling, such as highly modular and extensible code. There are real benefits even if you don't plan to scale to a large distributed system. similar to Rails it creates a predictable program design which makes joining new projects easier and deters NIH syndrome that is far too common in Java/C++/etc. And ultimately, regardless of what you are building, it provides very high performance by default for the type of async style applications that are popular on the web today. So the key point here is not that the end goal was achieved (that you can scale) but how you get there.
framp 9y ago
Really lovely post! I wonder how Cloud Haskell would fare in such a scenario
khanan 9y ago
Problem is that Discord sucks since it does not have a dedicated server. Sorry, move along.
- vultour 9y ago
  That's actually why it doesn't suck for the vast majority. Not everyone wants to pay $ every month so they could have their own voice / chat server.
alberth 9y ago
Is there any update on BEAMJIT? It was super promising 3 or so years ago. But I haven't seen an update. Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that. https://www.sics.se/projects/beamjit https://www.sics.se/projects/beamjit
- jlouis 9y ago
  Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).
  - alberth 9y ago
    Amy ETA on when we can start using beamjit?
    - jlouis 9y ago
      Given that it has been postponed a couple of times, no. JITs are hard to pull off and it will probably have a period of worse stability as well before it matures. Another problem is getting a JIT to be faster than the interpreter. Erlang's BEAM is threaded code and also macro-instructions, so it almost looks like a JIT internally. The big gains would be in inlining across module boundaries and type speculation. But I hold that if we could compile bundles of modules in HiPE, we would have the same gain for a fraction of the development and maintenance effort. The biggest lure of native code generation would be that we could get rid of a lot of C code in the system as the native cogen would be able to rival the C code in speed. Many Erlang programs spend shockingly little time in the emulator loop, especially if they are communication heavy. If you need speed today, don't underestimate a port-program. My test is that you can pipeline about a million requests back and forth to an OCaml program per second per core. So if your work is on the order of 1+ milliseconds, this is usually a feasible strategy. Espcially because OS isolation means you can handle exceptions in the OCaml program from the Erlang side by restarting the port.
      - rozap 9y ago
jlouis 9y ago
A fun idea is to do away with the "guild" servers in the architecture and simply run message passes from the websocket process over the Manifold system. A little bit of ETS work should make this doable and now an eager sending process is paying for the work itself, slowing it down. This is exactly the behavior you want. If you are bit more sinister you also format most of the message in the sending process and makes it into a binary. This ensures data is passed by reference and not copied in the system. It ought to bring message sends down to about funcall overhead if done right. It is probably not a solution for current Discord as they rely on linearizability, but I toyed with building an IRCd in Erlang years ago, and there we managed to avoid having a process per channel in the system via the above trick. As for the "hoops you have to jump through", it is usually true in any language. When a system experiences pressure, how easy it is to deal with that pressure is usually what matters. Other languages are "phase shifts" and while certain things become simpler in that language, other things become much harder to pull off.
- mononcqc 9y ago
  The true evil approach is to send the socket around, not the message, so that there is no copying required no matter what ;)
  - rdtsc 9y ago
    Wah. Easy there, Satan :-) That is cool trick though. So it's basically sending the port itself around and changing its ownership, with something like port_connect(Port,NewOwner)? And btw, thank you for writing https://www.erlang-in-anger.com https://www.erlang-in-anger.com and http://learnyousomeerlang.com http://learnyousomeerlang.com !
    - mononcqc 9y ago
      The trick is more commonly used when writing to sockets. A socket owner is required for reading, not for writing. The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly). Also recently I've gotten http://propertesting.com/ http://propertesting.com/ out, you might enjoy it :)
ramchip 9y ago
Very interesting article! One thing I'm curious about is how to ensure a given guild's process only runs on one node at a time, and the ring is consistent between nodes. Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?
- jhgg 9y ago
  We use etcd.
deleted 9y ago
[deleted]
marcgcombi 9y ago
Looks like eSpam promotion?
neya 9y ago
Hi community, Let me share my experience with you. I'm a hardcore Rails guy and I've been advocating and teaching Rails to the community for years. My workflow for trying out a new language involves using the language for a small side project and gradually would try to scale it up. So, here's my summary, my experience of all the languages so far: Scala - It's a vast academic language (official book is with ~700 pages) with multiple ways of doing things and it's attractiveness for me was the JVM. It's proven, robust and highly scalable. However, the language was not quite easy to understand and the frameworks that I've tried (Play 2, Lift) weren't as easy to transition to, for a Rails developer like me. Nevertheless, I did build a simple calendar application, but it took me 2 months to learn the language and build it. GoLang - This was my next bet, although I didn't give up on Scala completely (I know it has its uses), I wanted something simple. I used Go and had the same experience as I had when I used C++. It's a fine language, but, for a simple language, I had to fight a lot with configuration to get it working for me - (For example, it has this crazy concept of GOPATH where your project should reside and if your project isn't there it'll keep complaining). Nevertheless, I build my own (simple) Rails clone in GO and realized this isn't what I was looking for. It took my about a month to conquer the language and build my (simple) side project. Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and decided to give it a go. I started off with Phoenix. The transition was definitely wayy smoother from Rails, especially considering the founding member of this language was a Rails dev. himself (the author of "devise" gem). At first some concepts seemed different (like piping), but once I got used to it, for me there was no looking back. All was fine until they released Phoenix 1.3, where they introduced the concept of contexts and (re) introduced Umbrella applications. Basically they encourage you to break your application into smaller applications by business function (similar to microservices) except that you can do this however you like (unopinionated). For example, I broke down my application by business units (Finance, Marketing, etc.). This forced me to re-think my application in a way I never would have thought and by this time I had finished reading all 3 popular books on this topic (Domain Driven Design). I loved how the fact that Elixir's design choices are really well suited for DDD. If you're new to DDD I suggest you try giving it a shot, it really can force you to re-think the way you develop software. By the end of two weeks after being introduced to Elixir, I picked up the language. In a month and a half, I built a complete Salesforce clone just working on the weekends. And this includes even the UI. And I love how my application is always blazing fast, picks up errors even before it compiles and warns me if I'm no using a variable I defined somewhere. P.S there IS a small learning curve involved if you're starting out fresh: 1) IF you're used to the Rails asset pipeline, you'll need to learn some new tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD (optional) if you want to better architect your application. 3) There is no return statement in Elixir! As a Ruby developer, here are my thoughts: 1. So, will I be developing with Rails again? Probably yes, for simpler applications / API servers. 2. Is Ruby dying? No. In fact, I can't wait for Ruby 3. Some drawbacks of Elixir: 1. Relatively new, so sometimes you'll be on your own and that's okay. 2. Fewer libraries as compared to the Ruby eco-system. But you can easily write your own. 3. Fewer developers, but should be fairly to onboard Ruby developers. Cheers.
omeid2 9y ago
I think while this is great, it is good to remember that your current tech stack maybe just fine! after all, Discord start with mongodb[0]. [1]. https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7 https://blog.discordapp.com/how-discord-stores-billions-of-m...
grantwu 9y ago
"Discord clients depend on linearizability of events" Could this be possibly be the cause of the message reordering and dropping that I experience when I'm on a spotty connection?
sriram_malhar 9y ago
I really like elxir the language, but find myself strangely hamstrung by the _mix_ tool. There is only an introduction to the tool, but not a reference to all the bells and whistles of the tool. I'm not looking for extra bells and whistles, but simple stuff like pulling in a module from GitHub and incorporate it. Is there such documentation? How do you crack Mix?
- amorphid 9y ago
  The docs for Mix are decent. You can start here: https://hexdocs.pm/mix https://hexdocs.pm/mix. When you are trying to get help w/ a specific task, you check checkout the mix tasks docs: https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content And honestly, I often just look at the source code: https://github.com/elixir-lang/elixir/tree/master/lib/mix https://github.com/elixir-lang/elixir/tree/master/lib/mix The Elixir Slack channel is pretty amazing, too: https://elixir-slackin.herokuapp.com/ https://elixir-slackin.herokuapp.com/
andy_ppp 9y ago
Just as an aside how would people build something like this if they were to use say Python and try to scale to these sort of user levels? Has anyone succeeded? I'd say it would be quite a struggle without some seriously clever work!
- MusaTheRedGuard 9y ago
  Yeah pretty much impossible with python.
KrishnaHarish 9y ago
Scale!
KrishnaHarish 9y ago
What is Discord and Elixir?
majidazimi 9y ago
It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime? As soon as you introduce standard message format, then all nice features such as built-in distribution, automatic reconnect, ... are almost useless. You have to do all these manually. May be I'm missing something. Correct me if I'm wrong. For a fast time to market it seems quite nice approach. But for a long running maintainable back-end it not enough.
- ramchip 9y ago
  Distributed Erlang compatibility is guaranteed for not just one, but two major versions. You can do a standard rolling upgrade and the newer nodes will talk to older nodes just fine. http://erlang.org/pipermail/erlang-questions/2011-June/059463.html http://erlang.org/pipermail/erlang-questions/2011-June/05946... Foreign nodes in C, Java, Python, etc. can also join an Erlang cluster: http://erlang.org/doc/tutorial/cnode.html http://erlang.org/doc/tutorial/cnode.html http://erlang.org/doc/apps/jinterface/jinterface_users_guide.html http://erlang.org/doc/apps/jinterface/jinterface_users_guide... That being said, a more typical architecture is to have Erlang spawn external processes and talk to them via stdio.
  - majidazimi 9y ago
    It's great that they support C and JVM stack. But for other major platforms such as NodeJS, Go, ... we still need to approach the goal with some C extensions integrated in both platforms. Erlang/OTP distribution/clustering is really not designed for heterogeneous environment which is fine, since it is used in telecom industry with nice commercial support contracts backed by Ericsson. However I don't consider it as an amazing piece of engineering from a perspective of a system designer who is dealing with many teams with different language/stack preferences.
    - ramchip 9y ago
      Erlang is what you would use to implement a single service (which may run on a cluster of multiple nodes). It's not a generic messaging layer to use between services, you can use RabbitMQ or ZeroMQ or whatever you like for that.
OOPMan 9y ago
5 million concurrent users is great and all, but it would be nice if Discord could work out how to use WebSockets without duplicating sent messages. This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around). It's terribly annoying though and makes using the app via the mobile client to be very tedious.
- k__ 9y ago
  Coming from IRC with netsplits and stuff, this seems like a first world problem to me hehe
  - OOPMan 9y ago
    You know there are discord bots for bridging between IRC and Discord? Pretty funky :-)
StreamBright 9y ago
Whatsapp's story is somewhat similar. Relevant read to this subject. http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2...
dandare 9y ago
What is the business model behind Discord? They boast about being free multiple times, how do they make money? Or plan to make money?
- satsuma 9y ago
  They currently have a Nitro service that's $5/month for little features, but I don't know how far that takes them towards profitability.
deleted 9y ago
[deleted]
agentgt 9y ago
I realize this is off topic but how does Discord make money? I can't figure out their biz model (I'm not a gamer so I didn't even know about them).
renaudg 9y ago
It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs. If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io https://socketcluster.io doesn't provide ?
- lostcolony 9y ago
  Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io doesn't serve over HTTPS) But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone"). Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that. So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.
  - jondubois 9y ago
    Hi, main author of SocketCluster here. SC does support automatic horizontal scaling across any number of machines out of the box if you're running it on Kubernetes. There's also a CLI tool to deploy it automatically to any Kubernetes cluster: https://www.npmjs.com/package/baasil https://www.npmjs.com/package/baasil See https://github.com/SocketCluster/socketcluster/blob/master/scc-guide.md#scc-guide https://github.com/SocketCluster/socketcluster/blob/master/s...
    - lostcolony 9y ago
etblg 9y ago
Reading posts like this about widely distributed applications always gets me interested in it as a career path. Currently I'm working as a front-end dev with moderate non-distributed back-end experience. How would someone in my situation, with no distributed back-end experience, break in to a position working on something like Discord?
concatime 9y ago
Sad to see some people taking raw and insignificant benchmarks to evaluate a language[0]. [0] https://news.ycombinator.com/item?id=14479757 https://news.ycombinator.com/item?id=14479757
oldpond 9y ago
When have you ever read, "How Acme scaled J2EE to 5M Concurrent Users"? I became an IT architect in 1998 at IBM, the year Sun released j2ee and IBM released Websphere. I have experienced 20 years of enterprise Java and object oriented computing, and I was thrilled when Elixir came out. I was a mainframe programmer before OO became all the rage, so I never really felt at home doing objects. Functional programming feels completely natural to me though. What I like about this article is that they shared everything they learned with the community. Thank you for that excellent experience report.

802 points by b1naryth1ef 9y ago

I love discord's posts they are very informative and easy to read.

_ar7 9y ago

Really liked the blog post. Elixir and the capabilities of the BEAM VM seems really awesome, but I can't really find an excuse to really use them in my day to day anywhere.

brightball 9y ago

I so appreciate write ups that get into details of microsecond size performance gains at that scale. It's a huge help for the community.

_asummers 9y ago
Erlang and Elixir measure in microseconds, so it'd be them throwing away information if they did otherwise!

rdtsc 9y ago

Good stuff. Erlang VM FTW! > mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data There is a nice new OTP 20.0 optimization - now the value doesn't get copied even on message sends on the local node. Jesper L. Andersen (jlouis) talked about it in his blog: https://medium.com/@jlouis666/an-erlang-otp-20-0-optimization-efde8b20cba7 https://medium.com/@jlouis666/an-erlang-otp-20-0-optimizatio... > After some research we stumbled upon :ets.update_counter/4 Might not help in this case but 20.0 adds select_replace so can do a full on CAS (compare and exchange) pattern http://erlang.org/doc/man/ets.html#select_replace-2 http://erlang.org/doc/man/ets.html#select_replace-2 . So something like acquiring a lock would be much easier to do. > We found that the wall clock time of a single send/2 call could range from 30μs to 70us due to Erlang de-scheduling the calling process. There are few tricks the VM uses there and it's pretty configurable. For example sending to a process with a long message queue will add a bit of a backpressure to the sender and un-schedule them. There are tons of configuration settings for the scheduler. There is to bind scheduler to physical cores to reduce the chance of scheduler threads jumping around between cores: http://erlang.org/doc/man/erl.html#+sbt http://erlang.org/doc/man/erl.html#+sbt Sometimes it helps sometimes it doesn't. Another general trick is to build the VM with the lcnt feature. This will add performance counters for locks / semaphores in the VM. So then can check for the hotspots and know where to optimize: http://erlang.org/doc/man/lcnt.html http://erlang.org/doc/man/lcnt.html

cschneid 9y ago
Are there good resources / books on advanced BEAM? I'd love to know more of the nitty-gritty details of the VM, and how to make the best use of it.
- mononcqc 9y ago
  https://github.com/happi/theBeamBook https://github.com/happi/theBeamBook for the VM http://erlang-in-anger.com/ http://erlang-in-anger.com/ for introspecting it in production
pmarreck 9y ago
I would like to subscribe to your newsletter Seriously, though, I might need your services in the future, if you're available. Also, I'm pretty sure the security of your career is pretty guaranteed at this point lol (looking at Indeed data, interest in Elixir has risen 20fold in the past 3 years and the slope of that line is far steeper than all other languages in that space)
- rdtsc 9y ago
  I don't have a newsletter. I thought of having a blog for a bunch of stuff like this but I like writing software more than blog posts and I knew it'd abandon it after a while. Maybe there is "turn all your hn posts higher than X upvotes into a blog post series" script somewhere. However you can subscribe to the Erlang mailing list. That's often a good place to start for tricky or interesting questions you might have: http://erlang.org/mailman/listinfo/erlang-questions http://erlang.org/mailman/listinfo/erlang-questions To contact me directly check my profile. Cheers!
jlouis 9y ago
It isn't that likely the OTP20 optimization helps here. If the process never sends a message containing the literal value, then there is no benefit in the optimization. What `mochiglobal` and friends are good at is when you have a large set of data (A ring, say) which update rarely, so you can treat it as semi-static data in the system. But then you shouldn't really send that ring data around in the system too much, although it will now be free. [There is a nice subscription-based approach to updates which are now feasible in OTP20, but that is more for convenience] if send/2 takes 30us to 70us, I'm guessing blocking as well, either on distributed communication or something else along those lines. For local message passes to take that long, my something-is-amiss-sixth-sense is tingling.

myth_drannon 9y ago

It's interesting how on StackOverflow Jobs Elixir knowledge is required more often than Erlang. http://www.reallyhyped.com/?keywords=erlang%2Celixir http://www.reallyhyped.com/?keywords=erlang%2Celixir

cutler 9y ago
Despite its appeal to HN geeks I doubt if Elixir will ever achieve mainstream adoption. Searching Indeed.co.uk's API by title, there are only 5 Elixir jobs in London, compared with 445 Python and 171 Ruby. I also attended a Silicon Roundabout jobs fair recently and was disappointed to find Elixir wasn't even listed in the literature.
- itbeho 9y ago
  It's still early. I've bet on the wrong horse (stack) before, so take my comments with a grain of salt, but I know several companies that are currently adopting Elixir/Phoenix with the same level of excitement that I recall from the early days of Ruby/Rails. It may never be "Rails big", but there is definitely some momentum building.
  - bigtunacan 9y ago
    When Rails arrived on the scene it was very different from everything else, but also there weren't 1000 new languages/ frameworks popping up all at once. I'm not implying in anyway that Elixir is bad. I just think there are too many horses these days to know which to bet on. Elixir? Node? Go? Rust? Something else?
    - cutler 9y ago
      Exactly. If Elixir didn't have to compete with Clojure, Rust, Go, Node, Scala and Kotlin it might have had a chance to become mainstream. Beyond the hype job stats are the ultimate indicator.
    - vertex-four 9y ago

mbesto 9y ago

This is one of those few instances where getting the technology choice right actually has an impact on cost of operations, service reliability, and overall experience of a product. For like 80% of all the other cases, it doesn't matter what you use as long as your devs are comfortable with it.

kornish 9y ago
Not sure why this comment saw a couple downvotes earlier. mbesto is correct: for most startups, most of the time, competitive advantage doesn't come from the underlying tech stack. To make a general statement, most things could be done similarly on any of several platforms. However, when product requirements match exceptionally well with a specialized technology, you can see things that would simply be infeasible or extremely tough using a different stack. WhatsApp + Erlang was one of those cases (watch this talk and imagine trying to recreate that system with only a handful of server engineers using any other tech: https://www.youtube.com/watch?v=c12cYAUTXXs https://www.youtube.com/watch?v=c12cYAUTXXs). Discord + Elixir appears to be another. Curious if anyone has any examples that spring to mind from outside the highly concurrent messaging space.

ShaneWilton 9y ago

Thanks for putting this writeup together! I use Elixir and Erlang every day at work, and the Discord blog has been incredibly useful in terms of pointing me towards the right tooling when I run into a weird performance bottleneck. FastGlobal in particular looks like it nicely solves a problem I've manually had to work around in the past. I'll probably be pulling that into our codebase soon.

pmarreck 9y ago
Note that Erlang 20 may have solved the problem that FastGlobal tries to fix (that of not copying large amounts of data unnecessarily)
- deleted 9y ago
  [deleted]
- deleted 9y ago
  [deleted]
- ShaneWilton 9y ago
  Erlang 20 fixes the case where you're copying a constant literal, but unfortunately won't help if you're sharing a dynamically generated, but infrequently modified, term; like Discord does in this post.

danso 9y ago

According to Wikipedia, Discord's initial release was March 2015. Elixir hit 1.0 in September 2014 [0]. That's impressively early for adoption of a language for prototyping and for production. [0] https://github.com/elixir-lang/elixir/releases/tag/v1.0.0 https://github.com/elixir-lang/elixir/releases/tag/v1.0.0

quaunaut 9y ago
If I remember right, they were using Erlang at the beginning, and moved slowly to Elixir as they got more comfortable, and the ecosystem built up around them.
- Vishnevskiy 9y ago
  We started with Elixir, some of our engineers have used Erlang prior though.

jmcgough 9y ago

Great to see more posts like this promoting Elixir. I've been really enjoying the language and how much power it gets from BEAM. Hopefully more companies see success stories like this and take the plunge - I'm working on an Elixir project right now at my startup and am loving it.

orliesaurus 9y ago

Unlike Discord's design team who seem to just copy all of Slack's designs and assets, the Engineering team seems to have their shit together, it is delightful to read your Elixir blogposts. Good job!

orliesaurus 9y ago
yeah go on with the downvotes, you know everyone's thinking what I said! Copying your way to success is a strategy and it's not the first time either way :)

joonoro 9y ago

Elixir was one of the reasons I started using Discord in the first place. I figured if they were smart enough to use Elixir for a program like this then they would probably have a bright future ahead of them. In practice, Discord hasn't been completely reliable for my group. Lately messages have been dropping out or being sent multiple times. Voice gets messed up (robot voice) at least a couple times per week and we have to switch servers to make it work again. A few times a person's voice connection has stopped working completely for several minutes and there's nothing we can do about it. I don't know if these problems have anything to do with the Elixir backend or the server. EDIT: Grammar

Vishnevskiy 9y ago
The messages struggles have been sadly due to issues with Cassandra and GC pauses caused by bugs within it. We have been trying to work with the Cassandra developers to resolve these. Voice issues should not be happening. Please contact our support with more information and we will gladly investigate.
- joonoro 9y ago
  Thanks for the response, it's good to know what's causing the problems with messages and that it's being worked on. I'll try to contact support next time I have voice issues with my group.
  - Vishnevskiy 9y ago
    :) np We love Cassandra, and hate it at the same time. Check out this nasty bug we got. https://issues.apache.org/jira/browse/CASSANDRA-13004 https://issues.apache.org/jira/browse/CASSANDRA-13004
    - jjirsa 9y ago
      Two things are true about Cassandra: It is by far the best at doing what it does It has plenty of room for improvement

zitterbewegung 9y ago

Compared to slack discord is a much better service for large groups . Facebook uses them for react.

btown 9y ago
I just wish they were more professional in their communications. Service went down hard this weekend and their response was a cat GIF. Hard to justify to anyone in business that they should take it seriously. A shame when their tech is such a great match for professional collaboration, and all it would take to grow it organically would be less marketing copy not more.
- notnight 9y ago
  We take outages seriously internally, but users generally don't care about the nitty-gritty. For a full, detailed postmortem on that outage you can check out our status site: https://status.discordapp.com/incidents/ywdwttd6b0hg https://status.discordapp.com/incidents/ywdwttd6b0hg

jakebasile 9y ago

I'm continually impressed with Discord and their technical blogs contribute to my respect for them. I use it in both my personal life (I run a small server for online friends, plus large game centric servers) and my professional life (instead of Slack). It's a delight to use, the voice chat is extremely high quality, text chat is fast and searchable, and notifications actually work. Discord has become the de facto place for many gaming communities to organize which is a big deal considering how discriminating and exacting PC gamers can be. My only concern is their long term viability and I don't just mean money wise. I'm concerned they'll have to sacrifice the user experience to either achieve sustainability or consent to a buyout by a larger company that only wants the users and brand. I hope I'm wrong, and I bought a year of Nitro to do my part.

lordCarbonFiber 9y ago
A closed source walled garden chat service that survives purely on the free flow of VC capital 100% has no long term viability. No federation means as soon as the "next thing" shows up and wins way the VC dollars they'll disappear. It's so incredibly frusterating that so many companies and users make/support these closed environments even as we enter a new golden age of open sourced and federated technologies.
- jakebasile 9y ago
  If there was an open source and federated equivalent to the features Discord provides I'd use it. There is no such product. Matrix is interesting, but the experience is no where near as polished as Discord and friends and that matters for mass adoption.
  - aphextron 9y ago
    >If there was an open source and federated equivalent to the features Discord provides I'd use it. But will you pay for Discord though? The features and quality Discord is able to provide are artificially propped up by VC funding. When it runs dry, we will be left with open source offerings, or the next product to take it's place and repeat the cycle.

jaequery 9y ago

Anyone know if Phoenix/Elixir have something similar to Ruby's bettererror gem? I see Phoenix has a built-in error stack trace page which looks like a clone of bettererror but it doesn't have the real-time console inside of it. Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this?

themgt 9y ago
Just use IEx.pry if you want a debug console.
kornish 9y ago
Curious about your thoughts on Ecto: https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto. Seems to be the most popular solution for database access.
lojack 9y ago
> Also, I wish they had a ORM like Sequel. These two are really what is holding me back from going full in on Elixir. Anyone can care to comment on this? May be a matter of semantics, but the concept of ORM simply can't exist in a functional language as there aren't objects. That said, Ecto is pretty much the go-to in Elixir. It's extremely powerful, and in my opinion provides the right amount of abstraction without going too far.
cschneid 9y ago
> like Sequel Ecto can do a reasonable impression with its DSL & direct use of tables. You don't need to go through the whole stack of schemas and such. What in particular do you like about Sequel (over what Ecto is providing)?
_asummers 9y ago
Ecto is hands down the best "ORM" I've used. At no point has it ever gotten in the way of us dynamically creating queries on the fly, just following the Ecto internals (not recommended, but totally doable) and has an extremely expressive syntax. The team behind it has been rolling out new features and are extremely responsive to requests on the mailing list. I encourage you to give it a chance. The mental shift is that "schemas" are not objects in an ORM sense, but instead are just data, or views over data. The functions come from taking in data or a Changeset and manipulating those, versus calling a function on a class. As far as errors, Elixir 1.4.5 has much better error messages, specifically printing the args to crashes and such, and OTP 20/ Elixir 1.5 should drastically improve Dialyzer error messages. I am not a Ruby guy, so perhaps you can say what you feel is missing from Elixir's messages and how the Ruby gem improves it. Also you can literally inspect running state on the fly with the BEAM tooling, so the need for things like debuggers goes down.

iagooar 9y ago

This writeup make me even more convinced of Elixir becoming one of the large players when it comes to hugely scaling applications. If there is one thing I truly love about Elixir, it is the easiness of getting started, while standing on the shoulders of a giant that is the Erlang VM. You can start by building a simple, not very demanding application with it, yet once you hit a large scale, there is plenty of battle-proven tools to save you massive headaches and costly rewrites. Still, I feel, that using Elixir is, today, still a large bet. You need to convince your colleagues as much as your bosses / customers to take the risk. But you can rest assured it will not fail you as you need to push it to the next level. Nothing comes for free, and at the right scale, even the Erlang VM is not a silver bullet and will require your engineering team to invest their talent, time and effort to fine tune it. Yet, once you dig deep enough into it, you'll find plenty of ways to solve your problem at a lower cost as compared to other solutions. I see a bright future for Elixir, and a breath of fresh air for Erlang. It's such a great time to be alive!

toolz 9y ago
I think the next big push in elixir is tooling around metrics. Once there are hard numbers to market to non-techies I think we'll see a large shift towards elixir.
- rubyn00bie 9y ago
  Just look for Erlang tools, there are loads available. The only problem with Elixir right now is sometimes people forget to look at Erlang for already solved problems/tooling/etc. Erlang/BEAM has been alive and well in some very serious mission critical applications for 20 years. Just look at how telecoms used it (its origin).
  - _asummers 9y ago
    We use Elixir at work, and Erlang tools have a bit of a discovery problem. You can peek at download counts on Hex, if the author uses it, but that doesn't take account CI systems and such that inflate DL counts, or GitHub only repos that are not on Hex (see things like e.g. shell history prior to OTP20). That said, when there is no Elixir tool to do what I want, the very next thing I do is look for an Erlang tool, and there is often one that gets me most of the way to where I need to go. Wrapper libs are nice sugar, but they're almost always unnecessary when you can just call Erlang directly (String v charlist conversions notwithstanding!).

Cieplak 9y ago

I know that the JVM is a modern marvel of software engineering, so I'm always surprised when my Erlang apps consume less than 10MB of RAM, start up nearly instantaneously, respond to HTTP requests in less than 10ms and run forever, while my Java apps take 2 minutes to start up, have several hundred millisecond HTTP response latency and horde memory. Granted, it's more an issue with Spring than with Java, and Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point, but perhaps there's an element of "less is more" going on here. Also, we're looking for Erlang folks with payments experience. cGF0cmljaytobkBmaW5peHBheW1lbnRzLmNvbQ==

kevincennis 9y ago
Off topic, but what's the story behind base-64 encoding your email? Is that just a spam prevention measure?
- Cieplak 9y ago
  Yes
  - Corrado 9y ago
    I actually really like this idea and am going to steal it. :)
- Deinumite 9y ago
  I think it is also sometimes a simple barrier to entry to weed out some candidates (think of it as a super simple fizzbuzz). I also see a lot of job postings asking people to decode some string for bonus "points"... 90% of the time its just rot13.
pencilhappen 9y ago
> Parallel Universe's Quasar is basically OTP for Java, so I know logically that Java is basically a superset of Erlang at this point It's not really a superset. The JVM has a single global heap, whereas BEAM has per-process heaps. Global state is the enemy of concurrency, and the best GC algorithm is one you don't need to run at all. The "several hundred millisecond HTTP response latency and horde (sic) memory" problems you note are in part a result of having a global heap.

ConanRus 9y ago

I do not see there any Elixir specific, it is all basically Erlang/Erlang VM/OTP stuff. When you using Erlang, you think in terms of actors/processes and message passing, and this is (IMHO) a natural way of thinking about distributed systems. So this article is a perfect example how simple solutions can solve scalability issues if you're using right platform for that.

nichochar 9y ago
You're right. Elixir doesn't pretend to do anything except make using the wonderful erlang VM/OTP stuff easier. The VM is an absolute marvel of engineering, and it's insane to me that it doesn't have more adoption yet in big tech companies. My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again.
- pmarreck 9y ago
  > and it's insane to me that it doesn't have more adoption yet in big tech companies. elixir (and winter) is coming > My best theory is that engineers in top engineering companies are actually not the best engineers but simply career engineers that learn one skill (python/java/C++) and then explain to every employer that this technology is the best for the problem they have, over and over again. you conflated "top" with "large".
- Anderkent 9y ago
  Coordination problems get more difficult in larger teams/companies. Getting everyone to use a particular non-standard language is a coordination problem. Thus large companies are unlikely to experiment with languages. It makes sense too - say it's 10x easier to write something in language X than Y. If there's only 10 other people that might interact with the thing / have to read the sources, that's a great tradeoff. If there's a thousand other people that might have to at some point understand how some part of your code works, suddenly all of them have to learn the new language X.

marlokk 9y ago

"How Discord Scaled Elixir to 5M Concurrent Users" click link [Error 504 Gateway time-out] only on Hacker News

didibus 9y ago

So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Sounds to me like languages don't matter for scale. In fact, that makes sense, scale is all about parallel processes, horizontally distributing work can be achieved in all language. Scale is not like perforance, where if you need it, you are restricted to a few languages only. That's why I'd like to hear more about productivity and ease now. Is it faster and more fun to scale things in certain languages then others. Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and procudctive way to scale, that doesn't matter. Anyways, learning how team scaled is interesting, but it's clear to me now languages aren't limiting factors to scale.

YorickPeterse 9y ago
While you don't need language X to scale, certain languages can definitely make it _easier_ and more cost-effective to scale. So it can matter depending on what you're trying to achieve.
nateberkopec 9y ago
If you have a share-nothing architecture, then yes, any language can scale to any load, some with more hardware than others.
dmix 9y ago
> So, at this point, every language was scaled to very high concurrent loads. What does that tell us? Just like any language vs language debate each one has benefits for various particular use-cases. Any meaningful comparison of languages must be prefaced with the use-case scenario. One of the strongest use-cases of Erlang/Elixir has always been building large distributed apps that need to scale (async web apps, telecom, chat servers, messaging mobile apps, etc). The ability to build these large distributed systems are baked into the very primitive parts of the language and standard library - to a degree that few other languages can compare to it, if any. With Erlang/Elixir you design ALL applications in a way where scaling is rarely an after thought but rather a natural extension of the program. > Beam is modeled on actors, and offer no alternatives. Java offers all sorts of models, including actors, but if actors are the currently most fun and productive way to scale, that doesn't matter. People often make the mistake of trivializing Erlang/Elixirs as merely programming with actors. It's development not only predated the actor model but it also goes well beyond that to being the standard programming style you use when developing any program when using the language - the same way Rails embraces MVP. When this is fundamental part of every Erlang application then the means of scaling to a large distributed system are also a fundamental part of each program. This built-in scaling is gained without any significant costs in terms of development time but also provides many benefits beyond scaling, such as highly modular and extensible code. There are real benefits even if you don't plan to scale to a large distributed system. similar to Rails it creates a predictable program design which makes joining new projects easier and deters NIH syndrome that is far too common in Java/C++/etc. And ultimately, regardless of what you are building, it provides very high performance by default for the type of async style applications that are popular on the web today. So the key point here is not that the end goal was achieved (that you can scale) but how you get there.

framp 9y ago

Really lovely post! I wonder how Cloud Haskell would fare in such a scenario

khanan 9y ago

Problem is that Discord sucks since it does not have a dedicated server. Sorry, move along.

vultour 9y ago
That's actually why it doesn't suck for the vast majority. Not everyone wants to pay $ every month so they could have their own voice / chat server.

alberth 9y ago

Is there any update on BEAMJIT? It was super promising 3 or so years ago. But I haven't seen an update. Erlang is amazing in numerous ways but raw performance is not one of them. BEAMJIT is a project to address exactly that. https://www.sics.se/projects/beamjit https://www.sics.se/projects/beamjit

jlouis 9y ago
Still ongoing work. My personal bet is a bit more on modernizing HiPE however (by using the LLVM backend more).
- alberth 9y ago
  Amy ETA on when we can start using beamjit?
  - jlouis 9y ago
    Given that it has been postponed a couple of times, no. JITs are hard to pull off and it will probably have a period of worse stability as well before it matures. Another problem is getting a JIT to be faster than the interpreter. Erlang's BEAM is threaded code and also macro-instructions, so it almost looks like a JIT internally. The big gains would be in inlining across module boundaries and type speculation. But I hold that if we could compile bundles of modules in HiPE, we would have the same gain for a fraction of the development and maintenance effort. The biggest lure of native code generation would be that we could get rid of a lot of C code in the system as the native cogen would be able to rival the C code in speed. Many Erlang programs spend shockingly little time in the emulator loop, especially if they are communication heavy. If you need speed today, don't underestimate a port-program. My test is that you can pipeline about a million requests back and forth to an OCaml program per second per core. So if your work is on the order of 1+ milliseconds, this is usually a feasible strategy. Espcially because OS isolation means you can handle exceptions in the OCaml program from the Erlang side by restarting the port.
    - rozap 9y ago

jlouis 9y ago

A fun idea is to do away with the "guild" servers in the architecture and simply run message passes from the websocket process over the Manifold system. A little bit of ETS work should make this doable and now an eager sending process is paying for the work itself, slowing it down. This is exactly the behavior you want. If you are bit more sinister you also format most of the message in the sending process and makes it into a binary. This ensures data is passed by reference and not copied in the system. It ought to bring message sends down to about funcall overhead if done right. It is probably not a solution for current Discord as they rely on linearizability, but I toyed with building an IRCd in Erlang years ago, and there we managed to avoid having a process per channel in the system via the above trick. As for the "hoops you have to jump through", it is usually true in any language. When a system experiences pressure, how easy it is to deal with that pressure is usually what matters. Other languages are "phase shifts" and while certain things become simpler in that language, other things become much harder to pull off.

mononcqc 9y ago
The true evil approach is to send the socket around, not the message, so that there is no copying required no matter what ;)
- rdtsc 9y ago
  Wah. Easy there, Satan :-) That is cool trick though. So it's basically sending the port itself around and changing its ownership, with something like port_connect(Port,NewOwner)? And btw, thank you for writing https://www.erlang-in-anger.com https://www.erlang-in-anger.com and http://learnyousomeerlang.com http://learnyousomeerlang.com !
  - mononcqc 9y ago
    The trick is more commonly used when writing to sockets. A socket owner is required for reading, not for writing. The trick then is that when you need to write lots of data to a socket to just send a copy of it to the writer so they can dump all their data for cheap, but without changing ownership (which is costly). Also recently I've gotten http://propertesting.com/ http://propertesting.com/ out, you might enjoy it :)

ramchip 9y ago

Very interesting article! One thing I'm curious about is how to ensure a given guild's process only runs on one node at a time, and the ring is consistent between nodes. Do you use an external system like zookeeper? Or do you have very reliable networking and consider netsplits a tolerable risk?

jhgg 9y ago
We use etcd.

deleted 9y ago

[deleted]

marcgcombi 9y ago

Looks like eSpam promotion?

neya 9y ago

Hi community, Let me share my experience with you. I'm a hardcore Rails guy and I've been advocating and teaching Rails to the community for years. My workflow for trying out a new language involves using the language for a small side project and gradually would try to scale it up. So, here's my summary, my experience of all the languages so far: Scala - It's a vast academic language (official book is with ~700 pages) with multiple ways of doing things and it's attractiveness for me was the JVM. It's proven, robust and highly scalable. However, the language was not quite easy to understand and the frameworks that I've tried (Play 2, Lift) weren't as easy to transition to, for a Rails developer like me. Nevertheless, I did build a simple calendar application, but it took me 2 months to learn the language and build it. GoLang - This was my next bet, although I didn't give up on Scala completely (I know it has its uses), I wanted something simple. I used Go and had the same experience as I had when I used C++. It's a fine language, but, for a simple language, I had to fight a lot with configuration to get it working for me - (For example, it has this crazy concept of GOPATH where your project should reside and if your project isn't there it'll keep complaining). Nevertheless, I build my own (simple) Rails clone in GO and realized this isn't what I was looking for. It took my about a month to conquer the language and build my (simple) side project. Elixir - Finally, I heard of Elixir on multiple HN Rails release threads and decided to give it a go. I started off with Phoenix. The transition was definitely wayy smoother from Rails, especially considering the founding member of this language was a Rails dev. himself (the author of "devise" gem). At first some concepts seemed different (like piping), but once I got used to it, for me there was no looking back. All was fine until they released Phoenix 1.3, where they introduced the concept of contexts and (re) introduced Umbrella applications. Basically they encourage you to break your application into smaller applications by business function (similar to microservices) except that you can do this however you like (unopinionated). For example, I broke down my application by business units (Finance, Marketing, etc.). This forced me to re-think my application in a way I never would have thought and by this time I had finished reading all 3 popular books on this topic (Domain Driven Design). I loved how the fact that Elixir's design choices are really well suited for DDD. If you're new to DDD I suggest you try giving it a shot, it really can force you to re-think the way you develop software. By the end of two weeks after being introduced to Elixir, I picked up the language. In a month and a half, I built a complete Salesforce clone just working on the weekends. And this includes even the UI. And I love how my application is always blazing fast, picks up errors even before it compiles and warns me if I'm no using a variable I defined somewhere. P.S there IS a small learning curve involved if you're starting out fresh: 1) IF you're used to the Rails asset pipeline, you'll need to learn some new tools like Brunch / Webpack / etc. 2) Understand about contexts & DDD (optional) if you want to better architect your application. 3) There is no return statement in Elixir! As a Ruby developer, here are my thoughts: 1. So, will I be developing with Rails again? Probably yes, for simpler applications / API servers. 2. Is Ruby dying? No. In fact, I can't wait for Ruby 3. Some drawbacks of Elixir: 1. Relatively new, so sometimes you'll be on your own and that's okay. 2. Fewer libraries as compared to the Ruby eco-system. But you can easily write your own. 3. Fewer developers, but should be fairly to onboard Ruby developers. Cheers.

omeid2 9y ago

I think while this is great, it is good to remember that your current tech stack maybe just fine! after all, Discord start with mongodb[0]. [1]. https://blog.discordapp.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7 https://blog.discordapp.com/how-discord-stores-billions-of-m...

grantwu 9y ago

"Discord clients depend on linearizability of events" Could this be possibly be the cause of the message reordering and dropping that I experience when I'm on a spotty connection?

sriram_malhar 9y ago

I really like elxir the language, but find myself strangely hamstrung by the _mix_ tool. There is only an introduction to the tool, but not a reference to all the bells and whistles of the tool. I'm not looking for extra bells and whistles, but simple stuff like pulling in a module from GitHub and incorporate it. Is there such documentation? How do you crack Mix?

amorphid 9y ago
The docs for Mix are decent. You can start here: https://hexdocs.pm/mix https://hexdocs.pm/mix. When you are trying to get help w/ a specific task, you check checkout the mix tasks docs: https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content https://hexdocs.pm/mix/Mix.Tasks.Deps.Get.html#content And honestly, I often just look at the source code: https://github.com/elixir-lang/elixir/tree/master/lib/mix https://github.com/elixir-lang/elixir/tree/master/lib/mix The Elixir Slack channel is pretty amazing, too: https://elixir-slackin.herokuapp.com/ https://elixir-slackin.herokuapp.com/

andy_ppp 9y ago

Just as an aside how would people build something like this if they were to use say Python and try to scale to these sort of user levels? Has anyone succeeded? I'd say it would be quite a struggle without some seriously clever work!

MusaTheRedGuard 9y ago
Yeah pretty much impossible with python.

KrishnaHarish 9y ago

Scale!

KrishnaHarish 9y ago

What is Discord and Elixir?

majidazimi 9y ago

It seems awkward to me. What if Erlang/OTP team can not guarantee message serialization compatibility across a major release? How you are going to upgrade a cluster one node at a time? What if you want to communicate with other platforms? How you are going to modify distribution protocol on a running cluster without downtime? As soon as you introduce standard message format, then all nice features such as built-in distribution, automatic reconnect, ... are almost useless. You have to do all these manually. May be I'm missing something. Correct me if I'm wrong. For a fast time to market it seems quite nice approach. But for a long running maintainable back-end it not enough.

ramchip 9y ago
Distributed Erlang compatibility is guaranteed for not just one, but two major versions. You can do a standard rolling upgrade and the newer nodes will talk to older nodes just fine. http://erlang.org/pipermail/erlang-questions/2011-June/059463.html http://erlang.org/pipermail/erlang-questions/2011-June/05946... Foreign nodes in C, Java, Python, etc. can also join an Erlang cluster: http://erlang.org/doc/tutorial/cnode.html http://erlang.org/doc/tutorial/cnode.html http://erlang.org/doc/apps/jinterface/jinterface_users_guide.html http://erlang.org/doc/apps/jinterface/jinterface_users_guide... That being said, a more typical architecture is to have Erlang spawn external processes and talk to them via stdio.
- majidazimi 9y ago
  It's great that they support C and JVM stack. But for other major platforms such as NodeJS, Go, ... we still need to approach the goal with some C extensions integrated in both platforms. Erlang/OTP distribution/clustering is really not designed for heterogeneous environment which is fine, since it is used in telecom industry with nice commercial support contracts backed by Ericsson. However I don't consider it as an amazing piece of engineering from a perspective of a system designer who is dealing with many teams with different language/stack preferences.
  - ramchip 9y ago
    Erlang is what you would use to implement a single service (which may run on a cluster of multiple nodes). It's not a generic messaging layer to use between services, you can use RabbitMQ or ZeroMQ or whatever you like for that.

OOPMan 9y ago

5 million concurrent users is great and all, but it would be nice if Discord could work out how to use WebSockets without duplicating sent messages. This seems to happen a lot when you are switching between wireless networks (E.g. My home router has 2Ghz and 5Ghz wireless networks) or when you're on mobile (Seems to happen regularly, even if you're not moving around). It's terribly annoying though and makes using the app via the mobile client to be very tedious.

k__ 9y ago
Coming from IRC with netsplits and stuff, this seems like a first world problem to me hehe
- OOPMan 9y ago
  You know there are discord bots for bridging between IRC and Discord? Pretty funky :-)

StreamBright 9y ago

Whatsapp's story is somewhat similar. Relevant read to this subject. http://www.erlang-factory.com/upload/presentations/558/efsf2012-whatsapp-scaling.pdf http://www.erlang-factory.com/upload/presentations/558/efsf2...

dandare 9y ago

What is the business model behind Discord? They boast about being free multiple times, how do they make money? Or plan to make money?

satsuma 9y ago
They currently have a Nitro service that's $5/month for little features, but I don't know how far that takes them towards profitability.

deleted 9y ago

[deleted]

agentgt 9y ago

I realize this is off topic but how does Discord make money? I can't figure out their biz model (I'm not a gamer so I didn't even know about them).

renaudg 9y ago

It looks like they have built an interesting, robust and scalable system which is perfectly tailored to their needs. If one didn't want to build all of that in house though, is there anything they've described here that an off the shelf system like https://socketcluster.io https://socketcluster.io doesn't provide ?

lostcolony 9y ago
Yes. Discord is served over HTTPS. :P (your link is broken; socketcluster.io doesn't serve over HTTPS) But seriously, Discord actually benchmarked 5 million concurrent users, horizontally distributed, and having to ferry messages across the cluster, with specifically tailored fanout patterns (rather than just a global pub/sub. I.e., who a message goes to varies, rather than just "everyone"). Socketcluster.io only has benchmarks for a single machine, capped at 42k concurrent connections (though to be fair that was due to them running a single client, rather than a limitation of the server). They don't out of the box support horizontal scaling; you're required to spin up your own message queue solution for that. So, basically, you're advocating a technology that solves -the simplest part of the problem-, and nothing else. Whereas Phoenix + Elixir, even without any of the custom tweaking Discord describes, solve that AND more of the actual problem Discord had. So...yes, and no. Yes, there is plenty here they've describe that is not available in socketcluster.io, but no, nothing they've done here is no generally solved by an off the shelf system, because they're -using- an off the shelf system, Elixir + Phoenix.
- jondubois 9y ago
  Hi, main author of SocketCluster here. SC does support automatic horizontal scaling across any number of machines out of the box if you're running it on Kubernetes. There's also a CLI tool to deploy it automatically to any Kubernetes cluster: https://www.npmjs.com/package/baasil https://www.npmjs.com/package/baasil See https://github.com/SocketCluster/socketcluster/blob/master/scc-guide.md#scc-guide https://github.com/SocketCluster/socketcluster/blob/master/s...
  - lostcolony 9y ago

etblg 9y ago

Reading posts like this about widely distributed applications always gets me interested in it as a career path. Currently I'm working as a front-end dev with moderate non-distributed back-end experience. How would someone in my situation, with no distributed back-end experience, break in to a position working on something like Discord?

concatime 9y ago

Sad to see some people taking raw and insignificant benchmarks to evaluate a language[0]. [0] https://news.ycombinator.com/item?id=14479757 https://news.ycombinator.com/item?id=14479757

oldpond 9y ago

When have you ever read, "How Acme scaled J2EE to 5M Concurrent Users"? I became an IT architect in 1998 at IBM, the year Sun released j2ee and IBM released Websphere. I have experienced 20 years of enterprise Java and object oriented computing, and I was thrilled when Elixir came out. I was a mainframe programmer before OO became all the rage, so I never really felt at home doing objects. Functional programming feels completely natural to me though. What I like about this article is that they shared everything they learned with the community. Thank you for that excellent experience report.