20 ms·

Why Discord is switching from Go to Rust (2020)

232 points by harporoeder 6y ago

cheaprentalyeti 6y ago
I didn't know Discord was written in Go to begin with. I thought it was written using electron or something like that.
- TheChaplain 6y ago
  The client is electron, I believe Go/Rust is used in the backend systems.
  - mewpmewp2 6y ago
    I thought backend was elixir.
    - dnautics 6y ago
      Rust in hot paths
    - notamy 6y ago
      As I remember, - Real-time stuff (websocket gateway, etc.) is Elixir with some Rust NIFs - REST API is Python - Other stuff is Rust or Python afaik? Probably wrong here Obligatory "I don't work there"
- jhgg 6y ago
  We were experimenting with Go as a "fast" backend language to implement stuff like data services and high throughput systems in. Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
  - anticensor 6y ago
    Have you considered also switching the client to Rust?
    - politician 6y ago
      Are there any apps that you know of that use Servo as a rendering engine? I'm curious and would like to look at some examples; I thought Servo was intrinsically tied to Firefox.
      - anticensor 6y ago
        No, not yet. But Servo can be used headless: https://github.com/paulrouget/servo-embedding-example https://github.com/paulrouget/servo-embedding-example
    - jhgg 6y ago
      No. React/TypeScript is too productive for our project engineers. But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
  - throwawaysea 6y ago
    Does Discord use the cloud for infrastructure or are they big enough that they are going their own way?
    - jhgg 6y ago
      We are hybrid. We use bare metal for our RTC workload (voice video streaming) and GCP for the rest.
      - throwawaysea 6y ago
        I'm curious if you have any thoughts on power savings/sustainability from using Rust, or from using Rust with ARM-based processors on bare metal, for example. Is it meaningfully different at your scale?
- dnautics 6y ago
  The backend is still quite a bit of elixir (I imagine, I don't know as I don't work there) - as of October 2020 blog post
rvz 6y ago
> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense. Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.
- bobthebuilders 6y ago
  Rust will never replace C++. It is terrible langauge driven by terrible programmers.
- im3w1l 6y ago
  The impression I get when I hear they use Go, Elixir and Rust on the backend is that they are chasing every shiny trend. I doubt it makes sense from a purely tecnical perspective and is more a way of indulging and investing in the skills of their engineers.
  - jhgg 6y ago
    Elixir was not us chasing a trend - but rather betting on a language and runtime system we knew would be amazing if it worked. Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users? Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
    - masklinn 6y ago
      It's not really the subject of the discussion, but since you're here and interacting with people: would it be possible to ask the UI/UX folks if they could finally make the built-in shortcuts configurable (and deactivable)? It's really weird that such a game adjacent piece of software completely hardcodes shortcuts, especially when so many of them either conflict with or are near-misses for common chords, especially for very disruptive actions like starting voice calls. Even more so as it already has configurable keychords, just not for most of the useful stuff. Also they should fix the labelling around these, either it's just not working or it doesn't work reliably, but trying to override built-in shortcuts via the configuration (in order to disable them) has never worked for me.
steveklabnik 6y ago
This was covered a year ago, and is literally the most popular post for "Rust" ever on this site https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335
- deleted 6y ago
  [deleted]
- Koiwai 6y ago
  This, just realized that post was dated Feb 2020 not Feb 2021.
lrae 6y ago
(Feb 4, 2020)
sandGorgon 6y ago
How timely! https://stackoverflow.blog/2021/02/22/choosing-java-instead-of-c-for-low-latency-systems/ https://stackoverflow.blog/2021/02/22/choosing-java-instead-... I do believe that there is nothing better tuned for performance vs developer productivity than the JVM. The language semantics may be a question for personal distaste..but then you do have Kotlin (https://jooby.io/ https://jooby.io/ ) which ranks fairly near the top on techempower benchmarks https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite&l=wqmbun-sf https://www.techempower.com/benchmarks/#section=data-r20&hw=...
jjjeii3 6y ago
GO promised to be very fast, but if you check the actual benchmarks: https://www.techempower.com/benchmarks/#section=test&runid=8ca46892-e46c-4088-9443-05722ad6f7fb&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=test&runid=8... High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3. So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.
- anewaccount2021 6y ago
  Go is slower than Rust but three times as fast as you'll ever need it to be Everything in Techempower that is in the top half of performance is likely to be fast enough that your own architectural issues will be gating your performance, not the language/framework
- Shadonototro 6y ago
  Don't look at that kind of benchmark, most cheat, and notice how latency is bad, because C# is not native code, so the JIT needs warmup just like java, wich introduces hicups and non-deterministic performance So these perfect situations where you only do the same thing and call same code over and over never happen; never! Anyways, GO isn't popular for its RAW perf, GO is popular because: - Simple language - Native code - GC - Cross compilation - Fast iteration (build/rebuild/deploy) - Relatively tiny, statically compiled executable
  - felipellrocha 6y ago
    I think you're missing the point. The point is that you can get all those properties in Rust without the performance hit. (Except for, well, GC, obviously)
    - unanswered 6y ago
      Lack of garbage is a pro, not a con.
    - 6y ago
deleted 6y ago
[deleted]
dvfjsdhgfv 6y ago
In other words, they solved the problem they wouldn't have if they had stuck to C in the first place.
- Twisol 6y ago
  We may never know for sure, but I bet they avoided a whole bunch of problems by using Go instead of C. Probably wise to go lower only when needs must.
- deleted 6y ago
  [deleted]
kbar13 6y ago
i'm curious if allocating a memory ballast would have helped discord in this case? a one line patch in main() with something like ballast := make([]byte, 10<<30) might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so. https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- aw1621107 6y ago
  The GC cycles were triggered not due to heap size but due to the forced 2 minute max time between GC cycles [0], so a ballast wouldn't have helped here. [0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72d80f375f53358b/src/runtime/proc.go#L5015 https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
  - ttymck 6y ago
    This is really interesting. Are there not alternative runtimes, or configurable gc periods? My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
    - masklinn 6y ago
      The Go core team intensely dislikes tunables, so Go has very few of these (really the opposite of the JVM). There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430 https://github.com/golang/go/issues/42430
- masklinn 6y ago
  > i'm curious if allocating a memory ballast would have helped discord in this case? No. Twitch's ballast is because > At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute). Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below. But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.
deleted 6y ago
[deleted]
CobsterLock 6y ago
Off-topic: who is in charge of choosing colors for graphs? Being red-green colorblind, choosing blue and purple is makes it impossible for me to read the charts. I know its stylish, but every time i make a graph i choose very high contrast colors. Another point: is it just being color blind or do other people struggle in charts/maps that have shades of a color? like different shades of blue represent different rates of covid deaths across a map. just curious if anyone else struggles with this
- chungy 6y ago
  I am not colorblind, and the chosen colors for this graph are similar enough to make it difficult to read. High-contrast helps the rest of us, too.
- ihumanable 6y ago
  These are just the default colors for Datadog graphs, so I guess the answer to your question is Datadog.
- jlouis 6y ago
  Brewer colour schemes ftw.
jeffbee 6y ago
Feels like there might have been a middle ground short of a scratch rewrite. LRU caches are always suspect. LRU is pretty much the worst replacement policy you can choose. I wonder if a smaller cache with a better replacement policy (2Q, ARC, ...) would have solved this specific problem. Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.
- Jtsummers 6y ago
  > Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it. Just to check that I understand what you're suggesting here, this is comparable to how we embedded systems folks might pre-allocate (either statically or at the very start of a program execution) a large chunk of memory and use it instead of using traditional malloc/free (or language equivalent), correct? So you bypass the GC and avoid any potential performance penalties introduced by having a smaller number of massive allocations that never get moved, freed, or reallocated.
  - jeffbee 6y ago
    Yes, the idea is similar. If you make a huge map in Go with millions of items, it will stress the GC. If you take the same information and just put it into a big byte slice, and index into it in a way that the GC doesn't think you're pointing to addresses, then it will be much faster. Of course now you are writing a little more code.
tschellenbach 6y ago
2021 will be about them switching back. All jokes aside, it's really common for engineers to randomly switch between solutions incurring huge costs for the companies they work for. Often as it is in this case with no gain for the company, other than some resume building for the involved team members. We use Go here at Stream and power activity feeds and chat for a billion end users. Works perfectly. Cockroach also wrote about GC optimization: https://www.cockroachlabs.com/blog/how-to-optimize-garbage-collection-in-go/ https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c... For caching specifically there is this library as well: https://dgraph.io/blog/post/introducing-ristretto-high-perf-go-cache/ https://dgraph.io/blog/post/introducing-ristretto-high-perf-... Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity. Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.
- franklampard 6y ago
  > Often as it is in this case with no gain for the company, other than some resume building for the involved team members This is too cynical. A large re-write will not be approved at any large companies without millions of meetings and documentations filled with justifications.
  - berkes 6y ago
    ... All of which are researched with proof of concepts. Followed by some small trials. And finalized with one or two actual rewrites of non-critical software. I'm not saying that all companies do it this way, and that there is no "resume building" or "magpie-ing" by some engineers in many companies. But it is perfectly possible to state a thesis "Rust is better for our X and Y and probably for Z". and to then research and prove that thesis.
- jhgg 6y ago
  Not a chance. We are going hard on rust in 2021 after some very successful projects in 2019 and 2020. our engineers have ramped up on the language - and we have good support internally (both in terms of tools, but also knowledge) to see its success. Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go. Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands. And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.
cfors 6y ago
Sounds like Discord was running gigantic in memory Go caches. Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense. Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite. [0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- masklinn 6y ago
  > I'm curious if they tried the Ballast[0] approach From the essay, the issue discord had was that they were not triggering enough GCs. The ballast approach serves to tune the GC frequency down. In Discord's case they were getting screwed by a periodic mandatory GC trigger. > along with smaller cache's per service They did use a smaller cache, but that increased their 99% since… there was now less stuff in the cache.
  - cfors 6y ago
    Makes me wonder what their partitioning strategy was. I can't imagine a cache that is over 4GB+ that couldn't be partitioned at least a bit further. But good call out, thank you for the correction.
- kyrra 6y ago
  There's a proposal to put in official support for minimum heap size: https://github.com/golang/go/issues/44309 https://github.com/golang/go/issues/44309
- deleted 6y ago
  [deleted]
mraza007 6y ago
I'm just curious why can't they use C or C++ instead of Rust
- cletus 6y ago
  Compile time memory safety? Unless you have an existing code base or a necessary integration or requirement that demands it, I honestly wouldn’t start a C/C++ project in 2021 all other things being equal.
- jeffbee 6y ago
  Because nobody would upvote that on HN. The HN zeitgeist of programming languages is absurdly biased. In actual industrial practice the dominant languages are Java, C++, C, C#, Python, Javascript, dozens of things people rarely discuss like matlab, VB, delphi, a bunch of other shit, and way, way down at the bottom, Rust, and then about fifty other languages, and finally Erlang. It's ten times easier to find a job writing MIPS assembly code than to find one programming in Rust, but nobody wants to blog about that.
  - jacksnipe 6y ago
    In actual industrial practice, most YOUNG C++ developers I know would love to be able to work with Rust at work.
    - jeffbee 6y ago
      There's a lot of territory between what one wishes to happen at work, and what actually happens at work.
      - jacksnipe 6y ago
        Right, I'm just saying that the HN zeitgeist is -- at least in this case -- reflecting a real belief. It's what people want to work with, not what currently is run everywhere in production; and what developers want to work with definitely matters. Discord rewrote this service in Rust because they had already had success using Rust for some performance critical services; it seems likely that they chose to invest in Rust and not C++ for those services because their developers pushed harder for or were more interested in Rust.
sgt 6y ago
I wouldn't mind looking into Rust to eventually replace our Java and Python based stacks, but every time I look at it, I tend to give some thought to the difficulty of writing Rust as opposed to reading/writing/maintaining Java for an entire team. Frankly I'm worried that not all team members will become good Rust developers, and the codebase will develop serious rot over time. Any experiences with this?
- l2cluster 6y ago
  We switched Ruby -> Scala, and it took considerable training effort and some years before everyone was comfortable. I would estimate 15% of the developers still can't write moderately good Scala code.
- johnnycerberus 6y ago
  I think you should be worried instead of the amount of libraries that you give up from the JVM and Python ecosystems before anything else. Also, the applications of Rust and Java/Python generally do not overlap, I would be cautious in approaching this transition. Scala would be a better choice since you have expertise on the JVM and it has a familiar syntax to Python devs.
- ris 6y ago
  > our Java and Python based stacks I always find it bizarre when python (and to some extent java) are lumped in as languages that you found Go or Rust a suitable replacement for. They're just such different ends of the spectrum - Go/Rust will just never be able to match the expressivity and rapid development ease of python because they have such different design goals. If you find porting a production application from python to Go/Rust a satisfactory outcome, then I suspect you should never have been using python for it in the first place or you've been using python All Wrong.
  - sgt 6y ago
    I have used Go on a project that could easily have been done in Python, though. So they are not that far apart in terms of suitability.
geodel 6y ago
Ah Discord, that provides electron based chat client. I am sure they can write efficient software when its their money.
- kemenaran 6y ago
  To be fair, they probably ship one of the best and most optimized Electron app out there. I still don’t like Electron, but Discord is doing a good job with it. Even their React-Native mobile apps feel decently snappy (where usually large React-Native apps are buggy and slow as fuck).
  - saagarjha 6y ago
    That's pretty debatable.
reikonomusha 6y ago
Meta-comment, and I’m not claiming this is what’s true about Discord. Every time I see a company post about how they’re changing programming languages, I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you didn’t provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change. (It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.) This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be). As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not? I think using any language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the new deficiencies will be completely unknown.
- aaomidi 6y ago
  Well, on the positive side a company like Discord moving to Rust means better tooling than what is currently available. So that's exciting.
  - reikonomusha 6y ago
    Better tooling for the programmers you mean? I don’t regularly program in either language (Go or Rust), but I thought Google ran a pretty tight ship on good Go tooling, except with a bit of module SNAFU in its past.
silentsea90 6y ago
Incredibly brave of them to go with nightly build of Rust for async functionality. Not as in the weeds, but I would have been afraid of release being delayed forever and being stuck with the nightly release for a long time.
swiley 6y ago
The main thing that bothers me about rust is that there is one complete implementation of the language (and one or two very incomplete implementations) and the singular complete one isn't even self hosting. Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.
- steveklabnik 6y ago
  What makes rustc not self hosting?
- masklinn 6y ago
  > The main thing that bothers me about rust is that there is one complete implementation of the language The essay is about Go which is pretty much in the same position: gccgo is a non-entity. > and the singular complete one isn't even self hosting. What are you talking about? Rust was self-hosting before it was even stable (a common source of problems as the language can then get optimised for implementing the language, thankfully it doesn't seem to have harmed it too badly).
  - I_AM_A_SMURF 6y ago
    I think they don't consider it self-hosted because only the front-end is written in rust, rustc uses llvm as the back-end, which is not written in rust.
    - mlindner 6y ago
      By that argument gcc isn't self hosting either because it uses various assembly syntaxes in its code that requires an assembler.
ibraheemdev 6y ago
previous discussion: https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335
ufmace 6y ago
This post is getting a bit long in the tooth, at least regarding relatively new and fast-moving ecosystems like Go and Rust. I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published. I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.
- aw1621107 6y ago
  > I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published. Go 1.12 might have addressed this particular issue [0, 1], but it's possible that that came out too late for Discord to try before deciding to rewrite the service. [0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live." [1]: https://old.reddit.com/r/golang/comments/eywx4q/why_discord_is_switching_from_go_to_rust/fgnp7h4/ https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...
KuhlMensch 6y ago
Note: this is an account of using Rust in a single service > The service we switched from Go to Rust is the “Read States” service. And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack: > When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.
swyx 6y ago
> Discord has never been afraid of embracing new technologies that look promising. For example, we were early adopters of Elixir, React, React Native, and Scylla. If a piece of technology is promising and gives us an advantage, we do not mind dealing with the inherent difficulties and instability of the bleeding edge. This is one of the ways we’ve quickly reached 250+ million users with less than 50 engineers. this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?
Sikul 6y ago
Original author here. Cool to see it posted again. We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code. We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen. I'm happy to answer questions here. P.S. We're still growing and hiring (including remotely). You can find my email in my profile.
- xtracto 6y ago
  What happened with your Elixir stuff? ( https://blog.discord.com/scaling-elixir-f9b8e1e7c29b https://blog.discord.com/scaling-elixir-f9b8e1e7c29b ). Is it still there or are you using something else now?
  - Sikul 6y ago
    Elixir is without a doubt still one of the core languages. We try to use the right tool for the job and Elixir is the right tool for parts of our system. We also combo Elixir NIFs and rust to speed up parts of the Elixir services https://blog.discord.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
    - xtracto 6y ago
      Interesting. We are researching Elixir for an online transaction ledger (thousands of payments per second) along with kafka. So far thats the technology that seems more suitable. That discord article was really appreciated to understand some of the intricacies of using Elixir.
      - fbernier 6y ago
        I certainly know of a few Adtech companies using Erlang to handle > 1M bid requests per seconds :). OTP is pretty great for that.
rufusroflpunch 6y ago
These threads are always full of people criticizing these moves by companies. But nothing is gained of nothing is risked, right? Sometimes these things pay off, and we would never know without early adopters.
- rsj_hn 6y ago
  The reason why a lot of people criticize moves to rewrite large codebases is because the industry has collected a lot of knowledge and experience with these types of moves and it rarely works out. This is why it's considered a bad practice. We are not babes in the woods, walking around with our eyes newly opened, exploring this strange, wonderous land. The lay of the land is already known in some areas, and this is one such area. Here I would recommend the following twenty year old article by Joel Spolsky, and Joel was merely reciting knowledge that was already well understood, but in an entertaining way. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
strangelove026 6y ago
I really enjoyed reading this article last time it was posted. I particularly like it because it shows evidence that current tooling wasn't doing the job. I believe I remember seeing some comments on how Discord was on an old version of go that wasn't receiving the benefit around gc collections, but we'll never know if that would've been a solution. That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.
dang 6y ago
Discussed at the time: Why Discord is switching from Go to Rust - https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640 comments) also https://news.ycombinator.com/item?id=23874336 https://news.ycombinator.com/item?id=23874336 - July 2020 (7 comments)
jancker 6y ago
I think the response time / latency to use Cassandra on the "read state" will be higher, and mind you cassandra is written in java (JVM) and has more garbage collection magic and latency than Go. From my point of view, it's not the language, it's your data store.
AlchemistCamp 6y ago
In its early days, this forum was full of excited young devs who had read PG's old essays and we're actively looking for technologies that would give them a competitive edge. Now, it's completely the opposite. Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.
- postalrat 6y ago
  Saying the right thing has always been easier than saying the correct thing.
vonsydov 6y ago
tl;dr because of garbage collection
mamcx 6y ago
Switching languages is THE most efficient way to refactor a messy codebase. The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from worse to bad or for same to same-ish. But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults. --- So, if a codebase is in trouble you can: - Try to refactor it with "better" idioms... but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale. - Change to "shinny lang" where do the right thing is the default... And now if the developers try to do wrong, them will fight against the current. --- Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang. The case of Rust is similar: it make the "best" practiques the default, and very painful to do wrong (and sometimes to do right, but this is not the focus here :) ). So, a language switch is the ultimate clean up for a project. I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble. The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great. Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...
hpen 6y ago
Could something like Discord be written in Swift?
tbodt 6y ago
I've never seen anyone look into just why Go forces a GC on 2 minute intervals. The code originally appeared in https://codereview.appspot.com/5451057 https://codereview.appspot.com/5451057 and there's a bit of discussion on https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E. After reading this, I still don't understand. But I find https://codereview.appspot.com/5451057#msg38 https://codereview.appspot.com/5451057#msg38 particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.
caffeine 6y ago
Very misleading title makes it sound like they are rewriting all code wholesale - in fact it's a targeted performance rewrite of a smallish component.

232 points by harporoeder 6y ago

cheaprentalyeti 6y ago

I didn't know Discord was written in Go to begin with. I thought it was written using electron or something like that.

TheChaplain 6y ago
The client is electron, I believe Go/Rust is used in the backend systems.
- mewpmewp2 6y ago
  I thought backend was elixir.
  - dnautics 6y ago
    Rust in hot paths
  - notamy 6y ago
    As I remember, - Real-time stuff (websocket gateway, etc.) is Elixir with some Rust NIFs - REST API is Python - Other stuff is Rust or Python afaik? Probably wrong here Obligatory "I don't work there"
jhgg 6y ago
We were experimenting with Go as a "fast" backend language to implement stuff like data services and high throughput systems in. Go has proven itself disappointing in many ways. Rust however has only continued to evolve and get better as the community matures. 2021 is definitely the year of "rust" for us - and we will be using it in many more places. Both on the backend but also on the client - where we embed native modules written in rust.
- anticensor 6y ago
  Have you considered also switching the client to Rust?
  - politician 6y ago
    Are there any apps that you know of that use Servo as a rendering engine? I'm curious and would like to look at some examples; I thought Servo was intrinsically tied to Firefox.
    - anticensor 6y ago
      No, not yet. But Servo can be used headless: https://github.com/paulrouget/servo-embedding-example https://github.com/paulrouget/servo-embedding-example
  - jhgg 6y ago
    No. React/TypeScript is too productive for our project engineers. But we do write our perf critical stuff in the client in C++/Rust! For example, our entire video capture pipeline is in C++/Rust (the meaty bits in rust, some glue in C++ to interact with WebRTC.)
- throwawaysea 6y ago
  Does Discord use the cloud for infrastructure or are they big enough that they are going their own way?
  - jhgg 6y ago
    We are hybrid. We use bare metal for our RTC workload (voice video streaming) and GCP for the rest.
    - throwawaysea 6y ago
      I'm curious if you have any thoughts on power savings/sustainability from using Rust, or from using Rust with ARM-based processors on bare metal, for example. Is it meaningfully different at your scale?
dnautics 6y ago
The backend is still quite a bit of elixir (I imagine, I don't know as I don't work there) - as of October 2020 blog post

rvz 6y ago

> When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense. Big takeaway here for those who are married to their tools. Rust happens to be just another tool for this company, and it works well for them; where it makes sense of course.

bobthebuilders 6y ago
Rust will never replace C++. It is terrible langauge driven by terrible programmers.
im3w1l 6y ago
The impression I get when I hear they use Go, Elixir and Rust on the backend is that they are chasing every shiny trend. I doubt it makes sense from a purely tecnical perspective and is more a way of indulging and investing in the skills of their engineers.
- jhgg 6y ago
  Elixir was not us chasing a trend - but rather betting on a language and runtime system we knew would be amazing if it worked. Discord would not exist in the form it does today if it was not for BEAM/OTP. The majority of the chat system was built and maintained by a handful of people. Fundamentally our architecture has not dramatically changed since it's inception in 2015. Can you say the same for other real time distributed systems as they have scaled from thousands to hundreds and of millions of users? Rust was a similar bet - the language looked promising and we were right. It's been amazing to use - and it's only gotten better since we started. rust-analyzer is amazing!
  - masklinn 6y ago
    It's not really the subject of the discussion, but since you're here and interacting with people: would it be possible to ask the UI/UX folks if they could finally make the built-in shortcuts configurable (and deactivable)? It's really weird that such a game adjacent piece of software completely hardcodes shortcuts, especially when so many of them either conflict with or are near-misses for common chords, especially for very disruptive actions like starting voice calls. Even more so as it already has configurable keychords, just not for most of the useful stuff. Also they should fix the labelling around these, either it's just not working or it doesn't work reliably, but trying to override built-in shortcuts via the configuration (in order to disable them) has never worked for me.

steveklabnik 6y ago

This was covered a year ago, and is literally the most popular post for "Rust" ever on this site https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335

deleted 6y ago
[deleted]
Koiwai 6y ago
This, just realized that post was dated Feb 2020 not Feb 2021.

lrae 6y ago

(Feb 4, 2020)

sandGorgon 6y ago

How timely! https://stackoverflow.blog/2021/02/22/choosing-java-instead-of-c-for-low-latency-systems/ https://stackoverflow.blog/2021/02/22/choosing-java-instead-... I do believe that there is nothing better tuned for performance vs developer productivity than the JVM. The language semantics may be a question for personal distaste..but then you do have Kotlin (https://jooby.io/ https://jooby.io/ ) which ranks fairly near the top on techempower benchmarks https://www.techempower.com/benchmarks/#section=data-r20&hw=ph&test=composite&l=wqmbun-sf https://www.techempower.com/benchmarks/#section=data-r20&hw=...

jjjeii3 6y ago

GO promised to be very fast, but if you check the actual benchmarks: https://www.techempower.com/benchmarks/#section=test&runid=8ca46892-e46c-4088-9443-05722ad6f7fb&hw=ph&test=plaintext https://www.techempower.com/benchmarks/#section=test&runid=8... High-level C# (asp.net) is almost twice as fast as GO in the benchmark... Rust is also in the top 3. So, why should I use GO instead of C#, if ASP.NET/C# is so blazing fast and requires a lot less lines of code to achieve the same? -> It makes absolutely sense to switch to C# or Rust.

anewaccount2021 6y ago
Go is slower than Rust but three times as fast as you'll ever need it to be Everything in Techempower that is in the top half of performance is likely to be fast enough that your own architectural issues will be gating your performance, not the language/framework
Shadonototro 6y ago
Don't look at that kind of benchmark, most cheat, and notice how latency is bad, because C# is not native code, so the JIT needs warmup just like java, wich introduces hicups and non-deterministic performance So these perfect situations where you only do the same thing and call same code over and over never happen; never! Anyways, GO isn't popular for its RAW perf, GO is popular because: - Simple language - Native code - GC - Cross compilation - Fast iteration (build/rebuild/deploy) - Relatively tiny, statically compiled executable
- felipellrocha 6y ago
  I think you're missing the point. The point is that you can get all those properties in Rust without the performance hit. (Except for, well, GC, obviously)
  - unanswered 6y ago
    Lack of garbage is a pro, not a con.
  - 6y ago

deleted 6y ago

[deleted]

dvfjsdhgfv 6y ago

In other words, they solved the problem they wouldn't have if they had stuck to C in the first place.

Twisol 6y ago
We may never know for sure, but I bet they avoided a whole bunch of problems by using Go instead of C. Probably wise to go lower only when needs must.
deleted 6y ago
[deleted]

kbar13 6y ago

i'm curious if allocating a memory ballast would have helped discord in this case? a one line patch in main() with something like ballast := make([]byte, 10<<30) might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so. https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

aw1621107 6y ago
The GC cycles were triggered not due to heap size but due to the forced 2 minute max time between GC cycles [0], so a ballast wouldn't have helped here. [0]: https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72d80f375f53358b/src/runtime/proc.go#L5015 https://github.com/golang/go/blob/0f66fb7b856b02497cf801ce72...
- ttymck 6y ago
  This is really interesting. Are there not alternative runtimes, or configurable gc periods? My understanding of the JVM GC is limited, but it seems highly configurable: does it (or other languages/VMs) have an analogous construct, and is it (usually) configurable?
  - masklinn 6y ago
    The Go core team intensely dislikes tunables, so Go has very few of these (really the opposite of the JVM). There is a central issue for GC tunables, but it's mostly where the various concern GC have gone to die so far: https://github.com/golang/go/issues/42430 https://github.com/golang/go/issues/42430
masklinn 6y ago
> i'm curious if allocating a memory ballast would have helped discord in this case? No. Twitch's ballast is because > At steady state, our application was triggering ~8–10 garbage collection (GC) cycles a second (400–600 per minute). Go's GC triggers a collection every time the heap size doubles (until that has no return on investment and it just grows the heap). By creating a very large heap up-front they'd give themselves breathing room to avoid being just on the edge of (but remaining below) a resize, leading to continuous collections keeping them below. But Discord had the opposite issue: a large cache means a large heap size, in a steady state that would be extremely stable (since the heap size is a function of cache size, once the cache is full the only changes are replacing old keys for news). As a result, the only GC would be the mandatory periodic run, which would have to go through the entire (very large) heap, causing STW latency spikes every 2mn.

deleted 6y ago

[deleted]

CobsterLock 6y ago

Off-topic: who is in charge of choosing colors for graphs? Being red-green colorblind, choosing blue and purple is makes it impossible for me to read the charts. I know its stylish, but every time i make a graph i choose very high contrast colors. Another point: is it just being color blind or do other people struggle in charts/maps that have shades of a color? like different shades of blue represent different rates of covid deaths across a map. just curious if anyone else struggles with this

chungy 6y ago
I am not colorblind, and the chosen colors for this graph are similar enough to make it difficult to read. High-contrast helps the rest of us, too.
ihumanable 6y ago
These are just the default colors for Datadog graphs, so I guess the answer to your question is Datadog.
jlouis 6y ago
Brewer colour schemes ftw.

jeffbee 6y ago

Feels like there might have been a middle ground short of a scratch rewrite. LRU caches are always suspect. LRU is pretty much the worst replacement policy you can choose. I wonder if a smaller cache with a better replacement policy (2Q, ARC, ...) would have solved this specific problem. Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it.

Jtsummers 6y ago
> Another approach I've used with caches in Go is to just use a huge byte slice as the cache without any pointers, so the GC can ignore it. Just to check that I understand what you're suggesting here, this is comparable to how we embedded systems folks might pre-allocate (either statically or at the very start of a program execution) a large chunk of memory and use it instead of using traditional malloc/free (or language equivalent), correct? So you bypass the GC and avoid any potential performance penalties introduced by having a smaller number of massive allocations that never get moved, freed, or reallocated.
- jeffbee 6y ago
  Yes, the idea is similar. If you make a huge map in Go with millions of items, it will stress the GC. If you take the same information and just put it into a big byte slice, and index into it in a way that the GC doesn't think you're pointing to addresses, then it will be much faster. Of course now you are writing a little more code.

tschellenbach 6y ago

2021 will be about them switching back. All jokes aside, it's really common for engineers to randomly switch between solutions incurring huge costs for the companies they work for. Often as it is in this case with no gain for the company, other than some resume building for the involved team members. We use Go here at Stream and power activity feeds and chat for a billion end users. Works perfectly. Cockroach also wrote about GC optimization: https://www.cockroachlabs.com/blog/how-to-optimize-garbage-collection-in-go/ https://www.cockroachlabs.com/blog/how-to-optimize-garbage-c... For caching specifically there is this library as well: https://dgraph.io/blog/post/introducing-ristretto-high-perf-go-cache/ https://dgraph.io/blog/post/introducing-ristretto-high-perf-... Nothing wrong with Rust, exciting project, but personally think Go is better for building performant micro services like this. Rust seems more suited for building browsers, drivers etc. Rust's performance is slightly better, but it's not as refined around go routines and developer productivity. Edit: On the other hand, Discord could have made the very valid decision that they have an easier time recruiting for Rust than for Go. Which is in fact a good reason to switch.

franklampard 6y ago
> Often as it is in this case with no gain for the company, other than some resume building for the involved team members This is too cynical. A large re-write will not be approved at any large companies without millions of meetings and documentations filled with justifications.
- berkes 6y ago
  ... All of which are researched with proof of concepts. Followed by some small trials. And finalized with one or two actual rewrites of non-critical software. I'm not saying that all companies do it this way, and that there is no "resume building" or "magpie-ing" by some engineers in many companies. But it is perfectly possible to state a thesis "Rust is better for our X and Y and probably for Z". and to then research and prove that thesis.
jhgg 6y ago
Not a chance. We are going hard on rust in 2021 after some very successful projects in 2019 and 2020. our engineers have ramped up on the language - and we have good support internally (both in terms of tools, but also knowledge) to see its success. Once you've passed the learning curve - imo, rust is far easier and more productive to write than go - especially if you know how to leverage the type system to build idiomatic code and apis that are very hard to use incorrectly. Every piece of rust code we have shipped to production so far has gone perfectly thanks to the really powerful compile time checks and guarantees of the language. I can't say the same for our experiences with go. Our reasons go well beyond "oh the gc in go has given us problems" but more like "go as a language has willingly ignored and has rejected advances in programming languages". You can pry those algebraic data types, enums, borrow checker, and compile time memory management/safety, etc... from my cold dead hands. And for network programming - I've been incredibly delighted with the power and flexibility of rusts futures combined with tokio. The fact that you can just drop a future to cancel it, as opposed to having to thread around a ctx which may or may not work has been amazing.

cfors 6y ago

Sounds like Discord was running gigantic in memory Go caches. Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense. Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite. [0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

masklinn 6y ago
> I'm curious if they tried the Ballast[0] approach From the essay, the issue discord had was that they were not triggering enough GCs. The ballast approach serves to tune the GC frequency down. In Discord's case they were getting screwed by a periodic mandatory GC trigger. > along with smaller cache's per service They did use a smaller cache, but that increased their 99% since… there was now less stuff in the cache.
- cfors 6y ago
  Makes me wonder what their partitioning strategy was. I can't imagine a cache that is over 4GB+ that couldn't be partitioned at least a bit further. But good call out, thank you for the correction.
kyrra 6y ago
There's a proposal to put in official support for minimum heap size: https://github.com/golang/go/issues/44309 https://github.com/golang/go/issues/44309
deleted 6y ago
[deleted]

mraza007 6y ago

I'm just curious why can't they use C or C++ instead of Rust

cletus 6y ago
Compile time memory safety? Unless you have an existing code base or a necessary integration or requirement that demands it, I honestly wouldn’t start a C/C++ project in 2021 all other things being equal.
jeffbee 6y ago
Because nobody would upvote that on HN. The HN zeitgeist of programming languages is absurdly biased. In actual industrial practice the dominant languages are Java, C++, C, C#, Python, Javascript, dozens of things people rarely discuss like matlab, VB, delphi, a bunch of other shit, and way, way down at the bottom, Rust, and then about fifty other languages, and finally Erlang. It's ten times easier to find a job writing MIPS assembly code than to find one programming in Rust, but nobody wants to blog about that.
- jacksnipe 6y ago
  In actual industrial practice, most YOUNG C++ developers I know would love to be able to work with Rust at work.
  - jeffbee 6y ago
    There's a lot of territory between what one wishes to happen at work, and what actually happens at work.
    - jacksnipe 6y ago
      Right, I'm just saying that the HN zeitgeist is -- at least in this case -- reflecting a real belief. It's what people want to work with, not what currently is run everywhere in production; and what developers want to work with definitely matters. Discord rewrote this service in Rust because they had already had success using Rust for some performance critical services; it seems likely that they chose to invest in Rust and not C++ for those services because their developers pushed harder for or were more interested in Rust.

sgt 6y ago

I wouldn't mind looking into Rust to eventually replace our Java and Python based stacks, but every time I look at it, I tend to give some thought to the difficulty of writing Rust as opposed to reading/writing/maintaining Java for an entire team. Frankly I'm worried that not all team members will become good Rust developers, and the codebase will develop serious rot over time. Any experiences with this?

l2cluster 6y ago
We switched Ruby -> Scala, and it took considerable training effort and some years before everyone was comfortable. I would estimate 15% of the developers still can't write moderately good Scala code.
johnnycerberus 6y ago
I think you should be worried instead of the amount of libraries that you give up from the JVM and Python ecosystems before anything else. Also, the applications of Rust and Java/Python generally do not overlap, I would be cautious in approaching this transition. Scala would be a better choice since you have expertise on the JVM and it has a familiar syntax to Python devs.
ris 6y ago
> our Java and Python based stacks I always find it bizarre when python (and to some extent java) are lumped in as languages that you found Go or Rust a suitable replacement for. They're just such different ends of the spectrum - Go/Rust will just never be able to match the expressivity and rapid development ease of python because they have such different design goals. If you find porting a production application from python to Go/Rust a satisfactory outcome, then I suspect you should never have been using python for it in the first place or you've been using python All Wrong.
- sgt 6y ago
  I have used Go on a project that could easily have been done in Python, though. So they are not that far apart in terms of suitability.

geodel 6y ago

Ah Discord, that provides electron based chat client. I am sure they can write efficient software when its their money.

kemenaran 6y ago
To be fair, they probably ship one of the best and most optimized Electron app out there. I still don’t like Electron, but Discord is doing a good job with it. Even their React-Native mobile apps feel decently snappy (where usually large React-Native apps are buggy and slow as fuck).
- saagarjha 6y ago
  That's pretty debatable.

reikonomusha 6y ago

Meta-comment, and I’m not claiming this is what’s true about Discord. Every time I see a company post about how they’re changing programming languages, I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you didn’t provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change. (It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.) This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be). As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not? I think using any language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the new deficiencies will be completely unknown.

aaomidi 6y ago
Well, on the positive side a company like Discord moving to Rust means better tooling than what is currently available. So that's exciting.
- reikonomusha 6y ago
  Better tooling for the programmers you mean? I don’t regularly program in either language (Go or Rust), but I thought Google ran a pretty tight ship on good Go tooling, except with a bit of module SNAFU in its past.

silentsea90 6y ago

Incredibly brave of them to go with nightly build of Rust for async functionality. Not as in the weeds, but I would have been afraid of release being delayed forever and being stuck with the nightly release for a long time.

swiley 6y ago

The main thing that bothers me about rust is that there is one complete implementation of the language (and one or two very incomplete implementations) and the singular complete one isn't even self hosting. Coming from scheme where everyone and their dog has an R5RS compatible REPL that's pretty disturbing.

steveklabnik 6y ago
What makes rustc not self hosting?
masklinn 6y ago
> The main thing that bothers me about rust is that there is one complete implementation of the language The essay is about Go which is pretty much in the same position: gccgo is a non-entity. > and the singular complete one isn't even self hosting. What are you talking about? Rust was self-hosting before it was even stable (a common source of problems as the language can then get optimised for implementing the language, thankfully it doesn't seem to have harmed it too badly).
- I_AM_A_SMURF 6y ago
  I think they don't consider it self-hosted because only the front-end is written in rust, rustc uses llvm as the back-end, which is not written in rust.
  - mlindner 6y ago
    By that argument gcc isn't self hosting either because it uses various assembly syntaxes in its code that requires an assembler.

ibraheemdev 6y ago

previous discussion: https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335

ufmace 6y ago

This post is getting a bit long in the tooth, at least regarding relatively new and fast-moving ecosystems like Go and Rust. I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published. I also wonder how well Rust has continued to work for Discord, if it's been a problem to keep up with the rapid movement particularly in the async ecosystem in the last year or so.

aw1621107 6y ago
> I wonder if the Go team has determined any way to mitigate these long-running GC pauses for apps that don't allocate often since this article was published. Go 1.12 might have addressed this particular issue [0, 1], but it's possible that that came out too late for Discord to try before deciding to rewrite the service. [0]: From Go 1.12 patchnotes: "Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live." [1]: https://old.reddit.com/r/golang/comments/eywx4q/why_discord_is_switching_from_go_to_rust/fgnp7h4/ https://old.reddit.com/r/golang/comments/eywx4q/why_discord_...

KuhlMensch 6y ago

Note: this is an account of using Rust in a single service > The service we switched from Go to Rust is the “Read States” service. And the conclusion doesn't indicate "switching", but rather Rust is an "option" for Discord, and already used in the stack: > When starting a new project or software component, we consider using Rust. Of course, we only use it where it makes sense.

swyx 6y ago

> Discord has never been afraid of embracing new technologies that look promising. For example, we were early adopters of Elixir, React, React Native, and Scylla. If a piece of technology is promising and gives us an advantage, we do not mind dealing with the inherent difficulties and instability of the bleeding edge. This is one of the ways we’ve quickly reached 250+ million users with less than 50 engineers. this sounds like a recipe for disaster in every other company i have worked with. are Discord's engineers just THAT good or is there something else going on here?

Sikul 6y ago

Original author here. Cool to see it posted again. We're still using and investing in rust at Discord. For example, in the past year we successfully pulled part of the core messaging system out of python and into a rust service. Rust has been a boon for developer productivity and correctness of code. We've also been very happy with the way the rust ecosystem is evolving and the amount of adoption the language has seen. I'm happy to answer questions here. P.S. We're still growing and hiring (including remotely). You can find my email in my profile.

xtracto 6y ago
What happened with your Elixir stuff? ( https://blog.discord.com/scaling-elixir-f9b8e1e7c29b https://blog.discord.com/scaling-elixir-f9b8e1e7c29b ). Is it still there or are you using something else now?
- Sikul 6y ago
  Elixir is without a doubt still one of the core languages. We try to use the right tool for the job and Elixir is the right tool for parts of our system. We also combo Elixir NIFs and rust to speed up parts of the Elixir services https://blog.discord.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
  - xtracto 6y ago
    Interesting. We are researching Elixir for an online transaction ledger (thousands of payments per second) along with kafka. So far thats the technology that seems more suitable. That discord article was really appreciated to understand some of the intricacies of using Elixir.
    - fbernier 6y ago
      I certainly know of a few Adtech companies using Erlang to handle > 1M bid requests per seconds :). OTP is pretty great for that.

rufusroflpunch 6y ago

These threads are always full of people criticizing these moves by companies. But nothing is gained of nothing is risked, right? Sometimes these things pay off, and we would never know without early adopters.

rsj_hn 6y ago
The reason why a lot of people criticize moves to rewrite large codebases is because the industry has collected a lot of knowledge and experience with these types of moves and it rarely works out. This is why it's considered a bad practice. We are not babes in the woods, walking around with our eyes newly opened, exploring this strange, wonderous land. The lay of the land is already known in some areas, and this is one such area. Here I would recommend the following twenty year old article by Joel Spolsky, and Joel was merely reciting knowledge that was already well understood, but in an entertaining way. https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...

strangelove026 6y ago

I really enjoyed reading this article last time it was posted. I particularly like it because it shows evidence that current tooling wasn't doing the job. I believe I remember seeing some comments on how Discord was on an old version of go that wasn't receiving the benefit around gc collections, but we'll never know if that would've been a solution. That all said, I've noticed that people here are really off golang in favor of rust. I believe I sort of understand. I've recently switched from python to go at work, and I frequently bring up why I like it so much (caveat I'm overly enthusiastic about the things I like related to my work). But again, that all said, at the end of the day, both Python and golang will definitely get the job done.

dang 6y ago

Discussed at the time: Why Discord is switching from Go to Rust - https://news.ycombinator.com/item?id=22238335 https://news.ycombinator.com/item?id=22238335 - Feb 2020 (640 comments) also https://news.ycombinator.com/item?id=23874336 https://news.ycombinator.com/item?id=23874336 - July 2020 (7 comments)

jancker 6y ago

I think the response time / latency to use Cassandra on the "read state" will be higher, and mind you cassandra is written in java (JVM) and has more garbage collection magic and latency than Go. From my point of view, it's not the language, it's your data store.

AlchemistCamp 6y ago

In its early days, this forum was full of excited young devs who had read PG's old essays and we're actively looking for technologies that would give them a competitive edge. Now, it's completely the opposite. Now, when a company successfully leverages technologies as a competitive edge and becomes a multi-billion dollar success, the thread is dominated by stodgy criticism for not picking the absolute most mainstream technologies possible.

postalrat 6y ago
Saying the right thing has always been easier than saying the correct thing.

vonsydov 6y ago

tl;dr because of garbage collection

mamcx 6y ago

Switching languages is THE most efficient way to refactor a messy codebase. The past color the idea that a language switch is a waste of time, because in the past the choice of languages was from worse to bad or for same to same-ish. But MODERN languages have something that make this compelling: Actual good practiques baked in the language, as defaults. --- So, if a codebase is in trouble you can: - Try to refactor it with "better" idioms... but if this is not the norm in the language/ecosystem you depend in discipline, and that have proved to fail at scale. - Change to "shinny lang" where do the right thing is the default... And now if the developers try to do wrong, them will fight against the current. --- Switch to Go is a great way to solve a project with concurrency issues: Go is good at that, and make a compelling case of build good concurrency apps. Similar if switch to Elixir/Erlang. The case of Rust is similar: it make the "best" practiques the default, and very painful to do wrong (and sometimes to do right, but this is not the focus here :) ). So, a language switch is the ultimate clean up for a project. I have done several in the past (I'm contractor) and let me tell you: Any codebase I get is in deeeeeep trouble. The best results have been a language switch, when I move from "bad" to "better". Just the fact the new language have something that is truly better make the move great. Now, the opposite is true: If you are switching and NOT because the new language have that something that is truly in need or worse, is a language with the need of a LOT of discipline (like C, PHP, JS,...) then yeah, is doomed...

hpen 6y ago

Could something like Discord be written in Swift?

tbodt 6y ago

I've never seen anyone look into just why Go forces a GC on 2 minute intervals. The code originally appeared in https://codereview.appspot.com/5451057 https://codereview.appspot.com/5451057 and there's a bit of discussion on https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E https://groups.google.com/g/golang-nuts/c/WBlJjT-zu7E. After reading this, I still don't understand. But I find https://codereview.appspot.com/5451057#msg38 https://codereview.appspot.com/5451057#msg38 particularly entertaining: there was going to be an environment variable to control the interval, which might have allowed Discord to keep using Go, but Russ Cox shot it down.

caffeine 6y ago

Very misleading title makes it sound like they are rewriting all code wholesale - in fact it's a targeted performance rewrite of a smallish component.