34 ms·

Why Discord is switching from Go to Rust

• unlinked_dll 7y ago
  It'd be cool to look at more signal statistics from the CPU plot. It appears that Go has a lower CPU floor, but it's killed by the GC spikes, presumably due to the large cache mentioned by the author. This is interesting to me. It suggests that Rust is better at scale than Go, and I would have thought with Go's mature concurrency model and implementation would have been optimized for such cases while Rust would shine in smaller services with CPU bound problems. Great post!
  • arnsholt 7y ago
    My first guess for the slightly higher CPU floor of the Rust version is that the Rust code has to do slightly more work per request, since it will free memory as it gets dropped, whereas the Go code doesn't do any freeing per request, but then gets hit with the periodic spike every two minutes where the entire heap has to be traversed for GC.
    • jhgg 7y ago
      tokio 0.1 was definitely less efficient, when we compare go to 0.2, tokio uses less cpu consistently, even when compared to a cluster of the same size almost a year later with our growth over the time since we switched over.
  • pixel_fcker 7y ago
    Go's CPU floor is lower compared to the naive Rust port (roughly 20% vs 23% from eyeballing). Their optimized Rust version is shown in the next series of graphs as being ~12%.
• johnmc408 7y ago
  Non programmer here, but would it make sense to add a keyword (or flag) to Go to manually allocate a piece of memory (ie not use GC). That way, for some use cases, you could use avoid GC for the critical path. Then when GC happened, it could be very fast as there would be far less to pause-and-scan (in this use case example). Obviously this would have to be optional and discouraged...but there seems to be no way to write an intensive real-time app with a GC based language. (again non-programmer that is writing this to learn more ;-)
  • vips7L 7y ago
    So like D? https://dlang.org/spec/attribute.html#nogc https://dlang.org/spec/attribute.html#nogc
    • bluebasket 7y ago
      i thought go has a GOGC=off option or something? did they remove it?
      • wwarner 7y ago
        no it's there. https://golang.org/pkg/runtime/debug/#SetGCPercent https://golang.org/pkg/runtime/debug/#SetGCPercent
  • oconnor663 7y ago
    There are two things you'd have to do at the same time that make this complicated: - You'd have to ensure that your large data structure gets allocated entirely within the special region. That's simple enough if all you have is a big array, but it gets more complicated if you've got something like a map of strings. Each map cell and each string would need to get allocated in the special region, and all of the types involved would need new APIs to make that happen. - You'd have to ensure that data structures in your special region never hold references to anything outside. Since the whole point of the region is that the GC doesn't scan it, nothing in the region will be able to keep anything outside the region alive. Any external references could easily become dangling pointers to freed memory, which is the sort of security vulnerability that GC itself was designed to prevent. All of this is doable in theory, but it's sufficiently difficult, and it comes with sufficiently many downsides, that it makes more sense for a project with these performance needs to just use C or Rust or something.
    • zozbot234 7y ago
      > Since the whole point of the region is that the GC doesn't scan it, nothing in the region will be able to keep anything outside the region alive. You can treat external references as GC roots.
• carllerche 7y ago
  Tokio author here (mentioned in blog post). It is really great to see these success stories. I also think it is great that Discord is using the right tool for the job. It isn't often that you need the performance gains that Rust & Tokio so pick what works best to get the job done and iterate.
  • joseluisq 7y ago
    Basically because of: > Rust is blazingly fast and memory-efficient: with no runtime or garbage collector, it can power performance-critical services, run on embedded devices, and easily integrate with other languages.
  • Polyisoprene 7y ago
    No offense to Tokio and Rust, I really like Rust, but having someone rewriting their app because of performance limitations in their previous language choice, isn’t really someone picking the right tool for the job necessary. I’m not so sure they would have done the rewrite if the Go GC was performing better, and the choice of Rust seems primarily based on prior experience at the company writing performance sensitive code rather than delivering business value.
    • acheron9383 7y ago
      Right tool for the job should also take into account the experience of the devs you have at your disposal. For an omniscient Dev, is Rust the best tool for the job? Unsure. But for them with already significant rust experience? Sounds like it.
    • qaq 7y ago
      too much focus on "business value" often ends-up with codebase in a state that makes delivery of that business value pretty impossible. Boeing was delivering a lot of business value with MAX ...
      • deleted 7y ago
        [deleted]
• _--___-___ 7y ago
  "We want to make sure Discord feels super snappy all the time" is hilarious coming from a program that is infamous for making you read 'quirky' loading lines while a basic chat application takes several seconds to start up. Don't really know about Go versus Rust for this purpose, but don't really care because read states (like nearly everything that makes Discord less like IRC) is an anti-feature in any remotely busy server. Anything important enough that it shouldn't be missed can be pinned, and it encourages people to derail conversations by replying out of context to things posted hours or days ago.
  • anchpop 7y ago
    I don't see why that's hilarious. Lots of programs take a second or two to load and it only happens once on boot for me. "Read states" is just discord telling you which channels and servers you have unread messages in
    • wvenable 7y ago
      Discord takes longer to start up than Microsoft Word. Desktop development is a total wasteland these days -- there isn't nearly as much effort put into optimization as server side. They're not paying for your local compute, so they can waste as much of it as they want.
      • penagwin 7y ago
        I feel that it's not really fair to expect them to natively implement their app on every platform and put tons of resources into it's client performance - anecdotally discord is a very responsive app - see [0]. But think of it this way, all the effort they put into their desktop app works on all major OSes without a problem. They even get to reuse most of the code for access from the browser, with no installation required. Now imagine approaching your PM and saying "Look I know we put X effort into making our application work on all the platforms, but it would be even faster if we instead did 4x effort for native implementations + the browser". [0] From what I've seen in the "gamer community" is that most gamers don't care that much about that kind of extra performance. Discord itself doesn't feel slow once it's started. Joining a voice channel is instant, quickly switching to a different server and then to a chat channel to throw in a some text is fast and seamless (Looking at you MS Teams!!!). Sure Mumble/Teamspeak are native and faster, but where are their first party mobile apps and web clients? One of the incredible things Discord did to really aid in adoption was allow for web clients, so when you found some random person on the internet, you didn't have to tell them to download some chat client, they could try it through their browser first. tl;dr Yes electron apps can be slow, but discord IMO has fine client side performance, and they clearly do put resources into optimizing it. Yes it "could be faster" with native desktop apps, but their target community seems perfectly content as is.
• StreamBright 7y ago
  Pretty amazing write up from Jesse. I really like how they maxed out Go first before even thinking about a rewrite in Rust. It turns out no-GC has pretty significant advantages in some cases.
  • Rapzid 7y ago
    Unsafe or it doesn't count ;)
• correct_horse 7y ago
  I've heard lots of hot takes on "what Go really is". Here's mine. Go is what would have happened if Bell Labs wrote Java.
  • kick 7y ago
    Minor nitpick: That already happened, Limbo is what happened when Bell Labs wrote Java.
    • monocasa 7y ago
      And go is very very derived from plan 9. It could be considered a sibling of limbo in a lot of ways.
    • anthk 7y ago
      More like Limbo and Inferno.
      • kick 7y ago
        Limbo doesn't only run on Inferno; anything with the Dis VM will work.
    • deleted 7y ago
      [deleted]
    • correct_horse 7y ago
      Huh. I managed to hear about Inferno, but not remember the Limbo part. In that case, Go is Bell Labs' second attempt at Java.
      • kick 7y ago
        Third, there was also a language I can't remember the name of that happened at the same time as Alef.
        • steveklabnik 7y ago
• buboard 7y ago
  maybe next year : why discord is switching to C
• flafla2 7y ago
  > After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collection has not run for 2 minutes, regardless of heap growth, go will still force a garbage collection. > We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change the garbage collector GC Percent on the fly. Unfortunately, no matter how we configured the GC percent nothing changed. How could that be? It turns out, it was because we were not allocating memory quickly enough for it to force garbage collection to happen more often. As someone not too familiar with GC design, this seems like an absurd hack. That this 2-minute hardcoded limitation is not even configurable comes across as amateurish even. I have no experience with Go -- do people simply live with this and not talk about it?
  • JBReefer 7y ago
    Go always feels like an amateur language to me, I’ve given up on it. This feels right in line - similar to the hardcode GitHub magic.
    • reificator 7y ago
      I could be wrong, but I don't believe there is "hardcoded[d] GitHub magic". IIRC I have used GitLab and Bitbucket and self-hosted Gitea instances the same exact way, and I'm fairly sure there was an hg repo in one of those. Don't recall doing anything out of the ordinary compared to how I would use a github URL.
      • heinrich5991 7y ago
        There are a couple of hosting services hardcoded in Go. I believe it was about splitting the URL into the actual URL and the branch name.
        • Nullabillity 7y ago
          https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8349cba129d6d3a/src/cmd/go/internal/get/vcs.go#L1022 https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8... Ouch, Go never ceases to amaze. The Bitbucket case[0] is even more crazy, calling out to the Bitbucket API to figure out which VCS to use. It has a special case for private repositories, but seems to hard-code cloning over HTTPS. If only we had some kind of universal way to identify resources, that told you how to access it... [0]: https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8349cba129d6d3a/src/cmd/go/internal/get/vcs.go#L1113 https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8...
• justadudeama 7y ago
  > Changing to a BTreeMap instead of a HashMap in the LRU cache to optimize memory usage. Can someone explain to me how BTreeMap is more memory efficient than a HashMap?
  • jhgg 7y ago
    This is a bit unclear. The root map is still a hash map, but it's a "map of maps" the inner map is a BTreeMap - this is for memory efficiency, as the inner map is relatively smaller and we wouldn't have to deal with the growth factor of a hash map (and having to manually manage that.) where as the root hash map is pre allocated to its max size.
  • afranchuk 7y ago
    A BTreeMap should typically have O(n) memory usage, whereas a HashMap (depending on load factor) will usually have O(kn) memory usage, where k > 1. This is because a HashMap allocates the table into which it will store hashed values upfront (and when the load is too great), so it can't anticipate how many values may be added nor what sorts of collisions may occur at this time. Yes, collisions are typically stored as some allocate-per-item collection, but the desire of a HashMap is to avoid such collisions. A BTreeMap allocates for each new value. Note that this explanation is a bit handwavy, as both data structures have numerous optimizations in production scenarios.
    • cesarb 7y ago
      > collisions are typically stored as some allocate-per-item collection Rust's HashMap stores the collisions in the same table as the non-collisions (open addressing), not in a separate collection.
      • afranchuk 7y ago
        This is true, thanks for the specifics. I was answering the question from a more generic perspective, but failed to mention that many implementations rehash on collision...
    • 7y ago
• mperham 7y ago
  Better title: "One Discord microservice with extremely high traffic is moving to Rust"
  • jhgg 7y ago
    This is one of multiple, we did not blog about this one, but switching a Python http service for analytics ingest that was purely CPU bound to rust resulted in a 90% reduction in compute required to power it. However, that's not too interesting because it's known that Python is slow haha. We have 2 golang services left, one of them has a rewrite in rust in PR as of last week (as a fun side project an engineer wanted to try out.) Additionally, as we move towards a more SOA internally, we plan to write more high velocity data services, and rust will be our language of choice for that.
    • okgood288 7y ago
      Well sure when it’s a micro service that probably has more lines of infra config than biz logic LOC. This isn’t exactly “Linux kernel: now in Rust!” Glad you’re making tech for you all better. We get to take up the externalized runtime costs of the mess that is the Electron app. Engineers are super efficient at offloading the last mile of effort.
      • snazz 7y ago
        Let's not start the Electron debate again. That's been argued to death already.
        • deleted 7y ago
          [deleted]
        • jodrellblank 7y ago
          Maybe if we keep at it, we can argue it until Electron's death?
    • onebot 7y ago
      Think replacing elixir with Rust would ever be a consideration? Rust isn't there yet, but if you are NIF'ing a bunch of stuff, seems like it could make sense at some point?
• kardianos 7y ago
  I'm glad they found a good solution (rust) to solve their problem! Also note this was with Go1.9. I know GC work was ongoing during that time, I wonder if this time of situation would still happen?
  • biomcgary 7y ago
    I know latency for GC with large heaps improved in Go1.12. See: https://golang.org/doc/go1.12#runtime https://golang.org/doc/go1.12#runtime
    • kerkeslager 7y ago
      The article states specifically that part of the problem was heaps were never large. EDIT: Actually, no it didn't, I misunderstood it.
      • typical182 7y ago
        Where does it say that? It says things like: “We were not creating a lot of garbage.” ... but that statement there doesn’t say anything about the heap size, including the size and count of live objects (i.e., not garbage). It also says: “There are millions of Users in each cache. There are tens of millions of Read States in each cache.” Large is often in the eye of the beholder, but I missed it if it said anything specifically about not having a large heap size.
        • kerkeslager 7y ago
          > ... but that statement there doesn’t say anything about the heap size, including the size and count of live objects (i.e., not garbage). Not sure why you got downvoted, you're actually right, I'm wrong: I misread that and/or assumed one meant the other. That said, this is a case that should be ideal for generational GC, which Go specifically eschewed at one point. I'm not sure this is still the case, however--I have yet to wade through this[1] to update my knowledge here. [1] https://blog.golang.org/ismmkeynote https://blog.golang.org/ismmkeynote
• RcouF1uZ4gsC 7y ago
  > Changing to a BTreeMap instead of a HashMap in the LRU cache to optimize memory usage. Collections are one of the big areas where Go's lack of generics really hurts it. In Go, if one of the built in collections does not meet your needs, you are going to take a safety and ergonomic hit going to a custom collection. In Rust, if one of the standard collections does not meet your needs, you (or someone else) can create a pretty much drop-in replacement that does that has similar ergonomic and safety profiles.
  • correct_horse 7y ago
    I'm not sure what you mean by standard collections, but BTreeMap is in Rust's standard library.
    • pdpi 7y ago
      I think the point the GP is trying to make is that there’s no reason why BTreeMap couldn’t be an external crate, while only the core Go collections are allowed to be generic. A corollary to this is that adding more generic collections to Go’s standard library implies expanding the set of magical constructs.
      • The_rationalist 7y ago
        Rust has it's lot of weird hacks too. E.g array can take traits impls only if they have less than 32 elements... https://doc.rust-lang.org/std/array/trait.LengthAtMost32.html https://doc.rust-lang.org/std/array/trait.LengthAtMost32.htm...
        • jolux 7y ago
          That's a completely different and much more minor issue (red herring, more or less) than eschewing the one core language feature that makes performant type-safe custom data structures possible.
        • steveklabnik 7y ago
          This is purely temporary; it used to be less hacky, but in order to move to the no-hacks world, we had to make it a bit more hacky to start.
• bradhe 7y ago
  Replatforming to solve this problem was a bit silly in my opinion. The solution to the problem was "do fewer allocations" which can be done in any language.
  • pjmlp 7y ago
    Yeah, but this way one can add experience in yet another language to the CV.
  • jhgg 7y ago
    Your reply misses the point. We were already doing so few allocations that the GC only ran because it "had to" at every 2 minute mark. The issue was the large heap of many long lived objects.
    • _ph_ 7y ago
      Did you try to change that interval to a much larger time?
      • jhgg 7y ago
        When we investigated, there was no way to change that that we could find - barring compiling go from source (something we could have done, but wanted to avoid.)
        • _ph_ 7y ago
          Yes, you have to rebuild go, but that is literally done in a minute. It also would be interesting, if you happen to have some conclusive benchmarks, how the latest Go runtime would perform in this sense.
          • _ph_ 7y ago
            I don't get, why this is downvoted without comments. Compared to a rewrite, this would have been a miniscule change. Furthermore, considering that you wrote that long blog post (which I quite appreciate, as it contains interesting information), it would have been important knowledge, whether the setting of the parameter was the real culprit - and if it was, a good reason to shout out to the Go implementors to look closer at it.
• h2odragon 7y ago
  Excellent write up, and effective argument for Rust in this application and others. My cynical side sums it up as: "Go sucked for us because we refused to own our tooling and make a special allocator for this service. Switching to Rust forced us to do that, and life got better"
  • staticassertion 7y ago
    I'm confused. Build a special allocator for Go you mean? That feels like going well beyond typical "own your tooling".
    • h2odragon 7y ago
      I'm outdated. I used to have 4 different python interpreter builds, for different purposes, where the modern world would be using lua as a glue language. I had nothing like the scale, staff, or budget of Discord; all I had was need and tools that could bend to fill it. I think this is a great write up of why they chose a different tool. I don't say it was the wrong decision, they make that argument pretty well too. I'm still surprised that either Go isn't malleable enough to have bent around the need, or they didn't feel it worth more effort than parameter tweaking to bend it so.
  • monocasa 7y ago
    They were already not allocating, they were just stuck with a GC cycle that'd scan, not find any garbage, and scan again in two minutes.
• thedance 7y ago
  These kinds of posts would be much more interesting if they discussed alternatives considered and rejected. For example why did they choose Rust over C++?
  • therockhead 7y ago
    The article mentioned that they have already used Rust successfully in house, so when you consider that Rust is inherently safer than C++, it seems like they picked the right language.
  • The_rationalist 7y ago
    The most pressing undiscussed alternative is: why didn't they update their 3 years old Go version yet had the double standard of using rust nightly... This blog post is a scam and their only reason to use rust should be assumed: it's because they wanted to.
    • thedance 7y ago
      That's exactly the trolling I was looking for :-) . It just reads like they thought it would be cool to use Rust, so they did, which is fine.
    • deleted 7y ago
      [deleted]
• _bxg1 7y ago
  It's always good to see a case-study/anecdote, but nothing in here is surprising. It also doesn't really invalidate Go in any way. Rust is faster than Go. People use Go, like any other technology, when the tradeoffs between developer iteration/throughput/latency/etc. make sense. When those cease to make sense, a hot path gets converted down to something more efficient. This is the natural way of things.
  • kerkeslager 7y ago
    > It's always good to see a case-study/anecdote, but nothing in here is surprising. It also doesn't really invalidate Go in any way. Well, sure, because categorizing languages as "valid/invalid" doesn't make any sense. But it does show yet another example of how designing a language to solve Google's fairly-unique problems doesn't result in a general-purpose language suitable for solving most people's problems.
    • kikimora 7y ago
      Long GC pauses caused by large collections/caches are decade long problem with no real wide spread solution so far. With Java and .NET you can resort to off-heap data. Not sure if this is possible with Go.
      • kerkeslager 7y ago
        > Long GC pauses caused by large collections/caches are decade long problem with no real wide spread solution so far. This is arguable, but the fact is that "large collections/caches" isn't Discord's situation.
      • dnautics 7y ago
        Erlang's basically solved it (and, arguably, solved it decades ago); relevant as discord uses erlang VM in places.
        • bsder 7y ago
          Erlang "solved" the problem by breaking having lots of little heaps so a GC can blast through the entire heap extremely quickly. Would that actually work in this instance? It seems like that LRU cache they're talking about is kind of large.
• highfrequency 7y ago
  Curious about their definition of “response time” in the graph at the end. They’re quoting ~20 microseconds so I assume this doesn’t involve network hops? Is this just the CPU time it takes a Read State server to do one update?
  • jhgg 7y ago
    Correct. This is internal time it takes to process the message. Since once a node is "warm" thanks to their large caches, it's mostly in memory operations and queueing for persistence which happens in the background.
    • Sikul 7y ago
      Also worth noting: Most requests to the service have to update many Read States. For instance, when you @everyone in the Minecraft server we have to update over 500,000 Read States.
• joseluisq 7y ago
  That's why the {blazing-fast} term is becoming popular. Rust won again.
• mangatmodi 7y ago
  Why would they switch to rust, rather than upgrading from 3 years old version?
  • jhgg 7y ago
    This blog post perhaps is a bit "after the fact" we had made the switch over mid 2019, and wanted to try out rust as well for services like this, due to adoption elsewhere in the company. Also, after upgrading between 4 golang versions on this service and noticing it didn't materially change performance, we decided to just spend our time on the rewrite (for fun, and latency) and to get a head start into the asynchronous rust ecosystem. This blog post kinda internally matches our upgrade to std::futures and tokio 0.2, away from futures 0.1.
    • The_rationalist 7y ago
      Out of curiosity, why didn't you choose Kotlin? It can reuse the Java ecosystem which allow you to save tons of money, and give you advanced features and scalability. It is a sexier and more ergonomic language too. And with e.g ZGC, you can have a GC that is fine tunable, and that has very low latency. By choosing rust you will suffer a great deal of the limitations of it's poor, not production ready, ecosystem. I'm not even talking about the immaturity of the async await support.
      • therockhead 7y ago
        > By choosing rust you will suffer a great deal of the limitations of it's poor, not production ready, ecosystem. Why do you think that? Seems like Rust is a great choice for this type of high performance work.
        • ncmncm 7y ago
          Rust does best when the number of lines of code that must be parsed in an edit-compile-test loop is small. When the sources that must be parsed get large, coders suffer. It is doubtful that this will improve, much, without breaking changes to the language. The range of code over which type inference operates, or at least programmers' reliance on it, would need to contract by quite a lot. There would be Complaints.
• donatj 7y ago
  I feel like from the definition of the service, the entire thing could easily be replaced with a Redis cluster.
  • deleted 7y ago
    [deleted]
  • Sikul 7y ago
    We originally cached this data with a Redis cluster but we hit scaling issues. The Read States service only exists because Redis had issues.
    • donatj 7y ago
      Hah, well now I feel like a dufus. Good info!
      • Sikul 7y ago
        No worries, we could have mentioned that in the post as part of the service history :)
    • hopia 7y ago
      We use Elixir too and ironically, Redis became our bottleneck as well. What a useless dependency it is when running on BEAM.
• _ph_ 7y ago
  If you have a problem at hand which does not really benefit from the presence of a garbage collector, switching to an implementation without a garbage collector has quite a potential to be at least somewhat faster. I remember myself to run onto this time trigger for garbage collection long in the past - though I don't remember why and mostly forgot about ever since until I read this article. As also written in the article, even if there are no allocations going on, Go forces a gc every two minutes, it is set here: https://golang.org/src/runtime/proc.go#L4268 https://golang.org/src/runtime/proc.go#L4268 The idea for this is (if I remember correctly) to be able to return unused memory to the OS. As returning memory requires a gc to run, it is forced in time intervals. I am a bit surprised that they didn't contact the corresponding Go developers, as they seem to be interested in practical use cases where the gc doesn't perform well. Besides that newer Go releases improved the gc performance, I am a bit surprised that they didn't just increase this time interval to an arbitrary large number and checked, if their issues went away.
  • KMag 7y ago
    Not only is there good potential for a speed improvement, but languages built around the assumption of pervasive garbage collection tend not to have good language constructs to support manual memory management. To be fair, most languages without GCs also don't have good language constructs to support manual memory management. If you're going to make wide use of manual memory management, you should think very carefully about how the language and ecosystem you're using help or hinder your manual memory management.
• geodel 7y ago
  Makes sense write most efficient stuff for in-house and give resource hog Electron apps to users.
  • jrockway 7y ago
    Discord pays for their servers, but not for their users's computers.
    • Hamuko 7y ago
      That's fine as long as you ignore the fact that the users are the customers.
• deleted 7y ago
  [deleted]
• nemo1618 7y ago
  I wonder if it would be feasible to rewrite the LRU cache (either fully or in part) in a way that does not require the GC to scan the entire cache.
  • kerkeslager 7y ago
    Yes, it's possible: that's generational garbage collection. But last I heard, Google decided writing a modern GC was too complicated. They're probably right, because Google doesn't need it. But for everyone else who decided to use a language designed to solve Google's fairly-unique problems as if it were a general-purpose language: that kind of sucks, doesn't it?
    • terminaljunkid 7y ago
      The fact seems to be that the go team is not so well funded as it seems. Go is not Google's language in the sense C# is MS' language or Java was Sun language.
  • ssoroka 7y ago
    Check out this post, that describes exactly that process. https://blog.gopheracademy.com/advent-2018/avoid-gc-overhead-large-heaps/ https://blog.gopheracademy.com/advent-2018/avoid-gc-overhead...
• the-alchemist 7y ago
  Looks like the big challenge is managing a large, LRU cache, which tends to be a difficult problem for GC runtimes. I bet the JVM, with its myriad tunable GC algorithms, would perform better, especially Shenandoah and, of course, the Azul C4. The JVM world tends to solve this problem by using off-heap caches. See Apache Ignite [0] or Ehcache [1]. I can't speak for how their Rust cache manages memory, but the thing to be careful of in non-GC runtimes (especially non-copying GC) is memory fragmentation. Its worth mentioning that the Dgraph folks wrote a better Go cache [2] once they hit the limits of the usual Go caches. From a purely architectural perspective, I would try to put cacheable material in something like memcache or redis, or one of the many distributed caches out there. But it might not be an option. It's worth mentioning that Apache Cassandra itself uses an off-heap cache. [0]: https://ignite.apache.org/arch/durablememory.html https://ignite.apache.org/arch/durablememory.html [1]: https://www.ehcache.org/documentation/2.8/get-started/storage-options.html#bigmemory-(off-heap-store) https://www.ehcache.org/documentation/2.8/get-started/storag... [2]: https://blog.dgraph.io/post/introducing-ristretto-high-perf-go-cache/ https://blog.dgraph.io/post/introducing-ristretto-high-perf-...
  • pshc 7y ago
    Great comment and thanks for the reading material. Now I'm wondering if there's a Rust library for a generational copying arena--one that compacts strings/blobs over time.
    • steveklabnik 7y ago
      Generational arenas yes, but copying, I'm not aware of one. It's very hard to get the semantics correct, since you can't auto-re-write pointers/indices.
      • ithkuil 7y ago
        Perhaps such a library could help you record the location of the variables that contain pointers to the strings and keep that pointer up to data as the ownership of the string moves from variable to variable? I'm other words, doing some of the work a moving compacting collector would do during compaction but continuously during normal program execution.
• adamnemecek 7y ago
  Rust is maturing. I legit don't think there are too many good reasons to use Go over Rust. You can call Rust from Go but not vice versa.
  • steveklabnik 7y ago
    (You can call Go from Rust: https://blog.arranfrance.com/post/cgo-sqip-rust/ https://blog.arranfrance.com/post/cgo-sqip-rust/ )
• kerkeslager 7y ago
  Go is not a general-purpose language. It's a Google language designed to solve Google's problems. If you aren't Google, you probably have different problems, which Go isn't intended to solve. EDIT: Currently at -4 downvotes. Would downvoters care to discuss their votes?
  • Corrado 7y ago
    I agree. One of Go's design goals was to be simple enough for thousands of developers to use it simultaneously across a huge monorepo. To me this is in the same class as companies use k8s; unless your Google (or Facebook or Netflix ...) you probably shoudn't be using it.
    • kerkeslager 7y ago
      > To me this is in the same class as companies use k8s; unless your Google (or Facebook or Netflix ...) you probably shoudn't be using it. I'll actually even say, that if you're Facebook or Netflix, you still shouldn't use Go, because you can write your own tools that solve your problems.
  • cmrdporcupine 7y ago
    As a Googler, I don't consider this accurate. I've been here 8 years and have yet to work on a Go code base. Yes, there are projects in Go. Certainly not a majority, nor even a significant minority, honestly. No, I wouldn't say Go is specific to Google's problems, though I'm sure some of the engineers had them in mind. I see Go used far more outside of Google than in.
    • takeda 7y ago
      Isn't that indication of a failure? It seems like Go aimed to replace Python and Java code at Google.
      • cmrdporcupine 7y ago
        My impression (and this was pre-Google and I haven't paid attention since I got here, so) is that it was Rob Pike's and Ken Thompson's project coming out of their long experience with Plan 9 and Inferno/Limbo. That it happened to meet some requirements for some Google projects -- I'm sure that might have been an intent. But that feels a bit like an explanation after the fact, since Go very obviously shows the biases and philosophy from the projects that the original authors had in their previous work.
• mister_hn 7y ago
  Why not C++, if performance was an issue?
  • wmf 7y ago
    Why C++? Why would you want the same performance as Rust with less safety?
  • loeg 7y ago
    Why would you pick C++ for a new codebase in 2019 or 2020 if Rust met your needs?
    • terminaljunkid 7y ago
      Programmer productivity Library support
    • nuclx 7y ago
      Compilation times.
      • Narishma 7y ago
        In my experience, C++ is slower to compile than Rust.
  • bluGill 7y ago
    Modern C++ is the right choice if you have an existing code base in C++, or you need to use features that only exist in a third party C++ library - there is a large collection of C++ libraries to choose from. Their use case doesn't seem to have either consideration (note that even when these are considerations a hybrid of languages is often a good idea) so there isn't a compelling reason to choose C++. That doesn't mean C++ is wrong, just that there is nothing wrong with rust. Maybe a great C++ programmer can get a few tenths of a percent faster code (mostly because compiler writers spend more effort figuring out how to optimize C++ - rust uses the same llvm optimizer but it might sometimes do something less optimal because it assumed C++ input), but in general if the difference matters in your environment you are too close to the edge and need to scale. Rust might be easier/faster to write than modern C++. If so that is a point in favor of rust. They seem to have people who know rust, which is important. There might be more people who know C++, but I can take any great programmer and make them good in any programming language in a few weeks in the worst case (worst case would be writing a large program in intercal or some such intentionally hard language) - not to be confused with expert which takes more experience.
• tiffanyh 7y ago
  It should also be noted that Rust interoperates extremely well with Erlang, which is the basis of Discord (via Rustler). https://github.com/rusterlium/rustler https://github.com/rusterlium/rustler https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b https://blog.discordapp.com/scaling-elixir-f9b8e1e7c29b
• The_rationalist 7y ago
  Borrowed from a comment: Garbage collection has gotten a lot of updates in the last 3 years. Why would you not take the exceedingly trivial step of just upgrading to the latest Go stable in order to at least try for the free win? From the go 1.12 release notes: “Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live. This reduces allocation latency immediately following a garbage collection.” ¯\_(ツ)_/¯ This sounds like “we just wanted to try Rust, ok?” Which is fine. But like, just say that.
• jrockway 7y ago
  This seems like a nice microservices success story. It's so easy to replace a low-performing piece of infrastructure when it is just a component with a well-defined API. Spin up the new version, mirror some requests to see how it performs, and turn off the old one. No drama, no year-long rewrites. Just a simple fix for the component that needed it the most.
  • thijsvandien 7y ago
    You don't need microservices for that, though. One might as well have moved that piece into a library.
    • kccqzy 7y ago
      And then deal with cross-language FFI boundaries and cross-language builds.
      • SlowRobotAhead 7y ago
        This is what clicked for me on microservices years back. That the language wasn’t important and if I couldn’t do it in python or C, someone else could in Go or Java or etc. Compared to if I wrote something in house entirely in C... lolno
        • Tomis02 7y ago
          Landing in a shop that uses N programming languages for N microservices would be a pretty miserable experience.
          • tcbasche 7y ago
            At a previous job we used Python for all microservices, except for 'legacy' systems which were in Groovy / Rails. That was a context switch if I ever experienced one.
          • koffiezet 7y ago
• echopom 7y ago
  This was an extremely interesting read. I'm quiet disappointed though they did not update their Go Version to 1.13[0][1] which would normally have remove the spike issue and thus he latency before they move to Rust... Rust seems more performant with proper usage ( tokio + async ) but I'm more worried about the ecosystem that doesn't seem has mature has Go. We could quote the recent[2] Drama with Actix... [0]https://golang.org/doc/go1.13#runtime https://golang.org/doc/go1.13#runtime [1]https://golang.org/doc/go1.12#runtime https://golang.org/doc/go1.12#runtime [2]https://github.com/fafhrd91/actix-web-postmortem https://github.com/fafhrd91/actix-web-postmortem
  • chc 7y ago
    Why would you want to bring up the Actix author's drama? That doesn't seem like something that should reflect on a language one way or the other.
    • deweller 7y ago
      As an outsider to both the Go and Rust cultures, I read the Actix news and walked away with the impression that the Rust ecosystem is less mature.
      • faitswulff 7y ago
        Every community has it. The dep vs. vgo drama gave me the same impression of Go at the time: https://news.ycombinator.com/item?id=17063724 https://news.ycombinator.com/item?id=17063724
      • cies 7y ago
        Go's is more pragmatic. Rust's is more purist, and that reflects on the language features (more functional, more free in allowing you to use it for any purpose where Go is network-app specific, more strict in typing), the licensing and the attitude towards collaboration. That collaboration thing is why Actix exploded I think. While mostly an isolated incident it does show some clash between the author's values (and possibly the author's employer's (MSFT) values) and the values of the general Rust community. I would not say that reflects on the maturity of the langues or ecosystem. In Go a lot of stuff is Google dictated. In Rust it's a true open governance innovation project (looking to become a non-profit). Since the Go is a very specific language --made for networked apps and only has one way to do concurrency-- and Rust very broad --a true general purpose prog lang-- it is easy to see how Go mature so quickly (not much to mature) and also why it got a bit old so quickly as well (ignores most innovations in computer science of the last decades).
• reggieband 7y ago
  When I see this kind of GC performance, I wonder why you wouldn't change the implementation to use some sort of pool allocator. I am guessing each Read State object is identical to one another (e.g. some kind of struct) so why not pre-allocate your memory budget of objects and just keep an unused list outside of your HasMap? In a way this is even closer to a ring where upon ejection you could write the object to disk (or Cassandra), re-initialise the memory and then reuse the object for the new entry. I suppose that won't stop the GC from scanning the memory though ... so maybe they had something akin to that. I assume that a company associated with games and with some former games programmers would have thought to use pool allocators. Honestly, if that strategy didn't work then I would be a bit frustrated with Go. I have to say, out of all of the non-stop spamming of Rust I see on this site - this is definitely the first time I've thought to myself that this is a very appropriate use of the language. This kind of simple yet high-throughput workhorse of a system is a great match for Rust.
  • monocasa 7y ago
    Yeah, they already weren't allocating, it was a GC pause that just scanned and would come up with essentially no extra garbage every two minutes.
    • azakai 7y ago
      A pool allocator could have reduced the number of existing allocations (1 big one instead of many small ones), making those spikes less significant. (But that depends on how Go handles interior pointers and GC, so I'm not sure.)
      • runevault 7y ago
        Allocations weren't the problem. It was the fact that, every 2 minutes, the GC would trigger because of an arbitrary decision by the Go team and scan their entire heap, find little to nothing to deallocate, then go on its merry way.
      • monocasa 7y ago
        It has to track interior pointers. The problem for them seems to be the mark phase, where the GC has to track down all the pointers either way.
• karma_daemon 7y ago
  I wish the article would show a graph of the golang heap usage. I'm reminded of this cloudflare article [0] from a while back where they created an example that seemed to exhibit similar performance issues when they created many small objects to be garbaged collected. They solved it by using a pooled allocator instead of relying solely on the GC. Wonder if that would have been applicable here to the go version. [0] https://blog.cloudflare.com/recycling-memory-buffers-in-go/ https://blog.cloudflare.com/recycling-memory-buffers-in-go/
• jaten 7y ago
  just use an off heap hash table. simple. https://github.com/glycerine/offheap https://github.com/glycerine/offheap Also, as others have said, lots of big GC improvements were ignored by insisting on go1.9.2 and not the latest.
  • favorited 7y ago
    The graphs are from 1.9.2, but the author said they tried 1.8, 1.9, and 1.10 and saw the same thing.
• rvcdbn 7y ago
  Seems like you were hitting: runtime: Large maps cause significant GC pauses #9477 [0] Looks like this issue was resolved for maps that don't contain pointers by [1]. From the article, sounds like the map keys were strings (which do contain pointers, so the map would need to be scanned by the GC). If pointers in the map keys and values could be avoided, it would have (if my understanding is correct) removed the need for the GC to scan the map. You could do this for example by replacing string keys with fixed size byte arrays. Curious if you experimented this approach? [0] https://github.com/golang/go/issues/9477 https://github.com/golang/go/issues/9477 [1] https://go-review.googlesource.com/c/go/+/3288 https://go-review.googlesource.com/c/go/+/3288
  • ryuukk_ 7y ago
    exactly, they use 3 years old version with the improvements made to runtime and that issue fixed, then is safe to say that GO is much faster than rust, based on their graphs
  • jasondclinton 7y ago
    Finding out if that does resolve the author's issue would be interesting but I'm not sure that that would be particularly supportive data in favor of Go. If anything it would reinforce the downsides of Go's GC implementation: prone sudden pitfalls only avoidable with obtuse, error-prone fiddling that makes the code more complex. After spending weeks fighting with Java's GC tuning for a similar production service tail latency problem, I wouldn't want to be caught having to do that again.
    • masklinn 7y ago
      The good news are that Go's GC has basically no tunables, so you wouldn't have spent weeks on that. The bad news is that it has basically no tunables so if it's a tuning issue you're either fucked or have to put "tuning" hacks right into the code if you find any that works (e.g. twitch's "memory ballast" to avoid overly aggressive GC runs: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...)
• Thaxll 7y ago
  Really interesting post, however they're using a 2+years old runtime, Go 1.9.2 was released 2017/10/25 why did they not even try Go 1.13? For me the interesting part is that their new implementation in Rust with a new data structure is less than 2x faster than an implementation in Go using a 2+years old runtime. It shows how fast Go is vs an very optimized language + new data structure with no GC. Overall I'm pretty sure there was a way to make the spikes go away. Still great post.
  • yazaddaruvala 7y ago
    The graphs were in different units. The final Rust version was over 100x faster.
    • Thaxll 7y ago
      Which doesn't make any sense. Rust is not x100 faster than Go.
      • yazaddaruvala 7y ago
        Rust and Go likely translate into similar enough assembly for similar code to make the performance close enough. However, bigger caches will always have more cache hits than smaller caches. Therefore could easily be 100x faster. The blog does a better job explaining everything than I can but simply put the “granular” memory management Rust allows gave them an improved ability to create a bigger cache. Go (at the time) while great did not work well for that particular usecase and required smaller cache sizes.
• blackrock 7y ago
  Would it have been better if they went with Elixir? Write their code in a functional style. Get the benefits of the Erlang BEAM platform. Their system runs over the web, so time sensitivity isn’t as important, in comparison to video games, VR, or AR. Anyone ever done a performance comparison breakdown between something like Elixir vs. Rust?
  • steveklabnik 7y ago
    Discord is a heavy Elixir user, and even uses it with Rust via NIF: https://blog.discordapp.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 https://blog.discordapp.com/using-rust-to-scale-elixir-for-1...
  • jerf 7y ago
    "Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. As a language, it is much slower than Rust. Plus, if you tried to implement a gigantic shared cache map in Erlang/Elixir, you'd have two major problems: One is that you'd need huge chunks of the map in single (BEAM) processes, and you'd get hit by the fact BEAM is not set up to GC well in that case. It wants lots of little processes, not a small number of processes holding tons of data. Second is that you'd be trading what in Rust is "accept some bytes, do some hashing, look some stuff up in memory" with generally efficient, low-copy operations, with "copy the network traffic into an Erlang binary, do some hashing, compute the PID that actually has the data, send a message to that PID with the request, wait for the reply message, and then send out the answer", with a whole lot of layers that expect to have time to make copies of lots of things. Adding this sort of coordination into these nominally fast lookups is going to slow this to a crawl. It's like when people try to benchmark Erlang/Elixir/Go's threading by creating processes/goroutines to receive two numbers and add them together "in parallel"; the IPC completely overshadows the tiny amount of work being done. (They mention tokio, but that's still going to add a lot less coordination overhead than Erlang messages.) Go is a significantly better language for this use case than Elixir/Erlang/BEAM is, let alone Rust. (This is not a "criticism" of Erlang/Elixir/BEAM. It's an engineering analysis. Erlang/Elixir/BEAM are still suitable for many tasks, just as people still use Python for many things despite the fact it would be a catastrophically bad choice for this particular task. This just isn't one of the tasks it would be suitable for.)
• dennisgorelik 7y ago
  > Changing to a BTreeMap instead of a HashMap in the LRU cache to optimize memory usage. Why would BTreeMap be faster than HashMap? HashMap performance is O(1), while BTreeMap performance is O(log N).
  • nemothekid 7y ago
    1. They never said it was faster, only that memory usage was better. Regardless, it could be the case that log N < C, if C is sufficiently large. 2. Memory usage on a hash map would be worse especially if the fill ratio is relatively low.
  • scott_s 7y ago
    This subthread explains why it's more memory efficient to use a tree-based structure: https://news.ycombinator.com/item?id=22239393 https://news.ycombinator.com/item?id=22239393. Short version is that in order to get good performance out of a hashtable based structure, you want to have more than n slots in order to achieve good performance. Which brings me to my second point: hashtable based data structures are not worst-case O(1). They are worst-case O(n), because in the worst case, you will either have to scan every entry in your table (open addressing) or walk a list of size n (separate chaining). Of course, good hashtable implementations will not allow a situation with so many collisions, but in order to avoid that, they will need to allocate a new table and copy over the contents of the old, which is also a O(n) operation. Given two kinds of data structures, one which is average-case O(1), but worst-case O(n) versus best- and worst-case O(log n), which one you choose depends on what kinds of performance you're optimizing for, and how bad the constants are that we've been ignoring. If you care more about throughput, then you usually want average-case O(1), as the occasional latency spikes aren't important to you. But if you care more about latency, then you'll probably want to choose worst-case O(log n), assuming that its implementations constants aren't too bad.
    • Jweb_Guru 7y ago
      Cuckoo hashmaps are worst case O(1) when implemented correctly, up to resizing (however, they do need more space and perform worse in virtually all real benchmarks).
• harikb 7y ago
  > Discord has never been afraid of embracing new technologies that look promising. > Embracing the new async features in Rust nightly is another example of our willingness to embrace new, promising technology. As an engineering team, we decided it was worth using nightly Rust and we committed to running on nightly until async was fully supported on stable. > Changing to a BTreeMap instead of a HashMap in the LRU cache to optimize memory usage. It is always an algorithm change
• deleted 7y ago
  [deleted]
• dancemethis1 7y ago
  Well, none of it matters since Discord is hostile software. No language will solve their privacy-trampling deeds.
• fmakunbound 7y ago
  Why /does/ it run a GC every 2 minutes? I went looking and didnd't find a reason in the code... https://github.com/golang/go/search?q=forcegcperiod&unscoped_q=forcegcperiod https://github.com/golang/go/search?q=forcegcperiod&unscoped... Go's GC seems kind of primitive.
• truthwhisperer 7y ago
  other example of bad IT management. Spend those millions on improving Go instead of refactoring code and moving to rust. And why the hell did you choose for Go anyway. Because some hancy fancy developper try to copy Google? Bad.
• dfee 7y ago
  The one problem I’m curious as to how channel-based chat applications solve, to which my google-fu has never lead me in the right direction: how do you handle subscriptions? I imagine a bunch of front end servers managing open web sockets connections, and also proving filtering/routing of newly published messages. Alas, it’s probably best categorized as a multicast-to-server, multicast-to-user problem. Anyways, if there’s an elegant solution to this problem, would love to learn more.
  • Pfhreak 7y ago
    Not sure if this is exactly what you are looking for, but I'd do some digging into consistent hash rings.
    • dfee 7y ago
      Oh, interesting: https://en.m.wikipedia.org/wiki/Consistent_hashing https://en.m.wikipedia.org/wiki/Consistent_hashing > Consistent hashing maps objects to the same cache machine, as far as possible. It means when a cache machine is added, it takes its share of objects from all the other cache machines and when it is removed, its objects are shared among the remaining machines. I guess the challenge here is that subscriptions are sparse: I.e. one ws connection can carry multiple channel subscriptions, thus undermining the consistent hash.
      • Pfhreak 7y ago
        There's a number of ways to tweak the algorithm, e.g. by generating multiple hashes per endpoint and then distributing them around a unit circle. I've seen this used to consistently allocate customers to a particular set of servers, not just ensure you are hitting the right cache. It doesn't fully solve the subscription issue where multiple people are in multiple channels, but it could probably be used as a building block there.
• yippir 7y ago
  I chose Rust over Go after weighing the pros and cons. It was an easy decision. I wouldn't consider using a high level language that lacks generics. The entire point of using a high level language is writing less code.
  • shdh 7y ago
    The syntax looks pedantic to me. Going to require some adjusting.
• nottorp 7y ago
  Can someone wake me up when they switch from javascript to something native in the client? I just checked and as usually, I have an entry labeled "Discord Helper (Not Responding)" in my process list. I don't think i've ever seen it in a normal state.
  • zlynx 7y ago
    That is kind of bad Windows programming but easy to do when writing an app that doesn't need to handle Windows event messages. It probably sits in a loop waiting on socket events and doesn't care if you sent it a WM_QUIT or not. It would be easy to pump the message loop and ignore all, but why bother?
    • nottorp 7y ago
      Lol it's a javascript thing that instantiates a copy of Chrome, not a Windows program. I doubt they know what a WM_QUIT is...
• viraptor 7y ago
  The next step I expected after LRU tunning was to do simple sharding per user, so that there are more services with smaller caches, (cancelling out the impact) with smaller GC spikes, offset in time from each other. I'm curious if that was considered and not done for some reason.
• pkolaczk 7y ago
  This is consistent with my observations of porting Java code to Rust. Much simpler and nicer to read safe Rust code (no unsafe tricks) compiles to programs that outperform carefully tuned Java code.
  • marta_morena 7y ago
    Sorry, but `Much simpler and nicer` is something that I highly doubt when you talk about Java to Rust. Unless the people writing the Java code were C programmers, lol, in which case I feel for you.
    • efaref 7y ago
      Rust's type system is more expressive than Java's so you can end up with much nicer to read code with stricter and more obvious invariants. There also tends to be way less of the `EnterpriseJavaBeanFactory`-style code in idiomatic Rust.
      • Polyisoprene 7y ago
        Trolling is fun and all, but I wouldn’t say Rust’s type system that much more advanced than Java’s. The borrow checker definitely helps to catch errors, but I would rate them at basically the same level.
        • gameswithgo 7y ago
          So, being able to have a value or a reference vs everything always being a reference is a pretty massive difference. Option types instead of nulls is also a pretty large difference. Generics being better from a performance perspective is a large difference too. As well, traits are quite a bit different than classes, but not always in a good way!
          • Polyisoprene 7y ago
            Hopefully Java will support proper values regarding inlining and on-stack allocation for non-primitive types soon, but yes option types are a benefit in Rust as long as you’re not using any more advanced reference libraries.
• romaniitedomum 7y ago
  You're switching to Rust because Go is too slow? Colour me sceptical, but this seems more like an excuse to adopt a trendy language than a considered technical decision. Rust is designed first and foremost for memory safety, and it sacrifices a lot of developer time to achieve this, so if memory safety isn't high in your list of concerns Rust is probably not going to bring many benefits.
  • hajile 7y ago
    Did you read the article? The naive Rust version was better than the tuned golang version in every metric. The most important one (latency) simply wasn't fixable due to golang's GC (something that is a bit of a general GC issue I might add).
    • romaniitedomum 7y ago
      Did you read my comment? I don't dispute that the Rust version is faster in every way. I am disputing that rewriting in Rust was a sensible technical decision, and in support of this I point you to where the author describes having to use a nightly build of the compiler to get async support. Given that they had to jump through a lot of hoops to make this work, I am saying they could have achieved the same speed increase with less effort using a stable C or C++ compiler. Hell, had they invested a fraction of the time spent rewriting in Rust in the Go version, I'll bet they could have improved it to the point where there was no need to rewrite it at all. It's clear that Discord use Rust a lot, and that they are looking for any excuse to replace existing code with Rust code.
  • smabie 7y ago
    What would you recommend that doesn’t have a GC? Zig? C? Rust is a fine choice. Besides if you really don’t care, just make the entire program unsafe and you’ll still reap benefits over C or C++.
    • romaniitedomum 7y ago
      Whatever language and toolset gets the job done with the least amount of effort. Given the hoops that Discourse had to jump through to get Rust working, that wasn't a good technical decision. They'd have got the same result with less pain with C++.
• shanev 7y ago
  When a company switches languages like this, it's usually because the engineers want to learn something new on the VC's dime. They'll make any excuse to do it. As many comments here show, there are other ways to solve this problem.
• deepsun 7y ago
  Wait, isn't Go devs said they solved GC latency problems [1]? (from 2015): "Go is building a garbage collector (GC) not only for 2015 but for 2025 and beyond: A GC that supports today’s software development and scales along with new software and hardware throughout the next decade. Such a future has no place for stop-the-world GC pauses, which have been an impediment to broader uses of safe and secure languages such as Go." [2] [1] https://www.youtube.com/watch?v=aiv1JOfMjm0 https://www.youtube.com/watch?v=aiv1JOfMjm0 [2] https://blog.golang.org/go15gc https://blog.golang.org/go15gc
  • terminaljunkid 7y ago
    That seems to be written by some Manager with slight clue of tech, tbh.
• arjunbajaj 7y ago
  Question for the Discord team: Was implementing the same service in Elixir an option? Did you try it/why not?
  • robocat 7y ago
    Discord also use Elixir - there are comments elsewhere above for why Elixir might be a bad choice in this case.
    • arjunbajaj 7y ago
      Thanks!
• brylie 7y ago
  What are some recommended resources for a gentle introduction to Rust?
  • fatbird 7y ago
    I read the Rust Programming Language book over Christmas and it's a very good introduction to it, probably one of the best I've seen for any language. It's got a good voice, and it's very good about putting enough context around Rust design decisions to understand the why as well as the how. But's it's not so long that it feels like a slog.
    • steveklabnik 7y ago
      Thank you!
    • brylie 7y ago
      Link, for convenience: https://doc.rust-lang.org/book/ https://doc.rust-lang.org/book/
• woah 7y ago
  Switching to Rust is a good idea, but I was wondering- would it be possible to run two identical instances in parallel and return results from the fastest one? This would almost completely eliminate GC pauses from the final output.
• archi42 7y ago
  Uhm, I'd suppose the service runs on one or more dedicated nodes - so there should be no competition for RAM (or if a node runs multiple services, the I'd expect a fixed memory amount to be available). In such an environment, each fixed size LRU cache could just allocate a huge chunk of RAM for data + indices (index size is bound by data size). That's nothing to do with the ownership model, it's just manually managed memory. Yes, reality is more complex since they probably have multi socket servers/NUMA, which might add memory access latencies and atomic updates to the LRU might require a locking scheme, which also isn't trivial (and where async Rust might be useful).
• mc3 7y ago
  More accurately "Why Discord is switching a service from Go to Rust"
• esjeon 7y ago
  I wonder if they actually did their homework. Doesn't matter if they like it, but they could have avoided rewriting, if they wanted. The thing is, you can allocate memory outside of Go, and GC will simply ignore such regions, since GC only scan regions known to it. (Mmap should work like a charm here.) A drawback is that pointers in such regions will not be counted, but it's easy to workaround by copying whole data, which is encouraged by the language itself. TBH, Go sucks for storing a large amount of data. As you can see here, even the simplest cache can be problematic. The language is biased towards large datacenters, where the amount of available resources are less of a concern. Say, this problem can be solved by having external cache servers and extra nodes around them. Latency will not be idealistic, but the service will survive with minimal changes.
• pmarreck 7y ago
  Is the Discord server-side still coded in Elixir?
• LaserToy 7y ago
  I’m sorry, but isn’t it cashing 101 ? Do not keep long living objects in GC managed memory. And there are ways to do it in both go and even java.
• meirelles 7y ago
  The Twitch folks were facing a related situation with the GC. They developed a workaround that they called Ballast, reducing the overall latency and making it more predictable. Quite impressive results [0]. The Go's GC is groundbreaking in several aspects, but probably needs to provide ways to fine-tune it. Posts like this make me believe that one-size-fits-all settings are yet to be seen. [0]: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
• blazespin 7y ago
  Confused, aren't they losing memory safety? I get for certain core code situations, you want to manage all memory safety yourself (or use built in static GC), but beyond that it seems to me at a higher level you'd rather have the automatic GC. Why burden all of your developers rather than just a core few? I don't think GC issues is a compelling argument to move everything to Rust. I'm not saying there aren't compelling arguments, but that just seems a bit odd that that's their main argument.
  • echeese 7y ago
    Nah, guaranteed memory safety is actually one of Rust's main selling points
  • buzzerbetrayed 7y ago
    I’ve never heard the argument that moving to rust reduces memory safety. Isn’t memory safety what rust is known for?
    • Matthias247 7y ago
      It is! But in Rust you still have an escape hatch in the form of the `unsafe` annotation which allows for mistakes which break memory safety. I don't think Go has something like that, unless you use the FFI. So saying that Go is at least as memory safe as Rust might not be too wrong of a statement. However I think in total Rust is safer. E.g. Rust prevents a ton of race conditions in multithreaded code, which Go can not do.
      • Jweb_Guru 7y ago
        Go has data races on multiple cores in safe code, without using any unsafe intrinsics or C FFI.
• musicale 7y ago
  1st Law of Garbage Collection: Consistent speed and efficiency usually requires circumventing the garbage collector.
• yobert 7y ago
  Not to participate in the flaming-- but I'd love to hear some stats about compile times for the two versions of the service. (Excellent write-up by the way! Thanks!)
• sayusasugi 7y ago
  Great, can the client be ported to Rust while you're at it? Electron is such a joke.
• eric-hu 7y ago
  I'm curious what the product-engineering landscape in the company looks like to allow for a language rewrite to happen. I feel like this would be a hard sell in all companies I've worked at. Was this framed as a big bug fix? Or was faster performance framed as a feature?
  • modo_mario 7y ago
    I think they're at a scale now where the cost of running it starts to become important as well. At least when we're talking about big performance increases like this.
• moneywoes 7y ago
  Another blow for Google.
• stiray 7y ago
  And brings me back to my years old nag. "Ok, you got GC, fine. But DO give me option to hand free specific memory when I want to. I don't consider hand allocation and deallocation such a pain than GC going wild." This doesn't only go for Go.
• dis-sys 7y ago
  I believe the problem described in this blog has been at least partially addressed in the Go 1.12 release.
• tonyferguson 7y ago
  Wow, Rust is amazing, so fast! It is like these people never learnt c? Why did they spend all this time trying to optimise such a high level language? Surely they can afford a more experienced engineer who will tell them that is a path that isn't worth it? I jump straight to c when there is anything like this, although I guess Rust is an option these days.
• raverbashing 7y ago
  > but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references Yeah please tell me again how GC is a superior solution to reference counting in cases when you know exactly when you don't need the object anymore. (Hint: RC is not GC if the object is dealocating itself)
• fxtentacle 7y ago
  Sounds like badly reinventing the wheel. If you need a large in-memory LRU cache, use memcached. Problem solved, because then Go doesn't need to allocate much memory anymore. And I'd wager that JSON serialization for sending a reply back to the client will dominate CPU load anyway, so that the overhead for Go to talk to Memcached will be barely noticeable.
• FisherGuy44 7y ago
  This is not a fair comparison. Go 1.9.2 was released over 2 years ago. In that time they have fixed a lot of the GC stutter issues. Comparing rust nightly to a 2 year old compiler is unfair.
• willvarfar 7y ago
  This is a bit late to add, but from the description of the problem in the article, the way to make the program faster, irregardless of language, is to use a big array rather than lists and trees. Carve the array up as necessary, so the array of users to offsets in the array where the data is. Basically, be your own memory allocator, with all the loss of safety but the order of magnitude improvement in efficiency that that brings.
• juskrey 7y ago
  Golang: a post-academic delusion built around single petty feature. Rust: everything you needed and hated about C++, now in edible packaging with flavor.
• crazypython 7y ago
  In D, you may explicitly delete memory while having a GC.
• luord 7y ago
  Usually this kind of article is about "migrating from massively popular language to more niche language that we like better". This is more from niche to niche. Thought that was interesting, but yet the discussion here wasn't all that different to the usual. Guess it's flamewars always, regardless of popularity.
• Fire-Dragon-DoL 7y ago
  I run the same question here: can't a memory pool be used in this case? In gaming industry there are similar problems with GC and they were solved with memory pools