9 ms·
Meta-comment, and I’m not claiming this is what’s true about Discord. Every time I see a company post about how they’re changing programming languages, I tend
by reikonomusha 6y ago
Meta-comment, and I’m not claiming this is what’s true about Discord.
Every time I see a company post about how they’re changing programming languages, I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. But, of course, you’d be lambasted by your colleagues if you didn’t provide technical reasons, so you go and hunt for technical reasons that seem insurmountable and build a case for it, turning a blind eye to perhaps less exciting solutions that don’t require a programming language change.
(It’s sort of like how a cop in America can find you doing something illegal in any circumstance if they look hard enough. Except in this case, it’s a matter of finding technical deficiencies.)
This strategy highly depends on clout though, which is why I think technical arguments are frequently really a façade. If, say, a junior programmer doesn’t have clout, even presenting an equivalent technical argument won’t lead to a change in programming language, especially if it’s the senior’s favorite. I’m sorry to say, but most tech organizations aren’t meritocratic or egalitarian in this way (and perhaps they shouldn’t be).
As it pertains to the article—which again I don’t claim is all “fake” politicking (I don’t know anything about or anyone at Discord)—they mostly ripped on garbage collection. They specifically talked about how GC was bad because their LRU cache had to be scanned, which was part of why they observed spikes in their metrics. I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not?
I think using any language with any serious and large project will find deficiencies in the language. At some point in a product lifecycle it’s more profitable to just work with and ameliorate those deficiencies than do rewrites where the new deficiencies will be completely unknown.
- aaomidi 6y agoWell, on the positive side a company like Discord moving to Rust means better tooling than what is currently available. So that's exciting.
- reikonomusha 6y agoBetter tooling for the programmers you mean? I don’t regularly program in either language (Go or Rust), but I thought Google ran a pretty tight ship on good Go tooling, except with a bit of module SNAFU in its past.
- aaomidi 6y agoGo's tooling is pretty nice. I was talking about better tooling for Rust. Maybe JetBrains will make a Rust-specific IDE. So far though rust-analyzer >>>>> JetBrains' plugin
- unanswered 6y ago> I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it Ah yes, because what I really want out of an ostensibly memory-safe language like go is to have to resort to C, or what amounts to C, to get anything done. Or I could use a language which doesn't have this garbage issue. But that would just be political nonsense apparently.
- reikonomusha 6y agoI’m not saying that it’s a perfect solution, and yes, using C memory defeats the purpose of memory safety. But it’s a trade-off, as most engineering things are. Rewriting in Rust is also a gigantic trade-off along many many more axes, and Rust isn’t immune to people using `unsafe` in practice either. In both Go and Rust, it’s possible to carefully engineer so as to isolate where sources of memory unsafety are, either through the language or through principled policy and software design.
- aaomidi 6y agoI really am glad Discord did not go with this route. It's a chatting application, which means you're dealing with a TON of user generated content. Unmanaged memory and user generated content is an absolutely terrible idea.
- ctvo 6y ago...? The article is about a very specific service handling state for if you've seen a message in a channel. It has nothing to do with processing user content.
- unanswered 6y agoThe problem with go here is (a) garbage collection, such that rust likely wouldn't have needed any unsafe outside of well-known libraries; and worse (b) when you need to do stupid C shit in go, there's no `unsafe` to contain it; it just spews over the rest of your code. "Principled policy and software design" is exactly what's needed to get rid of memory bugs in C too! So we can conclude that these things are impossible, or else we would not need memory safe languages in the first place. Since it's increasingly apparent that go requires these mythical safeguards after all, it's safe to say that it is not a memory safe language in the useful sense.
- silentsea90 6y ago+1 going from meme language to meme language seems highly suspicious. Go and Rust are fine languages, but given these are relatively new and immature, I find it startling that Discord had to do writes/rewrites so often. I'd wager that if Discord had started sooner than 2015 (say, 2013ish) - we'd have seen Scala thrown into the fray as well.
- mraza007 6y agoYup this makes me wonder why do companies go for relatively newer languages compared to old ones such as Java,C++ or C as they have been matured and has been widely used
- wrenky 6y agoEasier to hire people working in the new hot languages, and get younger programmers out of college.
- johannes1234321 6y agoI don't think so. There is a full load of people with experience in C++ or Java, but people with experience in performance critical (whatever that means) Rust code is more limited for being a newer language. You however win on the hipness scale and attention in places like hacker news. As others have said, however, a big factor certainly is that an internal lead developer lobbied for Rust, which they experimented with.
- ModernMech 6y agoAt the same time, in college we are teaching our students Java and C++ because that's what companies expect our students to know. Half our CS curriculum is in Java. The students I teach loathe C and C++ and want to learn Rust, but we won't teach it to them except in an experimental elective that won't count toward their graduation. So it's hard to say how many young programmers out of college actually know these "new hot languages".
- 6y ago
- deleted 6y ago[deleted]
- zn44 6y agoi believe this also often happens when original designers of the systems are out and not involved anymore, the new team wants to rewrite everything from scratch but that's hard to sell. So they find few technical reasons and pick some latest fun tech to play with.
- FalconSensei 6y agoYeah, unfortunately in some cases is easier to sell rewriting to a new language, than just refactoring/rewriting in the current stack.
- MengerSponge 6y agoHam Sandwich Development
- xtracto 6y agoRight, specially because last thing I read Discord was using Elixir: https://blog.discord.com/scaling-elixir-f9b8e1e7c29b https://blog.discord.com/scaling-elixir-f9b8e1e7c29b
- adamnemecek 6y agoThey are using both https://blog.discord.com/using-rust-to-scale-elixir-for-11-million-concurrent-users-c6f19fc029d3 https://blog.discord.com/using-rust-to-scale-elixir-for-11-m...
- hedgehog 6y agoAgree, Rust seems like a good choice but I'm curious why they needed to replace Go. GC pause time is mostly due to the amount of pointer-containing heap memory. Moving the the cache storage to slabs of pointer-free "dumb" slices should pretty much avoid the problem so long as you force GC frequently (large heap will confuse the heuristic). Shard the slabs across a handful of goroutines, one channel each to handle locking, and that should perform fine compared to an equivalently complex Rust implementation.
- ctvo 6y agoThe article lays out the points as if they exhausted all their Go options and ran into some deep language problem. I agree with you that this isn't a huge deal in Go, and could have been solved without a language, tooling, etc. change. It reads like a tool (Rust) looking for a problem.
- alexfwulf 6y agoHow could that have been solved?
- spoiler 6y ago> I tend to believe it actually happened because engineers with enough political clout decided they wanted everybody to change to their preferred language, not truly because of technical reasons, but because they just wanted to be paid to write in a cool language. I would say it's cool _because_ of the technical reasons, though! You make it seem like people are inventing technical reasons to make rust seem cool. I could make a long "technical reasons" list why I would personally prefer to work in Rust, but it boils down that it makes me more productive, and the code more performant with fewer gotchas and less sacrifices/compromises (compared to say C++ or C). Do I think Rust is cool? I guess... But that which attracted me to Rust was there already; I didn't have to make it up[1]. I understand not every language is for everyone. I've had colleagues dismiss rust simply because they prefer C++ (and we work with C++). Some will even agree Rust is better when presented with technical reasons, but still not like Rust "just because." > I wonder why they didn’t try doing things like putting the LRU cache into foreign, unmanaged memory to avoid garbage collecting it. Easier said than done, but why not? I write Go intermittently since I participate a little in the Kubernetes ecosystem, and haven't dwelled that deeply into it. How would this be done without reaching for C, though? Or would it be fitting to swap languages in this "subset" of the implementation? [1]: I feel like "make up" is the wrong phrase here, but I can't think of a better one right now. I guess what I want to say that Rust's "cool" reputation is well deserved. I don't think the technical reasons are made-up to serve as office-politics cannon fodder.
- alephu5 6y agoThis is my experience too. I've seen a lot of decisions made due to personal preference, at considerable technical and financial expense. The most recent example was a replacement for a complex YAML DSL, served by spring config sever and interpreted by a python program on every instantiation. This thing was literally using 20% of our CPU cycles over the whole compute cluster and estimated to cost around €6k per month. Long story short I provided a fully working pre-processing solution with effectively no runtime cost, no dependency on the spring config server and a more powerful & type-safe DSL. It was dhall with a bit of bash and python, although you could also have used jsonnet or any configuration language. Another developer with more clout pushed me out in favour of their own solution, which I believe was to keep me away from his political pedestal. A few months later I left and from what I've heard the company is still burning 20% of it's compute budget on this...
- phabora 6y agoThis is a fine suspicion to have. It’s a fine theory, even. But why raise this suspicion for this particular neo-language case? Why not raise it across the board? Such a theory is more robust if it predicts how the organization will work in general. Thus one would also be partial to suspecting that neo-languages are declined for political reasons as well -- that a few powerful voices simply prefer to find other solutions, and then find post-hoc technical arguments. Such a theory works less well if it is only used to explain some particular (ostensibly) technical decisions.
- agumonkey 6y agoI skimmed through this https://blog.kraken.com/post/7964/oxidizing-kraken/ https://blog.kraken.com/post/7964/oxidizing-kraken/ yesterday now its php->rust so a widely different migration but rust was measured for its latency cutting value, not just fad factor I'd believe discord and others are on rust for this and not just the new shiney (that would be to their loss)