3 ms·

Funny anecdote about one of the FAANGs: before the idea of service meshes and Consul and Envoy a similar idea was being rolled out to transparently proxy, encry

Funny anecdote about one of the FAANGs: before the idea of service meshes and Consul and Envoy a similar idea was being rolled out to transparently proxy, encrypt, and authenticate traffic between services. The initial implementation was on erlang and didn’t cause any operational issues for our team, from what I remember. The project was moved to another team who decided for version 2 to rewrite in Java (high made sense from an integration perspective). Unfortunately, they hadn’t tested at scale along with whatever it was servicing and as it was being rolled out it quickly ballooned in memory causing systems to swap and fall over. They eventually got it working, but the rollout was less than ideal and unfortunately an erlang project that affected a significant amount of traffic without notice was deprecated.

weatherlight 4y ago
Everyone always talks/jokes about, "Rewrite it in Rust," think about all the business applications over the years that had been rewritten to Java, just because. Usually under the guise of, "Kids coming out of college will have an easier time wrapping their heads around Java," than some other lang. In my time as an engineer, I've seen quite a few Scala and Clojure apps, get rewritten in this manner. This is all antidotal, of course.
- EdwardDiego 4y ago
  Porting Clojure and Scala to Java can also have performance advantages, but yeah, Java was an industrial language before Golang made it cool.
- newswasboring 4y ago
  I sometimes wonder if this logic makes sense. I'm currently leading a fully inexperienced team where my main challenge isn't delivering products but qualified people. I have to make them learn so many company specific and codebase specific things that feels like learning a new language would be a least of it. This is of course colored by my language learning loving lens, so I can't be sure about this.
  - somekyle2 4y ago
    I've been on teams responsible for components written in more niche, powerful languages by previous employees who were deep in that particular language community. The author was really smart, the code was good. It was a big headache. First off, different build setup, different runtime, couldn't use most of our standard libs, so it was an island. It being in a language most people didn't know well meant it mostly only got updated as necessary; we certainly had people who could read the code and make tweaks with some comfort, but it was mostly just tweaks because nobody had the depth to really engage with the high level design idioms. It being a niche lang/community, it moved fast, so after a year or so our tool was apparently using deprecated practices; in our supported languages, teams did large-scale fixup across repos for new versions, but this being a special lang/toolchain, it was on us. Eventually, I believe it got rewritten, somewhat simpler, in one of our supported languages, and it was fine. Maybe this is a success story: someone made a useful tool in a weird language and we used it successfully for at least a year. Maybe it wouldn't have been made or would've been worse or more expensive if they'd been forced to write it in a supported language. Maybe it would've needed to be rewritten anyway. My view is a simple one; as one of the leads responsible for it, it was a special case bit of code, something people tried to avoid touching. Nothing about it required a niche language technically, so the added difficulty of working around that was friction and risk. It wasn't that people couldn't learn the language; they could, we had lang PhDs. It's just better if you can keep people where they have existing expertise, and spend your learning points on solving problems users care about.
    - OkayPhysicist 4y ago
      I don't think "avoid otherwise-unused languages for secondary tools" is the same as "avoid less common languages for your main product". Every software developer at your company will likely work on your main product for a non-trivial amount of time. Any software developer who can't figure out a new language in a trivial amount of time isn't worth hiring. Thus, everyone you hire will eventually be competent in the language of your main product. In contrast, for secondary products, there's a good chance most developers will never work on it at all, and of those who do a significant fraction will spend only a trivial amount of time. Adding the overhead of working in a new language to that balance is significant. Therefore, the ideal is "Choose the best language for your main product, then write all supporting software either in that language, or a widely-used lingua franca like Python".
hinkley 4y ago
I’m running into this concern at work. If you have one main app or service constellation, then you can’t make any big changes like these because they have to scale from test benchtop to production with few steps in between and there’s just no accurate way to know what you’re going to see until you try. Traffic shaping only does so much. I’ve been moving our tools and sidecars and lesser services to do more reuse, not for the code and effort duplication reasons, but so these technologies get more burnin time before they have to do it live. We have one service that is particularly good for this because it doesn’t see a lot of traffic on average, but it bursts up to production requests levels during a certain workflow that is less time sensitive than live traffic but still gets scrutiny because it blocks a run book until it’s done. Parts of that run book can be run as often as you like if you stop before the steps that alter production state, so you can run it a few times to take the median time and see if you’ve improved or regressed. We turn things on there and if they haven’t caught on fire we feel a lot better about flipping it on for user facing traffic.

weatherlight 4y ago

Everyone always talks/jokes about, "Rewrite it in Rust," think about all the business applications over the years that had been rewritten to Java, just because. Usually under the guise of, "Kids coming out of college will have an easier time wrapping their heads around Java," than some other lang. In my time as an engineer, I've seen quite a few Scala and Clojure apps, get rewritten in this manner. This is all antidotal, of course.

EdwardDiego 4y ago
Porting Clojure and Scala to Java can also have performance advantages, but yeah, Java was an industrial language before Golang made it cool.
newswasboring 4y ago
I sometimes wonder if this logic makes sense. I'm currently leading a fully inexperienced team where my main challenge isn't delivering products but qualified people. I have to make them learn so many company specific and codebase specific things that feels like learning a new language would be a least of it. This is of course colored by my language learning loving lens, so I can't be sure about this.
- somekyle2 4y ago
  I've been on teams responsible for components written in more niche, powerful languages by previous employees who were deep in that particular language community. The author was really smart, the code was good. It was a big headache. First off, different build setup, different runtime, couldn't use most of our standard libs, so it was an island. It being in a language most people didn't know well meant it mostly only got updated as necessary; we certainly had people who could read the code and make tweaks with some comfort, but it was mostly just tweaks because nobody had the depth to really engage with the high level design idioms. It being a niche lang/community, it moved fast, so after a year or so our tool was apparently using deprecated practices; in our supported languages, teams did large-scale fixup across repos for new versions, but this being a special lang/toolchain, it was on us. Eventually, I believe it got rewritten, somewhat simpler, in one of our supported languages, and it was fine. Maybe this is a success story: someone made a useful tool in a weird language and we used it successfully for at least a year. Maybe it wouldn't have been made or would've been worse or more expensive if they'd been forced to write it in a supported language. Maybe it would've needed to be rewritten anyway. My view is a simple one; as one of the leads responsible for it, it was a special case bit of code, something people tried to avoid touching. Nothing about it required a niche language technically, so the added difficulty of working around that was friction and risk. It wasn't that people couldn't learn the language; they could, we had lang PhDs. It's just better if you can keep people where they have existing expertise, and spend your learning points on solving problems users care about.
  - OkayPhysicist 4y ago
    I don't think "avoid otherwise-unused languages for secondary tools" is the same as "avoid less common languages for your main product". Every software developer at your company will likely work on your main product for a non-trivial amount of time. Any software developer who can't figure out a new language in a trivial amount of time isn't worth hiring. Thus, everyone you hire will eventually be competent in the language of your main product. In contrast, for secondary products, there's a good chance most developers will never work on it at all, and of those who do a significant fraction will spend only a trivial amount of time. Adding the overhead of working in a new language to that balance is significant. Therefore, the ideal is "Choose the best language for your main product, then write all supporting software either in that language, or a widely-used lingua franca like Python".

hinkley 4y ago

I’m running into this concern at work. If you have one main app or service constellation, then you can’t make any big changes like these because they have to scale from test benchtop to production with few steps in between and there’s just no accurate way to know what you’re going to see until you try. Traffic shaping only does so much. I’ve been moving our tools and sidecars and lesser services to do more reuse, not for the code and effort duplication reasons, but so these technologies get more burnin time before they have to do it live. We have one service that is particularly good for this because it doesn’t see a lot of traffic on average, but it bursts up to production requests levels during a certain workflow that is less time sensitive than live traffic but still gets scrutiny because it blocks a run book until it’s done. Parts of that run book can be run as often as you like if you stop before the steps that alter production state, so you can run it a few times to take the median time and see if you’ve improved or regressed. We turn things on there and if they haven’t caught on fire we feel a lot better about flipping it on for user facing traffic.