3 ms·

This is a weird article. If the two systems the author described worked so well, how did the business ever decide to "bulldoze" it and replace it with something

by geofft 6y ago

This is a weird article. If the two systems the author described worked so well, how did the business ever decide to "bulldoze" it and replace it with something inferior? Even if you take the view that some non-technical person came in and demanded a particular technical change (??), any halfway-competent business is going to keep the old thing running while they build the new one so they can continue making money, and they'll notice that the new thing doesn't work as well as the old. Right? I get the sense something is missing, and I get the sense the missing thing was that the replacements were actually better for the business. A couple of highly-talented people on a small team where the original developers owned production operations could run it as designed, and it was "really neat," but the business was limited in how they could use the system. Perhaps you couldn't easily add developers and make them work on new features. Perhaps you couldn't integrate OSS components that worked fine, and you were left behind your competitors. Perhaps the "one or two developers" who operated the system got married and started a family and weren't able to be on-call 24/7/365 any longer. Perhaps this system was one of dozens run by the business, and all the other ones were using crash-only containers, and this system continued to be the odd one out. I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems. Until then, this seems entirely too much like "We built this awesome prototype with a bunch of neat properties in theory." My suspicion is that the other side of these migrations were also driven by highly-talented technical people who have technical stories to tell, but their story is that they cleared out a bunch of technical debt and replaced it with a higher-reliability system.

mononcqc 6y ago
Author here. For the first system, it was deprecated without replacement, and just let to run by managers and people who had moved on to other teams (but used to work on it) who did the minimal maintenance required, and former employees were given emergency contracts in weird circumstances to deal with things. Roughly 3-4 years after, they finally replaced it after 2-3 attempts at rewrites that had failed before. The old design with minimal maintenance for years finally approached limits to how it could be scale without bigger redesigns; I consider this to be extremely successful. It wasn't exactly bulldozed nearly as much as declared "done" and abandoned without adequate replacements while major parts of the business was just being rewritten to use Go and a more "standard" stack. Obviously these migrations always start with something easier and by replacing components that are huge pain points to its contemporaries, and you're left with more legacy stuff in the end that is much harder to replace done in the final pushes. I felt that the blog post would have veered off point if the whole thing became about that, though. The people on these teams left in part because the hiring budget was redirected towards hiring on the new projects. The idea was that everything could be done in Go for these stacks (by normalizing on tools and libraries developed in another project and wanting to have one implementation for both the private and shared platforms), and the rewrites were to start with Ruby components. You knew working on the Erlang side of things that no feature development would ever take place again, that no new hands would be hired to help, and that you would be stuck on call 24/7 with no relief for years. All efforts were redirected to Go and getting rid of Ruby, and your stuff fell in between the cracks. I was one of the people who left on the long tail there. After my departure, I was brought back on a lucrative part-time contract as a sort of retainer for years to help them in case of major outages (got 1 or 2 in 4 years) since that was the only way they could get expert knowledge once they drove us all away. I'm still on good terms with the people there, it's just that "maintaining a self-declared unmaintained legacy stack without budget or headcount until we get to rewrite it in many years" is not where any of us wanted to drive our careers. Interestingly, we tried very hard to add new developers. We wrote manuals, tooling, a book on operating these systems (see https://www.erlang-in-anger.com https://www.erlang-in-anger.com), wanted to set up internal internships so developers from other teams could come and work with us for a while, etc. Whereas our team was very willing, internal politics (which I can't easily get in a public forum) made it unworkable and most attempts were turned down. These things were not always purely business decisions, and organizational dynamics can be very funny things.
- geofft 6y ago
  Thanks, that's helpful. I think it is impressive, and actually kind of a selling point, that the system maintained itself without formal staffing. I was going to object that you didn't build a high-reliability system, you built a high-reliability system that worked as long as you had a couple of experts staffing it, but it sort of sounds like that's not actually what happened. On the other hand, it seems like a fairly common problem that using neat but niche technology makes it hard to hire for it and get continued development. My own employer has a pretty nice system in Clojure that is well-maintained and has people working on it but nonetheless is always on the wrong side of things because it's using a language (and a development and release workflow, in turn) that nobody else is using. Is the problem that we should build systems that have all the advantages of the neat tech we like but still look like the more boring and less complicated things, operationally? Was it that you had the backstop in your Erlang system but it wasn't enough to convince the business that it could be treated like a normal system? Or was it separate (and could it have been shaped to look like the more boring thing)? Alternatively, if we take the goal of reliability engineering broadly as building systems that work despite both dysfunctional computer systems and dysfunctional human systems, it seems like that argues in favor of picking the approach that doesn't line you up to be on the wrong side of organizational politics. I don't like this conclusion - I'm a huge fan of solving problems with small but well-applied amounts of technical expertise - but I don't really have a good sense of how to avoid it. (Maybe the answer is more blog posts like yours influencing expectations.)
  - mononcqc 6y ago
    I think it's a problem of "not being the tech of choice". Obscure languages have a higher cost in that you can't just out-source the work of training people to the rest of the industry as easily. But you'll also get issues with staffing and getting people interested in working in mainstream languages that are less cool than they used to be, frameworks that are on older versions and hard to migrate, deployed through systems that aren't as nice on your resume than newer ones, or on platforms that aren't seen as positively. I don't have a very clear answer to give about why Erlang specifically wasn't seen positively. The VP of Eng at the time (now the github CTO) saw Erlang very positively (https://twitter.com/jasoncwarner/status/1287383578435780608 https://twitter.com/jasoncwarner/status/1287383578435780608) but I know that some specific product people didn't like it, and so on. To some extent a lot of the work pushing us aside was just done by very eager Go developers who just started doing work on replacing our components with new ones on the other side of the org, and then propagating that elsewhere. Whether the roadmap or other policies ended up kneecapping our team on purpose or accidentally is not something I can actually know. I kept pushing for years to improve things for our team, but at some point I got tired and left for a different sort of role.
skinner927 6y ago
I agree with you. This article complains that generic industry practices are not as optimized as his niche solutions. Thats no surprise. Of course a build and deploy system specifically designed for your product/environment is going to be more feature rich and performant, but once the initial developers leave or move on, all is lost. I have seen it happen multiple times. Sticking with slower clunkier industry practices means you can hire someone off the street and they’ll have a much better chance of being able to understand the system. If you want longevity, you use the standards. All engineering (mechanical, electrical, etc.) agrees on that.
deleted 6y ago
[deleted]
toast0 6y ago
> I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems. I'm certainly not one of those people, but they'll tell you things like how they standardized the new acquisition onto "best practices" and eliminated the "cowboy deployment" culture that was previously running rampant. I was part of an acquired Erlang team that went through this, although I mostly didn't deploy into the acquirer's systems, I managed the legacy systems, until they were mostly retired, then I retired. They won't tell you that deploy times grew massively, requiring a larger team to manage (and that increasing the team size requires increasing the team size, because of increased time spent in coordination). They won't tell you that because they've built their system to handle unreliable environments, they've neglected to make their environments reliable, and the system redundancy factor had to be increased to 3x instead of 2x because the churn rate on systems is too high to expect at least one out of two systems in a pair to be running. Of course, increasing the number of systems, increases deployment complexity, too.

by geofft 6y ago

mononcqc 6y ago

Author here. For the first system, it was deprecated without replacement, and just let to run by managers and people who had moved on to other teams (but used to work on it) who did the minimal maintenance required, and former employees were given emergency contracts in weird circumstances to deal with things. Roughly 3-4 years after, they finally replaced it after 2-3 attempts at rewrites that had failed before. The old design with minimal maintenance for years finally approached limits to how it could be scale without bigger redesigns; I consider this to be extremely successful. It wasn't exactly bulldozed nearly as much as declared "done" and abandoned without adequate replacements while major parts of the business was just being rewritten to use Go and a more "standard" stack. Obviously these migrations always start with something easier and by replacing components that are huge pain points to its contemporaries, and you're left with more legacy stuff in the end that is much harder to replace done in the final pushes. I felt that the blog post would have veered off point if the whole thing became about that, though. The people on these teams left in part because the hiring budget was redirected towards hiring on the new projects. The idea was that everything could be done in Go for these stacks (by normalizing on tools and libraries developed in another project and wanting to have one implementation for both the private and shared platforms), and the rewrites were to start with Ruby components. You knew working on the Erlang side of things that no feature development would ever take place again, that no new hands would be hired to help, and that you would be stuck on call 24/7 with no relief for years. All efforts were redirected to Go and getting rid of Ruby, and your stuff fell in between the cracks. I was one of the people who left on the long tail there. After my departure, I was brought back on a lucrative part-time contract as a sort of retainer for years to help them in case of major outages (got 1 or 2 in 4 years) since that was the only way they could get expert knowledge once they drove us all away. I'm still on good terms with the people there, it's just that "maintaining a self-declared unmaintained legacy stack without budget or headcount until we get to rewrite it in many years" is not where any of us wanted to drive our careers. Interestingly, we tried very hard to add new developers. We wrote manuals, tooling, a book on operating these systems (see https://www.erlang-in-anger.com https://www.erlang-in-anger.com), wanted to set up internal internships so developers from other teams could come and work with us for a while, etc. Whereas our team was very willing, internal politics (which I can't easily get in a public forum) made it unworkable and most attempts were turned down. These things were not always purely business decisions, and organizational dynamics can be very funny things.

geofft 6y ago
Thanks, that's helpful. I think it is impressive, and actually kind of a selling point, that the system maintained itself without formal staffing. I was going to object that you didn't build a high-reliability system, you built a high-reliability system that worked as long as you had a couple of experts staffing it, but it sort of sounds like that's not actually what happened. On the other hand, it seems like a fairly common problem that using neat but niche technology makes it hard to hire for it and get continued development. My own employer has a pretty nice system in Clojure that is well-maintained and has people working on it but nonetheless is always on the wrong side of things because it's using a language (and a development and release workflow, in turn) that nobody else is using. Is the problem that we should build systems that have all the advantages of the neat tech we like but still look like the more boring and less complicated things, operationally? Was it that you had the backstop in your Erlang system but it wasn't enough to convince the business that it could be treated like a normal system? Or was it separate (and could it have been shaped to look like the more boring thing)? Alternatively, if we take the goal of reliability engineering broadly as building systems that work despite both dysfunctional computer systems and dysfunctional human systems, it seems like that argues in favor of picking the approach that doesn't line you up to be on the wrong side of organizational politics. I don't like this conclusion - I'm a huge fan of solving problems with small but well-applied amounts of technical expertise - but I don't really have a good sense of how to avoid it. (Maybe the answer is more blog posts like yours influencing expectations.)
- mononcqc 6y ago
  I think it's a problem of "not being the tech of choice". Obscure languages have a higher cost in that you can't just out-source the work of training people to the rest of the industry as easily. But you'll also get issues with staffing and getting people interested in working in mainstream languages that are less cool than they used to be, frameworks that are on older versions and hard to migrate, deployed through systems that aren't as nice on your resume than newer ones, or on platforms that aren't seen as positively. I don't have a very clear answer to give about why Erlang specifically wasn't seen positively. The VP of Eng at the time (now the github CTO) saw Erlang very positively (https://twitter.com/jasoncwarner/status/1287383578435780608 https://twitter.com/jasoncwarner/status/1287383578435780608) but I know that some specific product people didn't like it, and so on. To some extent a lot of the work pushing us aside was just done by very eager Go developers who just started doing work on replacing our components with new ones on the other side of the org, and then propagating that elsewhere. Whether the roadmap or other policies ended up kneecapping our team on purpose or accidentally is not something I can actually know. I kept pushing for years to improve things for our team, but at some point I got tired and left for a different sort of role.

skinner927 6y ago

I agree with you. This article complains that generic industry practices are not as optimized as his niche solutions. Thats no surprise. Of course a build and deploy system specifically designed for your product/environment is going to be more feature rich and performant, but once the initial developers leave or move on, all is lost. I have seen it happen multiple times. Sticking with slower clunkier industry practices means you can hire someone off the street and they’ll have a much better chance of being able to understand the system. If you want longevity, you use the standards. All engineering (mechanical, electrical, etc.) agrees on that.

deleted 6y ago

[deleted]

toast0 6y ago

> I'd like to hear from the other side of this - I'd like to hear the stories of the people who pushed the redesign away from the systems. I'm certainly not one of those people, but they'll tell you things like how they standardized the new acquisition onto "best practices" and eliminated the "cowboy deployment" culture that was previously running rampant. I was part of an acquired Erlang team that went through this, although I mostly didn't deploy into the acquirer's systems, I managed the legacy systems, until they were mostly retired, then I retired. They won't tell you that deploy times grew massively, requiring a larger team to manage (and that increasing the team size requires increasing the team size, because of increased time spent in coordination). They won't tell you that because they've built their system to handle unreliable environments, they've neglected to make their environments reliable, and the system redundancy factor had to be increased to 3x instead of 2x because the churn rate on systems is too high to expect at least one out of two systems in a pair to be running. Of course, increasing the number of systems, increases deployment complexity, too.