9 ms·

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from hi

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Not only had it gone into production, it made maintenance far easier.

deleted 2y ago
[deleted]
mjr00 2y ago
> Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.
- cratermoon 2y ago
  > needs to be a key leader for the part being broken out into a separate system Indeed, the developer was one of the best programmers I've known and absolutely the key person on the system. The New Guys Clique were the sort of developers, you might know some, who come in, look at the existing systems, decide it's all wrong and terrible, and set out to Do It Right.
  - usefulcat 2y ago
    I've seen almost this exact scenario play out, although in my case it was just one person as opposed to a clique. He had just come from a much larger company in the same business, and almost right away he proposed that we should rearchitect a significant portion of our software to match the way things were done at his previous employer. His proposed architecture wasn't without elegance, but it was also more complex and, more importantly, it didn't solve any problems that we actually had. So in the end it was more of an ideological thing. He seemed to take it personally that we didn't take him up on his proposal, and he left a few months later (and went back to his previous employer, though to a different group). Don't think he was around for even a year. He wasn't young either; he was pretty experienced.
    - 59nadir 2y ago
      What was the language and proposed architecture? Nothing is quite as grating as being hired for work on an Erlang/Elixir code base and seeing basically no correct usage of the platform and it'd definitely result in an instant suggestion to start correctly leveraging Erlang, piece by piece. It's not even a question of architecture as much as it is completely ignoring what makes the platform good, IMO.
      - usefulcat 2y ago
        It was all c++. The existing architecture consisted of a number of different executables, which were mostly statically linked (except for stuff like the C and C++ runtime libs). His proposal was to factor out the business logic from each executable into a shared library, and then (if memory serves) we would only need a single executable that could work with any of these new shared libraries. Like a plugin architecture. This would mean that when rolling out a new version, usually only the shared library would need to be replaced. So.. significantly more complexity, due to now having a more arms-length interface between the executable and the business logic in the shared libraries. And the payoff was.. when deploying, we would upgrade a shared library instead of an executable? Still doesn't seem worthwhile.
harrall 2y ago
Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when someone has done something before and when it’s their first time.
- LAC-Tech 2y ago
  literally what I wanted to do as an opinionated junior
- cratermoon 2y ago
  > a full rewrite as a separate project. And it was never constrained to rewriting the existing system. The rewrite plan was motivated by the entirely reasonable desire to make further improvements possible, an additional mistake was the attempt to add major improvements as part of the rewrite. The new guys made their disdain for the existing system obvious, to the extent that their intent for the rewrite ballooned into a ground-up rebuild of everything. Things You Should Never Do, Part I: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
- simoncion 2y ago
  > You can do successful rewrites but your rewrite has to be usable in production within like a month. I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0] The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work. [0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.
  - harrall 2y ago
LAC-Tech 2y ago
This is the way. You absolutely can turn shit codebases into something nicer and faster, and this is best done by people who know the system, and maybe even more important, knows the operational context the system exists in. I once came into an old codebase like this as a junior, thinking I can start again. And I was gently but firmly told by my boss that it wouldn't work, this software is crucial to operations that support all our revenue, and while it can be improved it has to keep working. And we improved the hell out of it.
photonthug 2y ago
Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.) Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exact definition of sufficiently large for a given context. https://en.m.wikipedia.org/wiki/Strangler_fig_pattern https://en.m.wikipedia.org/wiki/Strangler_fig_pattern
- mjr00 2y ago
  A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully. One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional! [0] http://sevangelatos.com/john-carmack-on-parallel-implementations/ http://sevangelatos.com/john-carmack-on-parallel-implementat...
  - Tknl 2y ago
    I second this approach. I've utilized it successfully in an ongoing migration. I also second the need to have engineering knowledge from the previous system available. I was lead for 5 years on a previous system before being tasked as solution architect of new total rewrite to cloud native. The hardest part of such a migration is maintaining sponsor buy-in while you build the parallel run evaluation strangler fig integration with the old system and get some existing flow polished enough to take live. If you happen to have a rule or scripting system in place piggy pack off of it so you can do an incremental migration.

deleted 2y ago

[deleted]

mjr00 2y ago

> Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.

cratermoon 2y ago
> needs to be a key leader for the part being broken out into a separate system Indeed, the developer was one of the best programmers I've known and absolutely the key person on the system. The New Guys Clique were the sort of developers, you might know some, who come in, look at the existing systems, decide it's all wrong and terrible, and set out to Do It Right.
- usefulcat 2y ago
  I've seen almost this exact scenario play out, although in my case it was just one person as opposed to a clique. He had just come from a much larger company in the same business, and almost right away he proposed that we should rearchitect a significant portion of our software to match the way things were done at his previous employer. His proposed architecture wasn't without elegance, but it was also more complex and, more importantly, it didn't solve any problems that we actually had. So in the end it was more of an ideological thing. He seemed to take it personally that we didn't take him up on his proposal, and he left a few months later (and went back to his previous employer, though to a different group). Don't think he was around for even a year. He wasn't young either; he was pretty experienced.
  - 59nadir 2y ago
    What was the language and proposed architecture? Nothing is quite as grating as being hired for work on an Erlang/Elixir code base and seeing basically no correct usage of the platform and it'd definitely result in an instant suggestion to start correctly leveraging Erlang, piece by piece. It's not even a question of architecture as much as it is completely ignoring what makes the platform good, IMO.
    - usefulcat 2y ago
      It was all c++. The existing architecture consisted of a number of different executables, which were mostly statically linked (except for stuff like the C and C++ runtime libs). His proposal was to factor out the business logic from each executable into a shared library, and then (if memory serves) we would only need a single executable that could work with any of these new shared libraries. Like a plugin architecture. This would mean that when rolling out a new version, usually only the shared library would need to be replaced. So.. significantly more complexity, due to now having a more arms-length interface between the executable and the business logic in the shared libraries. And the payoff was.. when deploying, we would upgrade a shared library instead of an executable? Still doesn't seem worthwhile.

harrall 2y ago

Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when someone has done something before and when it’s their first time.

LAC-Tech 2y ago
literally what I wanted to do as an opinionated junior
cratermoon 2y ago
> a full rewrite as a separate project. And it was never constrained to rewriting the existing system. The rewrite plan was motivated by the entirely reasonable desire to make further improvements possible, an additional mistake was the attempt to add major improvements as part of the rewrite. The new guys made their disdain for the existing system obvious, to the extent that their intent for the rewrite ballooned into a ground-up rebuild of everything. Things You Should Never Do, Part I: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
simoncion 2y ago
> You can do successful rewrites but your rewrite has to be usable in production within like a month. I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0] The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work. [0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.
- harrall 2y ago

LAC-Tech 2y ago

This is the way. You absolutely can turn shit codebases into something nicer and faster, and this is best done by people who know the system, and maybe even more important, knows the operational context the system exists in. I once came into an old codebase like this as a junior, thinking I can start again. And I was gently but firmly told by my boss that it wouldn't work, this software is crucial to operations that support all our revenue, and while it can be improved it has to keep working. And we improved the hell out of it.

photonthug 2y ago

Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.) Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exact definition of sufficiently large for a given context. https://en.m.wikipedia.org/wiki/Strangler_fig_pattern https://en.m.wikipedia.org/wiki/Strangler_fig_pattern

mjr00 2y ago
A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully. One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional! [0] http://sevangelatos.com/john-carmack-on-parallel-implementations/ http://sevangelatos.com/john-carmack-on-parallel-implementat...
- Tknl 2y ago
  I second this approach. I've utilized it successfully in an ongoing migration. I also second the need to have engineering knowledge from the previous system available. I was lead for 5 years on a previous system before being tasked as solution architect of new total rewrite to cloud native. The hardest part of such a migration is maintaining sponsor buy-in while you build the parallel run evaluation strangler fig integration with the old system and get some existing flow polished enough to take live. If you happen to have a rule or scripting system in place piggy pack off of it so you can do an incremental migration.