8 ms·

> The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, b

by mjr00 2y ago

> The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, but I have never seen that done by a team that wasn’t already fluent at shipping features inside the large codebase. You simply cannot redesign any non-trivial project (i.e. a project that makes real money) from first-principles. This resonates. At one former company, there was a clear divide between the people working on the "legacy monolith" in PHP and the "scalable microservices" in Scala/Go. One new Scala team was tasked with extracting permissions management from the monolith into a separate service. Was estimated to take 6-9 months. 18 months later, project was canned without delivering anything. The team was starting from scratch and had no experience working with the current monolith permissions model and could not get it successfully integrated. Every time an integration was attempted they found a new edge case that was totally incompatible with the nice, "clean" model they had created with the new service.

cratermoon 2y ago
I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Not only had it gone into production, it made maintenance far easier.
- deleted 2y ago
  [deleted]
- mjr00 2y ago
  > Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.
  - cratermoon 2y ago
    > needs to be a key leader for the part being broken out into a separate system Indeed, the developer was one of the best programmers I've known and absolutely the key person on the system. The New Guys Clique were the sort of developers, you might know some, who come in, look at the existing systems, decide it's all wrong and terrible, and set out to Do It Right.
    - usefulcat 2y ago
      I've seen almost this exact scenario play out, although in my case it was just one person as opposed to a clique. He had just come from a much larger company in the same business, and almost right away he proposed that we should rearchitect a significant portion of our software to match the way things were done at his previous employer. His proposed architecture wasn't without elegance, but it was also more complex and, more importantly, it didn't solve any problems that we actually had. So in the end it was more of an ideological thing. He seemed to take it personally that we didn't take him up on his proposal, and he left a few months later (and went back to his previous employer, though to a different group). Don't think he was around for even a year. He wasn't young either; he was pretty experienced.
      - 59nadir 2y ago
        What was the language and proposed architecture? Nothing is quite as grating as being hired for work on an Erlang/Elixir code base and seeing basically no correct usage of the platform and it'd definitely result in an instant suggestion to start correctly leveraging Erlang, piece by piece. It's not even a question of architecture as much as it is completely ignoring what makes the platform good, IMO.
        usefulcat 2y ago
        It was all c++. The existing architecture consisted of a number of different executables, which were mostly statically linked (except for stuff like the C and C++ runtime libs). His proposal was to factor out the business logic from each executable into a shared library, and then (if memory serves) we would only need a single executable that could work with any of these new shared libraries. Like a plugin architecture. This would mean that when rolling out a new version, usually only the shared library would need to be replaced. So.. significantly more complexity, due to now having a more arms-length interface between the executable and the business logic in the shared libraries. And the payoff was.. when deploying, we would upgrade a shared library instead of an executable? Still doesn't seem worthwhile.
- harrall 2y ago
  Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when someone has done something before and when it’s their first time.
  - LAC-Tech 2y ago
    literally what I wanted to do as an opinionated junior
  - cratermoon 2y ago
    > a full rewrite as a separate project. And it was never constrained to rewriting the existing system. The rewrite plan was motivated by the entirely reasonable desire to make further improvements possible, an additional mistake was the attempt to add major improvements as part of the rewrite. The new guys made their disdain for the existing system obvious, to the extent that their intent for the rewrite ballooned into a ground-up rebuild of everything. Things You Should Never Do, Part I: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
  - simoncion 2y ago
    > You can do successful rewrites but your rewrite has to be usable in production within like a month. I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0] The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work. [0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.
    - harrall 2y ago
- LAC-Tech 2y ago
  This is the way. You absolutely can turn shit codebases into something nicer and faster, and this is best done by people who know the system, and maybe even more important, knows the operational context the system exists in. I once came into an old codebase like this as a junior, thinking I can start again. And I was gently but firmly told by my boss that it wouldn't work, this software is crucial to operations that support all our revenue, and while it can be improved it has to keep working. And we improved the hell out of it.
- photonthug 2y ago
  Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.) Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exact definition of sufficiently large for a given context. https://en.m.wikipedia.org/wiki/Strangler_fig_pattern https://en.m.wikipedia.org/wiki/Strangler_fig_pattern
  - mjr00 2y ago
    A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully. One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional! [0] http://sevangelatos.com/john-carmack-on-parallel-implementations/ http://sevangelatos.com/john-carmack-on-parallel-implementat...
    - Tknl 2y ago
      I second this approach. I've utilized it successfully in an ongoing migration. I also second the need to have engineering knowledge from the previous system available. I was lead for 5 years on a previous system before being tasked as solution architect of new total rewrite to cloud native. The hardest part of such a migration is maintaining sponsor buy-in while you build the parallel run evaluation strangler fig integration with the old system and get some existing flow polished enough to take live. If you happen to have a rule or scripting system in place piggy pack off of it so you can do an incremental migration.
newaccountman2 2y ago
Am I naive for thinking that nothing like that should take as long as 6-9 months in the happy case and that it's absurd for it to not succeed at all?
- dennis_jeeves2 2y ago
  >Am I naive for thinking Yes.
- 59nadir 2y ago
  Only in the sense that you seem to not understand how terrible big companies, big teams and big code bases are for efficiency and productivity. It pushes the bar for reaching desired results way up, and the time it takes to get there even more. No one should ever want to be part of organizations, teams or code bases like this, for their own sake.
- murkt 2y ago
  No, you’re not naive. If it’s done by one-two people that know what they’re doing, it should be done much faster. If it’s a big new team that doesn’t know what they’re doing, working separately from existing codebase, with lots of meetings… I see no reason why it would finish at all.
- Aeolun 2y ago
  Maybe. There's a lot of dragons hidden inside enterprise code. Only if you know all of them can you really succeed the first time around.
- 000ooo000 2y ago
  You know so little about the team, the organisation, the codebase, other concurrent obligations (e.g. prod support), and the way the project is run. The only way I can imagine one having confidence in a statement like "nothing should take that long" is naïveté.
  - franktankbank 2y ago
    Then maybe 18 months wasn't too long and they should have been given more time. But seriously?

by mjr00 2y ago

cratermoon 2y ago

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Not only had it gone into production, it made maintenance far easier.

deleted 2y ago
[deleted]
mjr00 2y ago
> Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.
- cratermoon 2y ago
  > needs to be a key leader for the part being broken out into a separate system Indeed, the developer was one of the best programmers I've known and absolutely the key person on the system. The New Guys Clique were the sort of developers, you might know some, who come in, look at the existing systems, decide it's all wrong and terrible, and set out to Do It Right.
  - usefulcat 2y ago
    I've seen almost this exact scenario play out, although in my case it was just one person as opposed to a clique. He had just come from a much larger company in the same business, and almost right away he proposed that we should rearchitect a significant portion of our software to match the way things were done at his previous employer. His proposed architecture wasn't without elegance, but it was also more complex and, more importantly, it didn't solve any problems that we actually had. So in the end it was more of an ideological thing. He seemed to take it personally that we didn't take him up on his proposal, and he left a few months later (and went back to his previous employer, though to a different group). Don't think he was around for even a year. He wasn't young either; he was pretty experienced.
    - 59nadir 2y ago
      What was the language and proposed architecture? Nothing is quite as grating as being hired for work on an Erlang/Elixir code base and seeing basically no correct usage of the platform and it'd definitely result in an instant suggestion to start correctly leveraging Erlang, piece by piece. It's not even a question of architecture as much as it is completely ignoring what makes the platform good, IMO.
      - usefulcat 2y ago
        It was all c++. The existing architecture consisted of a number of different executables, which were mostly statically linked (except for stuff like the C and C++ runtime libs). His proposal was to factor out the business logic from each executable into a shared library, and then (if memory serves) we would only need a single executable that could work with any of these new shared libraries. Like a plugin architecture. This would mean that when rolling out a new version, usually only the shared library would need to be replaced. So.. significantly more complexity, due to now having a more arms-length interface between the executable and the business logic in the shared libraries. And the payoff was.. when deploying, we would upgrade a shared library instead of an executable? Still doesn't seem worthwhile.
harrall 2y ago
Also an issue is that the director attempted a full rewrite as a separate project. You can do successful rewrites but your rewrite has to be usable in production within like a month. If you don’t know how to achieve that, don’t even try. The quiet developer was able to get their own rewrite done because they understood that. Looks like the director of engineering showed some classic inexperience. You can tell when someone has done something before and when it’s their first time.
- LAC-Tech 2y ago
  literally what I wanted to do as an opinionated junior
- cratermoon 2y ago
  > a full rewrite as a separate project. And it was never constrained to rewriting the existing system. The rewrite plan was motivated by the entirely reasonable desire to make further improvements possible, an additional mistake was the attempt to add major improvements as part of the rewrite. The new guys made their disdain for the existing system obvious, to the extent that their intent for the rewrite ballooned into a ground-up rebuild of everything. Things You Should Never Do, Part I: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
- simoncion 2y ago
  > You can do successful rewrites but your rewrite has to be usable in production within like a month. I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0] The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work. [0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.
  - harrall 2y ago
LAC-Tech 2y ago
This is the way. You absolutely can turn shit codebases into something nicer and faster, and this is best done by people who know the system, and maybe even more important, knows the operational context the system exists in. I once came into an old codebase like this as a junior, thinking I can start again. And I was gently but firmly told by my boss that it wouldn't work, this software is crucial to operations that support all our revenue, and while it can be improved it has to keep working. And we improved the hell out of it.
photonthug 2y ago
Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.) Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exact definition of sufficiently large for a given context. https://en.m.wikipedia.org/wiki/Strangler_fig_pattern https://en.m.wikipedia.org/wiki/Strangler_fig_pattern
- mjr00 2y ago
  A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully. One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional! [0] http://sevangelatos.com/john-carmack-on-parallel-implementations/ http://sevangelatos.com/john-carmack-on-parallel-implementat...
  - Tknl 2y ago
    I second this approach. I've utilized it successfully in an ongoing migration. I also second the need to have engineering knowledge from the previous system available. I was lead for 5 years on a previous system before being tasked as solution architect of new total rewrite to cloud native. The hardest part of such a migration is maintaining sponsor buy-in while you build the parallel run evaluation strangler fig integration with the old system and get some existing flow polished enough to take live. If you happen to have a rule or scripting system in place piggy pack off of it so you can do an incremental migration.

newaccountman2 2y ago

Am I naive for thinking that nothing like that should take as long as 6-9 months in the happy case and that it's absurd for it to not succeed at all?

dennis_jeeves2 2y ago
>Am I naive for thinking Yes.
59nadir 2y ago
Only in the sense that you seem to not understand how terrible big companies, big teams and big code bases are for efficiency and productivity. It pushes the bar for reaching desired results way up, and the time it takes to get there even more. No one should ever want to be part of organizations, teams or code bases like this, for their own sake.
murkt 2y ago
No, you’re not naive. If it’s done by one-two people that know what they’re doing, it should be done much faster. If it’s a big new team that doesn’t know what they’re doing, working separately from existing codebase, with lots of meetings… I see no reason why it would finish at all.
Aeolun 2y ago
Maybe. There's a lot of dragons hidden inside enterprise code. Only if you know all of them can you really succeed the first time around.
000ooo000 2y ago
You know so little about the team, the organisation, the codebase, other concurrent obligations (e.g. prod support), and the way the project is run. The only way I can imagine one having confidence in a statement like "nothing should take that long" is naïveté.
- franktankbank 2y ago
  Then maybe 18 months wasn't too long and they should have been given more time. But seriously?