6 ms·

What do you do if you have ~20 people and a decade old million line monolith that you're now bumping up against the limits of?

spand 7y ago
Seems the solution would be to improve the thing rather than reimplement it as microservices.
- pageandrew 7y ago
  Or split out an individual domain, particularly one you’re having pain with, into its own “micro”service.
  - james_s_tayler 7y ago
    I think this is going to be the approach we wind up taking. In reality there are maybe 1 ~ 3 hotpots in the domain that reveice many more transactions than the rest.
- james_s_tayler 7y ago
  I think there are some things we can legitimately improve quite a lot under the current structure, and it seems there are other things that make certain improvements just untenable. Upgrading a major version of an ORM for instance. It seems there are certain key improvements that boil down to "the only way this can be improved would result in a big-bang release affecting the entire of the system and take a dedicated team quite a long time to do". It's just not palatable for anyone. For the parts we carve out as microservices we are essentially able to erase that technical debt. That comes at the cost of increased ops complexity. We're not ready or capable of having 50 microservices from an ops perspective, but we are capable of handling more on the order of ~5.
  - mamarjan 7y ago
    Every additional microservice adds complexity to the system, due to having to factor in many failure situations not present if everything is instead in one process. There are other options in your case, like isolating any dependencies in a module with a well defined API instead of having it all in a separate (micro-)service.
meowface 7y ago
You could consider splitting certain large, distinct pieces into separate services, maybe. You don't need a dichotomy of one monolith vs. dozens of microservices. You could have a monolith plus 3 or 4 services. Or only do that going forward; e.g. for future feature requests that would otherwise require adding another huge chunk to the monolith.
- james_s_tayler 7y ago
  This is pretty much my thinking.
  - collyw 7y ago
    Common sense, no?
ak39 7y ago
But what are the limits that warrant even considering microservices (a bit more detail please)? (Serious question)
- james_s_tayler 7y ago
  We need to support an order of magnitude more daily active users than we currently do. I'm not exactly sure how close the current system would actually get to that, but my gut feel is it wouldn't hold up. It does OK as is, but only OK. It's a combination of three different problems working against us in concert. 1) compute layer is multitenant but the databases are single tenant (so one physical DB server can hold several hundred tenant database with each customer having their own). 2) We're locked into some very old dependencies we cannot upgrade because upgrading one thing Cascades into needing to upgrade everything. This holds us back from leveraging some benefits of more modern tech. 3) certain entities in the system have known limits whereby when a customer exceeds a certain threshold the performance on loading certain screens or reports becomes unacceptable. Most customers don't come near those limits but a few do. The few that do sometimes wind up blowing up a database server from time to time affecting other clients. For most of the domain stuff, to be honest I'd like to fix the performance problems and deadlocks by just making data access as efficient as possible in those spots. I think that could get us quite a bit more mileage if we took it seriously and pushed it. For the single tenant database situation, I don't really know how to approach fixing that. I don't see us having enough resource to ever reengineer it as is. Maybe it's possible for us as a team, maybe it's not. The thinking is that for the parts of the domain we're able to split out, we could make those datastores multitenant. There's also a bunch of integration code stuck in the monolith that causes various noisy neighbor problems that we are trying to carve out. I think that's a legitimate thing to do and will be quite beneficial. But yeah... It's a path we're dipping our toes into this year in an effort to address all of these problems which are too big for us to tackle one by one.
  - guiriduro 7y ago
    Sounds like you would do best solving the problem you have clearly identified first - the need for a multi-tenant database solution with decent performance. To limit the re-engineering work necessary you could look to separate a coherent functional area as a service where query performance analysis shows heaviest contention, create a microservice for just that, then use best practice like adding a tenant_id key and a system that shards intelligently on that, e.g. citus.
ludamad 7y ago
Enumerating those limits and doing a boring risk benefit analysis of approaches to remove them. Sure microservices could fit in, but you need to get things pretty clean to split off a chunk in the first place
ClikeX 7y ago
I'd say you'd first need to see if the limits can be fixed in the current setup. If you really need to build a microservice to replace a part of your monolith. You need to hire a few devs that can act as a dedicated team.
- james_s_tayler 7y ago
  I definitely think a reasonable degree of performance could be gained under the current structure by streamlining data access and maybe getting a bit smarter with caching. That still leaves us with non-scaling related problems of being unable to upgrade some very core dependencies which are stuck on some quite outdated versions. Perhaps the strangler application approach could work to resolve that aspect over time.
echelon 7y ago
Probably keep it as a monolith. You don't need to define rigid ownership. Though some pieces will move slower than others (and some things will never change), a single deploy cadence is probably fine. There's also the danger of an incomplete microservice migration where you've now got hobbled together half-service weirdware that you have to support forever. Don't do microservices until engineers working on their thing break your thing.
- Sean1708 7y ago
  > an incomplete microservice migration where you've now got hobbled together half-service weirdware that you have to support forever. Ugh I have to deal with this at my current job, except it's not an incomplete microservice migration it's by design and comes with a weird hobbled-together event queue implemented as a table in the DB and that has a known race condition that pops up every couple of weeks.
kitd 7y ago
It's still possible, and arguably good design, to write a monolith as a cluster of separate cooperating internal logical services. That way you can keep the benefits of decoupling and ownership boundaries without all the deployment complexity. In Java, OSGi was particularly good for this, but you could easily build it around a fairly simple DI framework in pretty much every major language. The most important thing though is to keep your interfaces well-designed and well-communicated. They are the principal source of bugs and misunderstandings.
collyw 7y ago
The whole monolith won't be a bottleneck. Work out what parts need to scale and factor those out.
user5994461 7y ago
20 people wouldn't make a million line in a decade. That's more like the output of a few hundreds people.
- _se 7y ago
  Haha, I love comments like this. We've written over 10 million lines as 20-40 people in the past 15 years. I'll be sure to tell everyone we're not supposed to be moving so fast.
  - user5994461 7y ago
    Funny enough, you wouldn't be able to work fast at all in a 10 MLOC codebase. Working with and navigating that amount of code is a bummer. Imagine for a minute, if there were 50 developers (ignoring managers and administrative overhead) trying to understand 500 lines a day (a medium sized module in most languages), they still wouldn't be able to go through half the codebase in a whole year. So basically most of codebase can't be read or maintained.
- Merad 7y ago
  We have a 4 million LOC ~15 year old monolith made by about 10 people that would like to have a word with you.
  - user5994461 7y ago
    Tell me more about it? What industry is it, what does the software do? Does it have a dictionary of 3M words to support spell check? :D
- squiggleblaz 7y ago
  I work on a monolith. It was written by never more than ten developers, mostly fewer, in maybe 15 years. It contains just more than a million lines of code (1,074,010). This figure excludes other code bases those developers worked on (for instance, at the peak of those ten developers includes several who were working on other discontinued products part time - even for most of their time). It also excludes test data, schemas etc. And there are several developers who took great pride in deleting code - I know I used to have more - lines than + lines in my few years.

spand 7y ago

Seems the solution would be to improve the thing rather than reimplement it as microservices.

pageandrew 7y ago
Or split out an individual domain, particularly one you’re having pain with, into its own “micro”service.
- james_s_tayler 7y ago
  I think this is going to be the approach we wind up taking. In reality there are maybe 1 ~ 3 hotpots in the domain that reveice many more transactions than the rest.
james_s_tayler 7y ago
I think there are some things we can legitimately improve quite a lot under the current structure, and it seems there are other things that make certain improvements just untenable. Upgrading a major version of an ORM for instance. It seems there are certain key improvements that boil down to "the only way this can be improved would result in a big-bang release affecting the entire of the system and take a dedicated team quite a long time to do". It's just not palatable for anyone. For the parts we carve out as microservices we are essentially able to erase that technical debt. That comes at the cost of increased ops complexity. We're not ready or capable of having 50 microservices from an ops perspective, but we are capable of handling more on the order of ~5.
- mamarjan 7y ago
  Every additional microservice adds complexity to the system, due to having to factor in many failure situations not present if everything is instead in one process. There are other options in your case, like isolating any dependencies in a module with a well defined API instead of having it all in a separate (micro-)service.

meowface 7y ago

You could consider splitting certain large, distinct pieces into separate services, maybe. You don't need a dichotomy of one monolith vs. dozens of microservices. You could have a monolith plus 3 or 4 services. Or only do that going forward; e.g. for future feature requests that would otherwise require adding another huge chunk to the monolith.

james_s_tayler 7y ago
This is pretty much my thinking.
- collyw 7y ago
  Common sense, no?

ak39 7y ago

But what are the limits that warrant even considering microservices (a bit more detail please)? (Serious question)

james_s_tayler 7y ago
We need to support an order of magnitude more daily active users than we currently do. I'm not exactly sure how close the current system would actually get to that, but my gut feel is it wouldn't hold up. It does OK as is, but only OK. It's a combination of three different problems working against us in concert. 1) compute layer is multitenant but the databases are single tenant (so one physical DB server can hold several hundred tenant database with each customer having their own). 2) We're locked into some very old dependencies we cannot upgrade because upgrading one thing Cascades into needing to upgrade everything. This holds us back from leveraging some benefits of more modern tech. 3) certain entities in the system have known limits whereby when a customer exceeds a certain threshold the performance on loading certain screens or reports becomes unacceptable. Most customers don't come near those limits but a few do. The few that do sometimes wind up blowing up a database server from time to time affecting other clients. For most of the domain stuff, to be honest I'd like to fix the performance problems and deadlocks by just making data access as efficient as possible in those spots. I think that could get us quite a bit more mileage if we took it seriously and pushed it. For the single tenant database situation, I don't really know how to approach fixing that. I don't see us having enough resource to ever reengineer it as is. Maybe it's possible for us as a team, maybe it's not. The thinking is that for the parts of the domain we're able to split out, we could make those datastores multitenant. There's also a bunch of integration code stuck in the monolith that causes various noisy neighbor problems that we are trying to carve out. I think that's a legitimate thing to do and will be quite beneficial. But yeah... It's a path we're dipping our toes into this year in an effort to address all of these problems which are too big for us to tackle one by one.
- guiriduro 7y ago
  Sounds like you would do best solving the problem you have clearly identified first - the need for a multi-tenant database solution with decent performance. To limit the re-engineering work necessary you could look to separate a coherent functional area as a service where query performance analysis shows heaviest contention, create a microservice for just that, then use best practice like adding a tenant_id key and a system that shards intelligently on that, e.g. citus.

ludamad 7y ago

Enumerating those limits and doing a boring risk benefit analysis of approaches to remove them. Sure microservices could fit in, but you need to get things pretty clean to split off a chunk in the first place

ClikeX 7y ago

I'd say you'd first need to see if the limits can be fixed in the current setup. If you really need to build a microservice to replace a part of your monolith. You need to hire a few devs that can act as a dedicated team.

james_s_tayler 7y ago
I definitely think a reasonable degree of performance could be gained under the current structure by streamlining data access and maybe getting a bit smarter with caching. That still leaves us with non-scaling related problems of being unable to upgrade some very core dependencies which are stuck on some quite outdated versions. Perhaps the strangler application approach could work to resolve that aspect over time.

echelon 7y ago

Probably keep it as a monolith. You don't need to define rigid ownership. Though some pieces will move slower than others (and some things will never change), a single deploy cadence is probably fine. There's also the danger of an incomplete microservice migration where you've now got hobbled together half-service weirdware that you have to support forever. Don't do microservices until engineers working on their thing break your thing.

Sean1708 7y ago
> an incomplete microservice migration where you've now got hobbled together half-service weirdware that you have to support forever. Ugh I have to deal with this at my current job, except it's not an incomplete microservice migration it's by design and comes with a weird hobbled-together event queue implemented as a table in the DB and that has a known race condition that pops up every couple of weeks.

kitd 7y ago

It's still possible, and arguably good design, to write a monolith as a cluster of separate cooperating internal logical services. That way you can keep the benefits of decoupling and ownership boundaries without all the deployment complexity. In Java, OSGi was particularly good for this, but you could easily build it around a fairly simple DI framework in pretty much every major language. The most important thing though is to keep your interfaces well-designed and well-communicated. They are the principal source of bugs and misunderstandings.

collyw 7y ago

The whole monolith won't be a bottleneck. Work out what parts need to scale and factor those out.

user5994461 7y ago

20 people wouldn't make a million line in a decade. That's more like the output of a few hundreds people.

_se 7y ago
Haha, I love comments like this. We've written over 10 million lines as 20-40 people in the past 15 years. I'll be sure to tell everyone we're not supposed to be moving so fast.
- user5994461 7y ago
  Funny enough, you wouldn't be able to work fast at all in a 10 MLOC codebase. Working with and navigating that amount of code is a bummer. Imagine for a minute, if there were 50 developers (ignoring managers and administrative overhead) trying to understand 500 lines a day (a medium sized module in most languages), they still wouldn't be able to go through half the codebase in a whole year. So basically most of codebase can't be read or maintained.
Merad 7y ago
We have a 4 million LOC ~15 year old monolith made by about 10 people that would like to have a word with you.
- user5994461 7y ago
  Tell me more about it? What industry is it, what does the software do? Does it have a dictionary of 3M words to support spell check? :D
squiggleblaz 7y ago
I work on a monolith. It was written by never more than ten developers, mostly fewer, in maybe 15 years. It contains just more than a million lines of code (1,074,010). This figure excludes other code bases those developers worked on (for instance, at the peak of those ten developers includes several who were working on other discontinued products part time - even for most of their time). It also excludes test data, schemas etc. And there are several developers who took great pride in deleting code - I know I used to have more - lines than + lines in my few years.