8 ms·

If you have a large engineering team (hundreds+ devs) with a large codebase then having a monolith can slow down developer velocity. There’s massive scope with

If you have a large engineering team (hundreds+ devs) with a large codebase then having a monolith can slow down developer velocity. There’s massive scope with the monolith, build tools start to strain, etc

strix_varius 4y ago
Interesting. I found working on FB’s monolith to have, on average, higher velocity and fewer blockers, than working on microservicified systems at other places 2+ orders of magnitude smaller in both engineering and codebase size.
- roflyear 4y ago
  Agreed.
- ddenchev 4y ago
  It is not fair to compare the Facebooks monolith and the monolith at the average company, as they are not really the same thing. The tooling available at Facebook is built and maintained by a team larger than the engineering departments at most companies. There comes a point, where regular off the shelf tooling does not scale sufficiently well for a monolith. Tests suits and builds start to take too long. Deployments get increasingly complicated. Developers start to get into each other's way, even when working on unrelated features. Additionally, if you are using an untyped, interpreted language, keeping a large app well organized can also be a problem. Microservices is a tool for dealing with complexity and certainly not the only one. However, building the tooling and infra for a large and sophisticated monolith is not simple and not guaranteed to be an easier solution to the problems listed above.
  - strix_varius 4y ago
    How is this relevant? My comment is in response to an observation about "large engineering teams," not "the monolith at the average company." At the average company, standard tools will work fine, while companies with large engineering teams have the resources to maintain custom tooling.
    - dboreham 4y ago
      You are assuming that the observed tool strain scales with the number of developers. In my experience it scales with the number of coupled concerns inside the same repo. Now, this may be somewhat correlated with the number of developers, but not entirely. Therefore in my experience again you can end up with a moderately sized company running into tool limits with a monorepo. FB doesn't have those problems because they use different tools.
      - strix_varius 4y ago
        > FB doesn't have those problems because they use different tools. Exactly - instead of using microservice-oriented tools, they use tools organized around monoliths. And that decision serves them well. That's the whole point.
        deleted 4y ago
        [deleted]
  - cmrdporcupine 4y ago
    Microservices move the complexity rather than solve it. The dependency boundaries between portions of the data model can never be cleanly demarcated -- because that's not how information works, especially in a growing business -- so there's always going to be either some loss of flexibility or addition of complexity over time. Individual developers getting out of each other's way just ends up getting pushed to getting in each other's way at release time as the matrix of dependencies between services explodes. Engineers jobs become more about the lives and dramas of the services they work on than about business domain. You end up building your organization and reporting structure around these services, rather than the business needs of the customer. And then you end up indirectly or directly shipping that org chart to the world in delays or bugs caused by your fragmentation. Instead of modeling facts about data and their relationships, and constructing the relational model which can capture this, the developer in the microservice model becomes bogged down in service roles and activities instead, again taking them away from the actual problem: which is organizing information and making it accessible to users/customers. It's a shell game. The Facebook monolith works because engineers there invested time in building the tooling you're complaining is not available to others. Same with Google: Google invested in F1, etc. because it evaluated the cost to do otherwise and it made sense to invest in infrastructure. Yes, small companies can't often afford this. Luckily they have two things on their side: Most small companies don't have a fraction of the scale issues that a FB or a Google have. So they can afford to monolith away for a lot longer than they seem to think they can, while they put in infrastructure to scale the monolith. The industry as a whole has invested a lot in making existing things scale. e.g. you can do things with a single Postgres instance that we never would have dreamed about 10 years ago. And when that falls over, there's replication, etc. And when that falls over, guess what? There's now high performance distributed ACID SQL databases available for $use. Microservices is surely one of the longest lived, biggest cargo cults in our industry. I've seen others come and go, but microservices really seems to cling. I think because it has the perception of breaking business problems down into very small elegant independent atomic pieces, so it has a very.. industrial revolution, automation, factory floor, economies of scale vibe. But that's not what it is. There are places for it, I'm sure. But systems with highly interelated data and quickly changing requirements are not well suited. IMHO.
    - 4y ago
com2kid 4y ago
One awesome and often overlooked benefit of microservices is how they simplify security/dependency updates. With a monolith, dependency updates, especially breaking ones, often mean either all development stops for a "code freeze" so the update can happen, or you have a team responsible for doing the update and they are trying to update code faster than other devs add new code. The result of this is that updates get pushed back to the last minute, or are never just done. I've seen old (ancient) versions of OpenSSL checked into codebases way too often. With microservices, you can have a team that isn't as busy take a sprint to update their codebase, carefully document best practices for fixing breaking changes, document best practices for testing the changes, and then spread the learning out to other teams, who can then update as they have time or based on importance / exposure of their maintained services. It is a much better way of doing things. It also means some teams can experiment with different technologies or tool chains and see how things work out. The cost of failure is low and there isn't an impact to other teams, and build systems for microservices tend to be much simpler than for monoliths (understatement...)
- 0xFACEFEED 4y ago
  Microservices are a heavy handed way to draw boundaries around your software so that bad technical decisions don't bleed across different teams. Obviously there is some benefit to that but there is also a massive tradeoff - especially for certain types of software like complex UIs. > With a monolith, dependency updates, especially breaking ones, often mean either all development stops for a "code freeze" so the update can happen, or you have a team responsible for doing the update and they are trying to update code faster than other devs add new code. In all my years I've never seen a code freeze due to a dependency update. Maybe the project you were working was poorly engineered? > The result of this is that updates get pushed back to the last minute, or are never just done. I've seen old (ancient) versions of OpenSSL checked into codebases way too often. There should be nothing stopping you from running multiple versions of a dependency within a single monolothic project. > With microservices, you can have a team that isn't as busy take a sprint to update their codebase, carefully document best practices for fixing breaking changes, document best practices for testing the changes, and then spread the learning out to other teams, who can then update as they have time or based on importance / exposure of their maintained services. Gradual adoption of new dependencies has nothing to do with microservices.
vbezhenar 4y ago
Surely there should be something between gigantic monolith and micro services. I would call it service.
- mirekrusin 4y ago
  Yes, not sure why we have so many brainwashed fanatics who see world as hotdog and not-hotdog - microservices and monoliths - only.
  - geekbird 4y ago
    Seriously! I think there is a good space for the concept of "mid-services" - cluster similar and interdependent services and service fragments together, so they split in logical groups for updating. It would be sort of like "Module A is authentication and session management, module B is data handling layer, and module 3 is the presentation and processing layer." Each of those under a microservices dogma would be two to four microservices struggling to interoperate. I read the book written by the dev that advocated for microservices. I wanted to throw it across the room, but it was an ebook. He literally went for over half the book before he even addressed operability. Everything was about developer convenience, not operating it with an eye toward user satisfaction. The guy was clueless.
- strix_varius 4y ago
  Typically, "monolith" implies services - ie, the "backing services" from a traditional 12-factor monolith: - https://12factor.net/backing-services https://12factor.net/backing-services Monolith vs. Microservices comes about because microservices proponents specifically set it up as an alternative to a traditional monolith + services stack: - https://www.nginx.com/learn/microservices-architecture/#:~:text=Microservices%20versus%20Monolithic%20Architecture https://www.nginx.com/learn/microservices-architecture/#:~:t...
  - gedy 4y ago
    > Typically, "monolith" implies services - ie, the "backing services" I don't know if that's a common interpretation - in the monolith-loving companies I've worked with it very much meant a single ball of code (e.g. huge Rails app), not separate services.
mbrodersen 4y ago
That’s what Google does. Seems to work fine for them.
- seadan83 4y ago
  I think services became really popular from Amazon, and that quickly went to microservices IMHO. It's perhaps analogous to say: "That's what Amazon does. Seems to work fine for them". It's very interesting there are big techs now with well known monoliths vs Amazon services. Though, Amazon's services are so that any internally useful tool can be externally exposed and then rented out to generate more revenue. The reasons/benefits are very different compared to say decoupling or the other reasons given for service orientated architectures.

strix_varius 4y ago

Interesting. I found working on FB’s monolith to have, on average, higher velocity and fewer blockers, than working on microservicified systems at other places 2+ orders of magnitude smaller in both engineering and codebase size.

roflyear 4y ago
Agreed.
ddenchev 4y ago
It is not fair to compare the Facebooks monolith and the monolith at the average company, as they are not really the same thing. The tooling available at Facebook is built and maintained by a team larger than the engineering departments at most companies. There comes a point, where regular off the shelf tooling does not scale sufficiently well for a monolith. Tests suits and builds start to take too long. Deployments get increasingly complicated. Developers start to get into each other's way, even when working on unrelated features. Additionally, if you are using an untyped, interpreted language, keeping a large app well organized can also be a problem. Microservices is a tool for dealing with complexity and certainly not the only one. However, building the tooling and infra for a large and sophisticated monolith is not simple and not guaranteed to be an easier solution to the problems listed above.
- strix_varius 4y ago
  How is this relevant? My comment is in response to an observation about "large engineering teams," not "the monolith at the average company." At the average company, standard tools will work fine, while companies with large engineering teams have the resources to maintain custom tooling.
  - dboreham 4y ago
    You are assuming that the observed tool strain scales with the number of developers. In my experience it scales with the number of coupled concerns inside the same repo. Now, this may be somewhat correlated with the number of developers, but not entirely. Therefore in my experience again you can end up with a moderately sized company running into tool limits with a monorepo. FB doesn't have those problems because they use different tools.
    - strix_varius 4y ago
      > FB doesn't have those problems because they use different tools. Exactly - instead of using microservice-oriented tools, they use tools organized around monoliths. And that decision serves them well. That's the whole point.
      - deleted 4y ago
        [deleted]
- cmrdporcupine 4y ago
  Microservices move the complexity rather than solve it. The dependency boundaries between portions of the data model can never be cleanly demarcated -- because that's not how information works, especially in a growing business -- so there's always going to be either some loss of flexibility or addition of complexity over time. Individual developers getting out of each other's way just ends up getting pushed to getting in each other's way at release time as the matrix of dependencies between services explodes. Engineers jobs become more about the lives and dramas of the services they work on than about business domain. You end up building your organization and reporting structure around these services, rather than the business needs of the customer. And then you end up indirectly or directly shipping that org chart to the world in delays or bugs caused by your fragmentation. Instead of modeling facts about data and their relationships, and constructing the relational model which can capture this, the developer in the microservice model becomes bogged down in service roles and activities instead, again taking them away from the actual problem: which is organizing information and making it accessible to users/customers. It's a shell game. The Facebook monolith works because engineers there invested time in building the tooling you're complaining is not available to others. Same with Google: Google invested in F1, etc. because it evaluated the cost to do otherwise and it made sense to invest in infrastructure. Yes, small companies can't often afford this. Luckily they have two things on their side: Most small companies don't have a fraction of the scale issues that a FB or a Google have. So they can afford to monolith away for a lot longer than they seem to think they can, while they put in infrastructure to scale the monolith. The industry as a whole has invested a lot in making existing things scale. e.g. you can do things with a single Postgres instance that we never would have dreamed about 10 years ago. And when that falls over, there's replication, etc. And when that falls over, guess what? There's now high performance distributed ACID SQL databases available for $use. Microservices is surely one of the longest lived, biggest cargo cults in our industry. I've seen others come and go, but microservices really seems to cling. I think because it has the perception of breaking business problems down into very small elegant independent atomic pieces, so it has a very.. industrial revolution, automation, factory floor, economies of scale vibe. But that's not what it is. There are places for it, I'm sure. But systems with highly interelated data and quickly changing requirements are not well suited. IMHO.
  - 4y ago

com2kid 4y ago

One awesome and often overlooked benefit of microservices is how they simplify security/dependency updates. With a monolith, dependency updates, especially breaking ones, often mean either all development stops for a "code freeze" so the update can happen, or you have a team responsible for doing the update and they are trying to update code faster than other devs add new code. The result of this is that updates get pushed back to the last minute, or are never just done. I've seen old (ancient) versions of OpenSSL checked into codebases way too often. With microservices, you can have a team that isn't as busy take a sprint to update their codebase, carefully document best practices for fixing breaking changes, document best practices for testing the changes, and then spread the learning out to other teams, who can then update as they have time or based on importance / exposure of their maintained services. It is a much better way of doing things. It also means some teams can experiment with different technologies or tool chains and see how things work out. The cost of failure is low and there isn't an impact to other teams, and build systems for microservices tend to be much simpler than for monoliths (understatement...)

0xFACEFEED 4y ago
Microservices are a heavy handed way to draw boundaries around your software so that bad technical decisions don't bleed across different teams. Obviously there is some benefit to that but there is also a massive tradeoff - especially for certain types of software like complex UIs. > With a monolith, dependency updates, especially breaking ones, often mean either all development stops for a "code freeze" so the update can happen, or you have a team responsible for doing the update and they are trying to update code faster than other devs add new code. In all my years I've never seen a code freeze due to a dependency update. Maybe the project you were working was poorly engineered? > The result of this is that updates get pushed back to the last minute, or are never just done. I've seen old (ancient) versions of OpenSSL checked into codebases way too often. There should be nothing stopping you from running multiple versions of a dependency within a single monolothic project. > With microservices, you can have a team that isn't as busy take a sprint to update their codebase, carefully document best practices for fixing breaking changes, document best practices for testing the changes, and then spread the learning out to other teams, who can then update as they have time or based on importance / exposure of their maintained services. Gradual adoption of new dependencies has nothing to do with microservices.

vbezhenar 4y ago

Surely there should be something between gigantic monolith and micro services. I would call it service.

mirekrusin 4y ago
Yes, not sure why we have so many brainwashed fanatics who see world as hotdog and not-hotdog - microservices and monoliths - only.
- geekbird 4y ago
  Seriously! I think there is a good space for the concept of "mid-services" - cluster similar and interdependent services and service fragments together, so they split in logical groups for updating. It would be sort of like "Module A is authentication and session management, module B is data handling layer, and module 3 is the presentation and processing layer." Each of those under a microservices dogma would be two to four microservices struggling to interoperate. I read the book written by the dev that advocated for microservices. I wanted to throw it across the room, but it was an ebook. He literally went for over half the book before he even addressed operability. Everything was about developer convenience, not operating it with an eye toward user satisfaction. The guy was clueless.
strix_varius 4y ago
Typically, "monolith" implies services - ie, the "backing services" from a traditional 12-factor monolith: - https://12factor.net/backing-services https://12factor.net/backing-services Monolith vs. Microservices comes about because microservices proponents specifically set it up as an alternative to a traditional monolith + services stack: - https://www.nginx.com/learn/microservices-architecture/#:~:text=Microservices%20versus%20Monolithic%20Architecture https://www.nginx.com/learn/microservices-architecture/#:~:t...
- gedy 4y ago
  > Typically, "monolith" implies services - ie, the "backing services" I don't know if that's a common interpretation - in the monolith-loving companies I've worked with it very much meant a single ball of code (e.g. huge Rails app), not separate services.

mbrodersen 4y ago

That’s what Google does. Seems to work fine for them.

seadan83 4y ago
I think services became really popular from Amazon, and that quickly went to microservices IMHO. It's perhaps analogous to say: "That's what Amazon does. Seems to work fine for them". It's very interesting there are big techs now with well known monoliths vs Amazon services. Though, Amazon's services are so that any internally useful tool can be externally exposed and then rented out to generate more revenue. The reasons/benefits are very different compared to say decoupling or the other reasons given for service orientated architectures.