4 ms·

grug not experience large teams stepping on each other's domains and data models, locking in a given implementation and requiring large, organizational efforts

grug not experience large teams stepping on each other's domains and data models, locking in a given implementation and requiring large, organizational efforts to to get features out at the team level. Team velocity is empowered via microservices and controlling their own data stores. "We want to modernize how we access our FOO_TABLE for SCALE_REASONS by moving it to DynamoDB out of MySQL - unfortunately, 32 of our 59 teams are directly accessing the FOO_TABLE or directly accessing private methods on our classes. Due to competing priorities, those teams cannot do the work to move to using our FOO_SERVICE and they can't change their query method to use a sharded table. To scale our FOO_TABLE will now be a multi-quarter effort providing the ability for teams to slow roll their update. After a year or two, we should be able to retire the old method that is on fire right now. In the meanwhile, enjoy oncall." Compare this to a microservice: Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. Users of the API report that it is now much faster.

jjtheblunt 3y ago
> Team velocity is empowered via microservices and controlling their own data stores. Bologna. Choosing clear abstractions is an enabler of focus, but that doesn’t necessarily imply those abstractions are a network call away.
- sethammons 3y ago
  our team of 300 - we _can't_ enforce the clear abstractions. New dev gets hired, team feels pressured to deliver despite leadership saying to prioritize quality, they are not aware of all the access controls, they push a PR, it gets merged. We have an org wide push to get more linting and more checks in place. The damage is done and now we have a multi-quarter effort to re-organize all our code. This _can_ be enforced via well designed modules. I've just not seen that succeed. Anywhere. Microservices are a pain for smaller teams and you have to have CI and observability and your pains shift and are different. But for stepping on eachother? I've found microservices to be a super power for velocity in these cases. Can microservices be a shitshow? Absolutely, esp. when they share data stores or have circular dependencies. They also allow teams to be uncoupled assuming they don't break their API.
  - ChrisMarshallNY 3y ago
    > leadership saying to prioritize quality Well, that's different. My experience is that "leadership" often finds quality to be expensive and unnecessary overhead. That's one reason that I stayed at a company that took Quality seriously. It introduced many issues that folks, hereabouts would find unbearable, but they consistently shipped some of the highest-Quality (and expensive) kit in the world. Quality is not cheap, and it is not easy. It is also not really the path to riches, so it is often actively discouraged by managers.
  - nicoburns 3y ago
    > They also allow teams to be uncoupled assuming they don't break their API. Presumably you would still need tooling to enforce teams not breaking their API, and also to prevent people just modifying services to expose private internals that should not be part of the public API?
  - jacquesm 3y ago
    You can't really retrofit culture and behavior, you can only very gradually move towards a particular goal and usually that's a multi-year effort, if it can be done at all.
  - shadowgovt 3y ago
    If you want to change that model, there's three things that need to happen: 1) engineering needs a reason to embrace abstraction. Because the only thing that stops a new cowboy engineer is their peers explaining to them in what ways this isn't the Wild Wesst. One can assume if rules would benefit the team, they'd be doing them already, so why don't they? Maybe they perceive feature output velocity to be too high to risk changing method. Maybe decisionmaking power rests in the hands of one system architect who is holding the whole machine in their head so things that look complex to others seem simple to them (in that case, the team needs to spread around peer review signoff responsibilities on purpose, so one engineer can't be the decisionmaker and the architecture itself must self-describe). Maybe (this is how I see it usually happen) they were a three-person startup and complexity crept up on them like a boiling frog. Whatever the reason, if you're gonna convince them otherwise, someone's gonna have to generate hard data on how changing the abstraction could make their jobs easier. 2) If management has no idea what "prioritize quality" means (meaning no metrics by which to measure it and no real grasp of the art of software engineering), the engineers will interpret buzzwords as noise and route around them. Management needs to give actionable goals other than "release feature X by Y date" if they want to change engineering culture. That can take many forms (I've seen rewarded fixit weeks and externally-reported issue burndown charts as two good examples). 3) Engineering leadership needs time and bandwidth to do training so they can see outside their prairie-dog hole over to how other teams solve problems. Otherwise, they get locked into the solutions they know, and the only way new approaches ever enter the team is by hires. And the key thing is: microservices may not be the tool for the job when all is said and done. This is an approach to give your engineering team ways to discover the right patterns, not to trick them into doing microservices. Your engineering team, at the end of the day, is still the best-equipped people to know the machinery of the product and how to shape it.
simonw 3y ago
> Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. ... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk. (I'm not saying this is good, but it's not an unlikely scenario.)
- sethammons 3y ago
  another reason why you provide data only over API - don't reach into my tables and lock me into an implementation.
  - simonw 3y ago
    An approach I like better than "only access my data via API" is this: The team that maintains the service is also responsible for how that service is represented in the data warehouse. The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such. If the team refactors, they also update the scripts that populate the data warehouse. If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.
    - hermanradtke 3y ago
      This same thing can be applied to contracts when firing events, etc. I point people to https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... and use the same approach to ownership.
      - simonw 3y ago
        Yeah, having a documented stream of published events in Kafka is a similar API contract the team can be responsible for - it might even double as the channel through which the data warehouse is populated.
shadowgovt 3y ago
As with so many software solutions, the success of microservices is predicated upon having sufficient prognostication about how the system will be used to recognize where the cut-points are. When I hear success stories like that, I have to ask "Is there some inherent benefit to the abstraction or did you get lucky in picking your cleave-points?"
- esafak 3y ago
  That comes with experience, but you can let time be the judge if you factor your monolith early enough. If the factorization proves stable, proceed with carving it into microservices.
charcircuit 3y ago
Other teams can access your data directly even if it's in your own separate database.
sally_glance 3y ago
This applies to performance optimizations which leave the interface untouched, but there are other scenarios, for example: - Performance optimizations which can't be realized without changing the interface model. For example, FOO_TABLE should actually be BAR and BAZ with different update patterns to allow for efficient caching and querying. - Domain model updates, adding/updating/removing new properties or entities. This kind of update will still require the 32 consumers to upgrade. The API-based approach has benefits in terms of the migration process and backwards-compatibility though (a HTTP API is much easier to version than a DB schema, although you can also do this on the DB level by only allowing queries on views).
crabbone 3y ago
Found the author of CISCO / Java Spring documentation. I honestly expected people used passive-aggressive corpo-slang only for work / ironically. But this reads intentionally obfuscated. But, to answer to the substance: you for some reason assumed that whoever designed first solution was an idiot and whoever designed the second was, at least clairvoyant. The problem you describe isn't a result of inevitable design decisions. You just described a situation where someone screwed up doing something and didn't screw up doing something else. And that led you to believe that whatever that something else is, it's easier to design. The reality of the situation is, unfortunately, the reverse. Upgrading microservices is much harder than replacing components in monolithic systems because it's easier to discover all users of the feature. There are, in general, fewer components in monolithic systems, so less things will need to change. Deployment of a monolithic system will be much more likely to discover problems created by incorrectly implemented upgrade. In my experience of dealing with both worlds, microservices tend to create a maze-like system where nobody can be sure if any change will not adversely affect some other part of the system due to the distributed and highly fragmented nature of such systems. So, your ideas about upgrades are uncorroborated by practice. If you want to be able to update with more ease, you should choose a smaller, more cohesive system.
- perrylaj 3y ago
  I think I view the situation in a similar fashion as you. There's absolutely nothing preventing a well architected modular monolith from establishing domain/module-specific persistence that is accessible only through APIs and connectable only through the owning domain/module. To accomplish that requires a decent application designer/architect, and yes, it needs to be considered ahead of time, but it's 100% doable and scalable across teams if needed. There are definitely reasons why a microservice architecture could make sense for an organization, but "we don't have good application engineers/architects" should not be one of them. Going to a distributed microservice model because your teams aren't good enough to manage a modular monolith sounds like a recipe for disaster.
nijave 3y ago
Microservices don't magically fix shared schema headaches. Getting abstraction and APIs right is the solution regardless of whether it's in-memory or over the network. Instead of microservices, you could add a static analysis build step that checks if code in packages is calling private or protected interfaces in different packages. That would also help enforce service boundaries without introducing the network as the boundary.
- tubthumper8 3y ago
  I guess I'm confused by the suggestion - doesn't that static analysis step to check that code isn't calling private interfaces already exist, and is called a "compiler"?
com2kid 3y ago
Or how about "we want to update a 3rd party library due to a critical security issue, but we can't because that library is used in 500 different parts of the code and no way in hell can we stop development across the entire org for multiple weeks". With microservices, you deploy the updates to public facing services first, write any learnings down, pass it along to the next most vulnerable tier. Heck on multiple occasions I've been part of projects where just updating the build system for a monolithic codebase was a year+ long effort involving dozens of engineers trying to work around commits from everyone else. Compare this to a microservice model where you just declare all new services get the new build tools. If the new tools come with a large enough carrot (e.g. new version of typescript) teams may very well update their own build tooling to the latest stuff without anyone even asking them to!
danmaz74 3y ago
Team size is probably the most important factor that should influence the choice about microservices. Unfortunately, there was a period when it looked like every project and every team had to adopt them or be declared a dinosaur.

jjtheblunt 3y ago

> Team velocity is empowered via microservices and controlling their own data stores. Bologna. Choosing clear abstractions is an enabler of focus, but that doesn’t necessarily imply those abstractions are a network call away.

sethammons 3y ago
our team of 300 - we _can't_ enforce the clear abstractions. New dev gets hired, team feels pressured to deliver despite leadership saying to prioritize quality, they are not aware of all the access controls, they push a PR, it gets merged. We have an org wide push to get more linting and more checks in place. The damage is done and now we have a multi-quarter effort to re-organize all our code. This _can_ be enforced via well designed modules. I've just not seen that succeed. Anywhere. Microservices are a pain for smaller teams and you have to have CI and observability and your pains shift and are different. But for stepping on eachother? I've found microservices to be a super power for velocity in these cases. Can microservices be a shitshow? Absolutely, esp. when they share data stores or have circular dependencies. They also allow teams to be uncoupled assuming they don't break their API.
- ChrisMarshallNY 3y ago
  > leadership saying to prioritize quality Well, that's different. My experience is that "leadership" often finds quality to be expensive and unnecessary overhead. That's one reason that I stayed at a company that took Quality seriously. It introduced many issues that folks, hereabouts would find unbearable, but they consistently shipped some of the highest-Quality (and expensive) kit in the world. Quality is not cheap, and it is not easy. It is also not really the path to riches, so it is often actively discouraged by managers.
- nicoburns 3y ago
  > They also allow teams to be uncoupled assuming they don't break their API. Presumably you would still need tooling to enforce teams not breaking their API, and also to prevent people just modifying services to expose private internals that should not be part of the public API?
- jacquesm 3y ago
  You can't really retrofit culture and behavior, you can only very gradually move towards a particular goal and usually that's a multi-year effort, if it can be done at all.
- shadowgovt 3y ago
  If you want to change that model, there's three things that need to happen: 1) engineering needs a reason to embrace abstraction. Because the only thing that stops a new cowboy engineer is their peers explaining to them in what ways this isn't the Wild Wesst. One can assume if rules would benefit the team, they'd be doing them already, so why don't they? Maybe they perceive feature output velocity to be too high to risk changing method. Maybe decisionmaking power rests in the hands of one system architect who is holding the whole machine in their head so things that look complex to others seem simple to them (in that case, the team needs to spread around peer review signoff responsibilities on purpose, so one engineer can't be the decisionmaker and the architecture itself must self-describe). Maybe (this is how I see it usually happen) they were a three-person startup and complexity crept up on them like a boiling frog. Whatever the reason, if you're gonna convince them otherwise, someone's gonna have to generate hard data on how changing the abstraction could make their jobs easier. 2) If management has no idea what "prioritize quality" means (meaning no metrics by which to measure it and no real grasp of the art of software engineering), the engineers will interpret buzzwords as noise and route around them. Management needs to give actionable goals other than "release feature X by Y date" if they want to change engineering culture. That can take many forms (I've seen rewarded fixit weeks and externally-reported issue burndown charts as two good examples). 3) Engineering leadership needs time and bandwidth to do training so they can see outside their prairie-dog hole over to how other teams solve problems. Otherwise, they get locked into the solutions they know, and the only way new approaches ever enter the team is by hires. And the key thing is: microservices may not be the tool for the job when all is said and done. This is an approach to give your engineering team ways to discover the right patterns, not to trick them into doing microservices. Your engineering team, at the end of the day, is still the best-equipped people to know the machinery of the product and how to shape it.

simonw 3y ago

> Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. ... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk. (I'm not saying this is good, but it's not an unlikely scenario.)

sethammons 3y ago
another reason why you provide data only over API - don't reach into my tables and lock me into an implementation.
- simonw 3y ago
  An approach I like better than "only access my data via API" is this: The team that maintains the service is also responsible for how that service is represented in the data warehouse. The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such. If the team refactors, they also update the scripts that populate the data warehouse. If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.
  - hermanradtke 3y ago
    This same thing can be applied to contracts when firing events, etc. I point people to https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... and use the same approach to ownership.
    - simonw 3y ago
      Yeah, having a documented stream of published events in Kafka is a similar API contract the team can be responsible for - it might even double as the channel through which the data warehouse is populated.

shadowgovt 3y ago

As with so many software solutions, the success of microservices is predicated upon having sufficient prognostication about how the system will be used to recognize where the cut-points are. When I hear success stories like that, I have to ask "Is there some inherent benefit to the abstraction or did you get lucky in picking your cleave-points?"

esafak 3y ago
That comes with experience, but you can let time be the judge if you factor your monolith early enough. If the factorization proves stable, proceed with carving it into microservices.

charcircuit 3y ago

Other teams can access your data directly even if it's in your own separate database.

sally_glance 3y ago

This applies to performance optimizations which leave the interface untouched, but there are other scenarios, for example: - Performance optimizations which can't be realized without changing the interface model. For example, FOO_TABLE should actually be BAR and BAZ with different update patterns to allow for efficient caching and querying. - Domain model updates, adding/updating/removing new properties or entities. This kind of update will still require the 32 consumers to upgrade. The API-based approach has benefits in terms of the migration process and backwards-compatibility though (a HTTP API is much easier to version than a DB schema, although you can also do this on the DB level by only allowing queries on views).

crabbone 3y ago

Found the author of CISCO / Java Spring documentation. I honestly expected people used passive-aggressive corpo-slang only for work / ironically. But this reads intentionally obfuscated. But, to answer to the substance: you for some reason assumed that whoever designed first solution was an idiot and whoever designed the second was, at least clairvoyant. The problem you describe isn't a result of inevitable design decisions. You just described a situation where someone screwed up doing something and didn't screw up doing something else. And that led you to believe that whatever that something else is, it's easier to design. The reality of the situation is, unfortunately, the reverse. Upgrading microservices is much harder than replacing components in monolithic systems because it's easier to discover all users of the feature. There are, in general, fewer components in monolithic systems, so less things will need to change. Deployment of a monolithic system will be much more likely to discover problems created by incorrectly implemented upgrade. In my experience of dealing with both worlds, microservices tend to create a maze-like system where nobody can be sure if any change will not adversely affect some other part of the system due to the distributed and highly fragmented nature of such systems. So, your ideas about upgrades are uncorroborated by practice. If you want to be able to update with more ease, you should choose a smaller, more cohesive system.

perrylaj 3y ago
I think I view the situation in a similar fashion as you. There's absolutely nothing preventing a well architected modular monolith from establishing domain/module-specific persistence that is accessible only through APIs and connectable only through the owning domain/module. To accomplish that requires a decent application designer/architect, and yes, it needs to be considered ahead of time, but it's 100% doable and scalable across teams if needed. There are definitely reasons why a microservice architecture could make sense for an organization, but "we don't have good application engineers/architects" should not be one of them. Going to a distributed microservice model because your teams aren't good enough to manage a modular monolith sounds like a recipe for disaster.

nijave 3y ago

Microservices don't magically fix shared schema headaches. Getting abstraction and APIs right is the solution regardless of whether it's in-memory or over the network. Instead of microservices, you could add a static analysis build step that checks if code in packages is calling private or protected interfaces in different packages. That would also help enforce service boundaries without introducing the network as the boundary.

tubthumper8 3y ago
I guess I'm confused by the suggestion - doesn't that static analysis step to check that code isn't calling private interfaces already exist, and is called a "compiler"?

com2kid 3y ago

Or how about "we want to update a 3rd party library due to a critical security issue, but we can't because that library is used in 500 different parts of the code and no way in hell can we stop development across the entire org for multiple weeks". With microservices, you deploy the updates to public facing services first, write any learnings down, pass it along to the next most vulnerable tier. Heck on multiple occasions I've been part of projects where just updating the build system for a monolithic codebase was a year+ long effort involving dozens of engineers trying to work around commits from everyone else. Compare this to a microservice model where you just declare all new services get the new build tools. If the new tools come with a large enough carrot (e.g. new version of typescript) teams may very well update their own build tooling to the latest stuff without anyone even asking them to!

danmaz74 3y ago

Team size is probably the most important factor that should influence the choice about microservices. Unfortunately, there was a period when it looked like every project and every team had to adopt them or be declared a dinosaur.