10 ms·
Microservices grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to gr
by jihadjihad 3y ago
Microservices
grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too
seem very confusing to grug
https://grugbrain.dev/#grug-on-microservices https://grugbrain.dev/#grug-on-microservices
- sethammons 3y agogrug not experience large teams stepping on each other's domains and data models, locking in a given implementation and requiring large, organizational efforts to to get features out at the team level. Team velocity is empowered via microservices and controlling their own data stores. "We want to modernize how we access our FOO_TABLE for SCALE_REASONS by moving it to DynamoDB out of MySQL - unfortunately, 32 of our 59 teams are directly accessing the FOO_TABLE or directly accessing private methods on our classes. Due to competing priorities, those teams cannot do the work to move to using our FOO_SERVICE and they can't change their query method to use a sharded table. To scale our FOO_TABLE will now be a multi-quarter effort providing the ability for teams to slow roll their update. After a year or two, we should be able to retire the old method that is on fire right now. In the meanwhile, enjoy oncall." Compare this to a microservice: Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. Users of the API report that it is now much faster.
- jjtheblunt 3y ago> Team velocity is empowered via microservices and controlling their own data stores. Bologna. Choosing clear abstractions is an enabler of focus, but that doesn’t necessarily imply those abstractions are a network call away.
- sethammons 3y agoour team of 300 - we _can't_ enforce the clear abstractions. New dev gets hired, team feels pressured to deliver despite leadership saying to prioritize quality, they are not aware of all the access controls, they push a PR, it gets merged. We have an org wide push to get more linting and more checks in place. The damage is done and now we have a multi-quarter effort to re-organize all our code. This _can_ be enforced via well designed modules. I've just not seen that succeed. Anywhere. Microservices are a pain for smaller teams and you have to have CI and observability and your pains shift and are different. But for stepping on eachother? I've found microservices to be a super power for velocity in these cases. Can microservices be a shitshow? Absolutely, esp. when they share data stores or have circular dependencies. They also allow teams to be uncoupled assuming they don't break their API.
- ChrisMarshallNY 3y ago> leadership saying to prioritize quality Well, that's different. My experience is that "leadership" often finds quality to be expensive and unnecessary overhead. That's one reason that I stayed at a company that took Quality seriously. It introduced many issues that folks, hereabouts would find unbearable, but they consistently shipped some of the highest-Quality (and expensive) kit in the world. Quality is not cheap, and it is not easy. It is also not really the path to riches, so it is often actively discouraged by managers.
- nicoburns 3y ago> They also allow teams to be uncoupled assuming they don't break their API. Presumably you would still need tooling to enforce teams not breaking their API, and also to prevent people just modifying services to expose private internals that should not be part of the public API?
- jacquesm 3y agoYou can't really retrofit culture and behavior, you can only very gradually move towards a particular goal and usually that's a multi-year effort, if it can be done at all.
- shadowgovt 3y agoIf you want to change that model, there's three things that need to happen: 1) engineering needs a reason to embrace abstraction. Because the only thing that stops a new cowboy engineer is their peers explaining to them in what ways this isn't the Wild Wesst. One can assume if rules would benefit the team, they'd be doing them already, so why don't they? Maybe they perceive feature output velocity to be too high to risk changing method. Maybe decisionmaking power rests in the hands of one system architect who is holding the whole machine in their head so things that look complex to others seem simple to them (in that case, the team needs to spread around peer review signoff responsibilities on purpose, so one engineer can't be the decisionmaker and the architecture itself must self-describe). Maybe (this is how I see it usually happen) they were a three-person startup and complexity crept up on them like a boiling frog. Whatever the reason, if you're gonna convince them otherwise, someone's gonna have to generate hard data on how changing the abstraction could make their jobs easier. 2) If management has no idea what "prioritize quality" means (meaning no metrics by which to measure it and no real grasp of the art of software engineering), the engineers will interpret buzzwords as noise and route around them. Management needs to give actionable goals other than "release feature X by Y date" if they want to change engineering culture. That can take many forms (I've seen rewarded fixit weeks and externally-reported issue burndown charts as two good examples). 3) Engineering leadership needs time and bandwidth to do training so they can see outside their prairie-dog hole over to how other teams solve problems. Otherwise, they get locked into the solutions they know, and the only way new approaches ever enter the team is by hires. And the key thing is: microservices may not be the tool for the job when all is said and done. This is an approach to give your engineering team ways to discover the right patterns, not to trick them into doing microservices. Your engineering team, at the end of the day, is still the best-equipped people to know the machinery of the product and how to shape it.
- simonw 3y ago> Team realizes their table wont scale, but their data is provided via API. They plan and execute the migration next sprint. ... followed by howls of anguish from the rest of the business when it turns out they were relying on reports generated from a data warehouse which incorporated a copy of that MySQL database and was being populated by an undocumented, not-in-version-control cron script running on a PC under a long-departed team member's desk. (I'm not saying this is good, but it's not an unlikely scenario.)
- sethammons 3y agoanother reason why you provide data only over API - don't reach into my tables and lock me into an implementation.
- simonw 3y agoAn approach I like better than "only access my data via API" is this: The team that maintains the service is also responsible for how that service is represented in the data warehouse. The data warehouse tables - effectively denormalized copies of the data that the service stores - are treated as another API contract - they are clearly documented and tested as such. If the team refactors, they also update the scripts that populate the data warehouse. If that results in specific columns etc becoming invalid they document that in their release notes, and ideally notify other affected teams.
- hermanradtke 3y agoThis same thing can be applied to contracts when firing events, etc. I point people to https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha... and use the same approach to ownership.
- simonw 3y agoYeah, having a documented stream of published events in Kafka is a similar API contract the team can be responsible for - it might even double as the channel through which the data warehouse is populated.
- shadowgovt 3y agoAs with so many software solutions, the success of microservices is predicated upon having sufficient prognostication about how the system will be used to recognize where the cut-points are. When I hear success stories like that, I have to ask "Is there some inherent benefit to the abstraction or did you get lucky in picking your cleave-points?"
- esafak 3y agoThat comes with experience, but you can let time be the judge if you factor your monolith early enough. If the factorization proves stable, proceed with carving it into microservices.
- charcircuit 3y agoOther teams can access your data directly even if it's in your own separate database.
- sally_glance 3y agoThis applies to performance optimizations which leave the interface untouched, but there are other scenarios, for example: - Performance optimizations which can't be realized without changing the interface model. For example, FOO_TABLE should actually be BAR and BAZ with different update patterns to allow for efficient caching and querying. - Domain model updates, adding/updating/removing new properties or entities. This kind of update will still require the 32 consumers to upgrade. The API-based approach has benefits in terms of the migration process and backwards-compatibility though (a HTTP API is much easier to version than a DB schema, although you can also do this on the DB level by only allowing queries on views).
- crabbone 3y agoFound the author of CISCO / Java Spring documentation. I honestly expected people used passive-aggressive corpo-slang only for work / ironically. But this reads intentionally obfuscated. But, to answer to the substance: you for some reason assumed that whoever designed first solution was an idiot and whoever designed the second was, at least clairvoyant. The problem you describe isn't a result of inevitable design decisions. You just described a situation where someone screwed up doing something and didn't screw up doing something else. And that led you to believe that whatever that something else is, it's easier to design. The reality of the situation is, unfortunately, the reverse. Upgrading microservices is much harder than replacing components in monolithic systems because it's easier to discover all users of the feature. There are, in general, fewer components in monolithic systems, so less things will need to change. Deployment of a monolithic system will be much more likely to discover problems created by incorrectly implemented upgrade. In my experience of dealing with both worlds, microservices tend to create a maze-like system where nobody can be sure if any change will not adversely affect some other part of the system due to the distributed and highly fragmented nature of such systems. So, your ideas about upgrades are uncorroborated by practice. If you want to be able to update with more ease, you should choose a smaller, more cohesive system.
- perrylaj 3y agoI think I view the situation in a similar fashion as you. There's absolutely nothing preventing a well architected modular monolith from establishing domain/module-specific persistence that is accessible only through APIs and connectable only through the owning domain/module. To accomplish that requires a decent application designer/architect, and yes, it needs to be considered ahead of time, but it's 100% doable and scalable across teams if needed. There are definitely reasons why a microservice architecture could make sense for an organization, but "we don't have good application engineers/architects" should not be one of them. Going to a distributed microservice model because your teams aren't good enough to manage a modular monolith sounds like a recipe for disaster.
- nijave 3y agoMicroservices don't magically fix shared schema headaches. Getting abstraction and APIs right is the solution regardless of whether it's in-memory or over the network. Instead of microservices, you could add a static analysis build step that checks if code in packages is calling private or protected interfaces in different packages. That would also help enforce service boundaries without introducing the network as the boundary.
- tubthumper8 3y agoI guess I'm confused by the suggestion - doesn't that static analysis step to check that code isn't calling private interfaces already exist, and is called a "compiler"?
- com2kid 3y agoOr how about "we want to update a 3rd party library due to a critical security issue, but we can't because that library is used in 500 different parts of the code and no way in hell can we stop development across the entire org for multiple weeks". With microservices, you deploy the updates to public facing services first, write any learnings down, pass it along to the next most vulnerable tier. Heck on multiple occasions I've been part of projects where just updating the build system for a monolithic codebase was a year+ long effort involving dozens of engineers trying to work around commits from everyone else. Compare this to a microservice model where you just declare all new services get the new build tools. If the new tools come with a large enough carrot (e.g. new version of typescript) teams may very well update their own build tooling to the latest stuff without anyone even asking them to!
- danmaz74 3y agoTeam size is probably the most important factor that should influence the choice about microservices. Unfortunately, there was a period when it looked like every project and every team had to adopt them or be declared a dinosaur.
- agumonkey 3y agonew role: lead grug
- insanitybit 3y agoNetwork calls are a powerful thing to introduce. It means that you have an impassable boundary, one that is actually physically enforced - your two services have to treat each other as if they are isolated. Isolation is not anything to scoff at, it's one of the most powerful features you can encode into your software. Isolation can improve performance, it can create fault boundaries, it can provide security boundaries, etc. This is the same foundational concept behind the actor model - instead of two components being able to share and mutate one another's memory, you have two isolated systems (actors, microservices) that can only communicate over a defined protocol.
- nicoburns 3y ago> Network calls are a powerful thing to introduce. It means that you have an impassable boundary, one that is actually physically enforced - your two services have to treat each other as if they are isolated. That is not too true at all. I've seen "microservice" setups where one microservice depends on the state within another microservice. And even cases where service A calls into service B which calls back into service A, relying on the state from the initial call being present. Isolation is good, but microservices are neither necessary nor sufficient to enforce it.
- insanitybit 3y agoWell, I'd say you've seen SoA setups that do that, maybe. But those don't sound like microservices :) Perhaps that's not a strong point though. Let me be a bit clearer on my point because I was wrong to say that you have to treat a service as being totally isolated, what I should have said that they are isolated, whether you treat them that way or not. There is a physical boundary between two computers. You can try to ignore that boundary, you can implement distributed transactions, etc, but the boundary is there - if you do the extra work to try to pretend it isn't, that's a lot of extra work to do the wrong thing. Concretely, you can write: rpc_call(&mut my_state) But under the hood what has to happen, physically, is that your state has to be copied to the other service, the service can return a new state (or an update), and the caller can then mutate the state locally. There is no way for you to actually transfer a mutable reference to your own memory to another computer (and a service should be treated as if it may be on another computer, even if it is colocated) without obscene shenanigans. You can try to abstract around that isolation to give the appearance of shared mutable state but it is just an abstraction, it is effectively impossible to implement that directly. But shared mutable state is trivial without the process boundary. It's just... every function call. Any module can take a mutable pointer and modify it. And that's great for lots of things, of course, you give up isolation sometimes when you need to.
- manicennui 3y agoI just wanted to note that static typing isn't required for autocomplete. JetBrains has IDEs for languages like Ruby and Python that can do it. If you open the REPL in a recent version of Ruby you get much of what you expect from an IDE with a statically typed language (with regards to autocomplete and syntax checking).
- manicennui 3y agoAlso, DRY is not about repeated code. This drives me crazy. Ruby developers love to make code worse by trying to "DRY it up".
- Capricorn2481 3y agoWhat are you replying to? The article is about how Dry shouldn't be over applied. Dry is literally "Don't Repeat Yourself" and is definitely pushed for cleaning up redundant code, so it's not unreasonable for people to think that's what about. It's only recently that people have pointed out that there's a difference between Duplicated code and Repeated code.
- manicennui 3y agoRedundant code is not code that looks the same. It's only reasonable for people to believe it is about code that looks the same if they have never bothered to learn what it means.
- Capricorn2481 3y agoThat's what I said, yes. There's a difference between duplicated logic and redundant code. And it's clear the Grug article agrees with that.
- 998244353 3y agoYou are correct, but static typing does make it a lot easier. Working with Rider feels like working with an IDE that fully understands the code, at least structurally. Working with PyCharm feels like working with an IDE that makes intelligent guesses.