7 ms·

This strikes me as overly cynical. My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able t

This strikes me as overly cynical. My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface. Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that > cross team meetings and project management every time you want to ship a feature is somehow avoidable in tech?

irjustin 3y ago
> This strikes me as overly cynical. > My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface. > Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that This is overly cynical to you likely because you haven't experienced maintaining micro-services in the long term. Being able to "break away" with a common interface is literally the microservice tag line. It's completely true too. Standing up a service w/ an interface is incredibly fast, rewarding and watching it hum is beautiful. The problem starts when there's a bug that's upstream of your service. It's not too hard to get all 15+ services running on your laptop, but the problem is the other teams no longer let external members directly deploy the services they maintain (that one time someone from another team deployed a big bug). So now you've got to get PR approval and seemingly no one wants to review your bug. So you ask for time from the Product Owner and it gets forgotten because they're really busy too. So you go to an eng manager, etc etc... The above scenario plays out SO many times. I've been there in all various forms. As a blunt statement - any team under 30 members, monolith.
- hmeh 3y ago
  If you have to "get all 15+ services running on your laptop", you've already lost. Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. The only reason we have Docker on our machine is to test the images we deploy (just to make sure they build and run). You have experienced what happens when web developers cosplay as software architects. That's when you get "microservices". It's a travesty, and it has ruined an entire generation of software developers who now believe that partitioning and autonomy are just myths. So, as a blunt statement - your experience is typical. That is, it's average. That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.
  - kamikaz1k 3y ago
    Just to clarify, what are components referring to? Are they independent web services? And what is the total size of the developers you worked with? Are the 12 just your team or the entire engineering org? Edit: found from another comment > We don’t call them microservices, because we aren’t web developers pretending to do architecture and recreating the mistakes of DCOM, CORBA, and Web Services. Most of them are autonomous event sourced components. The others are stand alone web applications that are stitched together with Nginx routing and SSI.
    - hmeh 3y ago
      Components are either autonomous event sourced back end components (using Eventide) or independent web applications that are combined with Nginx routes and SSI. The UI is (mostly) server-rendered Rails. The users have no idea they are hitting 20 or so different web applications, but the developers sure feel the productivity boost of every application being a small application that is (relatively) independent from one another. A dozen or so is the entire software development organization. Edit: Indeed! If you have any other questions, ask away. Most people think what we do isn't actually possible, but it is.
  - irjustin 3y ago
    > So, as a blunt statement - your experience is typical. That is, it's average. Definitely a fair judgement and would agree that this constitutes poor architecting on many many companies' parts including ones that I've been a part of. > Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team? More directly, what's wrong with a monolith at the dozen person team size?
    - hmeh 3y ago
      > Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team? > More directly, what's wrong with a monolith at the dozen person team size? Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well. The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning. Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested. When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity. We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing. When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption. The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible). Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control. The list goes on and on. Here are a few articles, feel free to poke around the others and ask any follow up questions. Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/master/partitions-and-compositions.md https://github.com/aaronjensen/software-development/blob/mas... The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/master/monoliths.md https://github.com/aaronjensen/software-development/blob/mas...
- 8n4vidtmkvmk 3y ago
  Standing up a service across the globe is non-trivial even with world class tools. You still run into issues like services not being properly co located with other services they need to call so you end up paying for inter continental hops. And cold connections because the new service doesn't have much traffic yet. And maybe DB leaders are only in North America so Asia has terrible perf.
roenxi 3y ago
> I'd love to be able to separate myself and my team from others by an agreed-upon interface. Don't doubt it, but that is what the grandparent was going to - you don't want to be held accountable for the success of the overall product, you want a defined contract that you maintain and the value-add of that is someone else's problem. The dysfunction of a business is made up of individuals doing what makes the most sense for them locally. It isn't unreasonable to want microservices, it just happens that the incentives that make developers like microservices also tend towards systems that lack accountability for achieving system-level outcomes.
- jpk 3y ago
  Isn't this precisely what leadership is for? If you've implemented an architecture that mirrors your org chart, then the interfaces aren't just in software, they exist between managers as well. System-level success depends on the managers and the directors above them to steer the boat in a way that places accountability on the teams and components of the architecture necessary to ship a feature, or whatever.
  - kaashif 3y ago
    > If you've implemented an architecture that mirrors your org chart In my experience this is almost invariably what happens. I would replace the if with a when. Wow, you're telling me our three teams want to write three separate services? No way. And the teams in New York want to use one technology while the teams in San Francisco want to use something else? Say it ain't so!
  - onion2k 3y ago
    Isn't this precisely what leadership is for? If your devs are just implementing whatever system design is coming from your tech leads, and having no opportunity to have an input or their own, you'll find you have very unhappy devs who don't hang around very long. That churn will make it impossible to build a decent system.
    - jpk 3y ago
Nextgrid 3y ago
> I'd love to be able to separate myself and my team from others by an agreed-upon interface. My experience suggests that speccing out such an interface well is a tremendous amount of work, and if you get it wrong (you most likely will) or if it needs to evolve it will generate lots of work since now it's set in stone and very hard to change so you're likely going to pay "interest" by working around its shortcomings for the foreseeable future.
- the_gipsy 3y ago
  The interface will evolve, and in fact that is the hard part of software development.
- twhitmore 3y ago
  Difficulty & cost evolving the interface between microservices can seriously hinder improving the actual product. I've seen the misguided adoption of microservices affect a couple of companies & products.

irjustin 3y ago

> This strikes me as overly cynical. > My current job is working with around 20 other engineers on an extremely bloated and coupled monolith. I'd love to be able to separate myself and my team from others by an agreed-upon interface. > Yes, "agreed-upon" is certainly doing some heavy lifting there, but I don't think it's realistic to expect that This is overly cynical to you likely because you haven't experienced maintaining micro-services in the long term. Being able to "break away" with a common interface is literally the microservice tag line. It's completely true too. Standing up a service w/ an interface is incredibly fast, rewarding and watching it hum is beautiful. The problem starts when there's a bug that's upstream of your service. It's not too hard to get all 15+ services running on your laptop, but the problem is the other teams no longer let external members directly deploy the services they maintain (that one time someone from another team deployed a big bug). So now you've got to get PR approval and seemingly no one wants to review your bug. So you ask for time from the Product Owner and it gets forgotten because they're really busy too. So you go to an eng manager, etc etc... The above scenario plays out SO many times. I've been there in all various forms. As a blunt statement - any team under 30 members, monolith.

hmeh 3y ago
If you have to "get all 15+ services running on your laptop", you've already lost. Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. The only reason we have Docker on our machine is to test the images we deploy (just to make sure they build and run). You have experienced what happens when web developers cosplay as software architects. That's when you get "microservices". It's a travesty, and it has ruined an entire generation of software developers who now believe that partitioning and autonomy are just myths. So, as a blunt statement - your experience is typical. That is, it's average. That is, you experienced an average team doing average work misguided by average bad advice. It says absolutely nothing about software design. It only speaks to the fat part of the bell curve doing what it always does.
- kamikaz1k 3y ago
  Just to clarify, what are components referring to? Are they independent web services? And what is the total size of the developers you worked with? Are the 12 just your team or the entire engineering org? Edit: found from another comment > We don’t call them microservices, because we aren’t web developers pretending to do architecture and recreating the mistakes of DCOM, CORBA, and Web Services. Most of them are autonomous event sourced components. The others are stand alone web applications that are stitched together with Nginx routing and SSI.
  - hmeh 3y ago
    Components are either autonomous event sourced back end components (using Eventide) or independent web applications that are combined with Nginx routes and SSI. The UI is (mostly) server-rendered Rails. The users have no idea they are hitting 20 or so different web applications, but the developers sure feel the productivity boost of every application being a small application that is (relatively) independent from one another. A dozen or so is the entire software development organization. Edit: Indeed! If you have any other questions, ask away. Most people think what we do isn't actually possible, but it is.
- irjustin 3y ago
  > So, as a blunt statement - your experience is typical. That is, it's average. Definitely a fair judgement and would agree that this constitutes poor architecting on many many companies' parts including ones that I've been a part of. > Every one of our 100+ components maintained by our dozen person team can be run and tested on its own. Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team? More directly, what's wrong with a monolith at the dozen person team size?
  - hmeh 3y ago
    > Can you elaborate on this? What is the primary benefit of running 100-300 components vs a monolith specifically with a 12 person team? > More directly, what's wrong with a monolith at the dozen person team size? Great question. It's honestly a hard question to answer because the real answer requires tacit knowledge, but I'll try anyway because you asked. I'll link a couple articles of longer form writing that may help to answer as well. The benefit ultimately is productivity. That productivity comes from autonomy, which comes from partitioning. Each component can be worked, tested, and deployed on its own. We never need to spin up more than one at a time, which means any time we are working on our code, we are typically working on 1/400th of it, and we know we are because we have 400 separate GitHub repositories. We can't accidentally get peanut butter in our mayonaise. Each part is independently built and tested. When it's time to upgrade Rails, we can upgrade one application. This is 1/20th of our Rails code. That's a smaller batch. We know from basic flow principles that (see: Principles of Product Development Flow, Lean, etc.) that small batches are better for productivity. We rarely have merge conflicts. We don't do pull requests. We branch when we need to. We are often working on a brand new, never been deployed project. We don't have our tests running in CI. Our tests run in less than a second in most projects on our laptops. With those that have UI, our tests typically run in less than 30 seconds which include thorough Capybara (UI Interaction) testing. When we are working in any application, we only need to consider its direct efferents when making a change to any interfaces it exposes. We can trace those changes through easily when we need to make them. Because we practice software design, we don't often need to make large sweeping changes, but when we do, we can do them methodically without disruption. The "worst" part is that some things are tedious. But we can automate tedious things. We can invent around tedious things. We used to have to deploy 15 web applications any time we had a style change. Now we deploy one application that supplies the CSS for all our applications via SSI (server-side-include). If we need to deploy 70 back end components, we can script that (with manual verification at each step). We can even spread that out across a couple team members and do it as an ensemble. It takes about an hour to deploy everything. We maybe do this once a month or so (when something highly afferent changes -- and guess what, we structure our design so that afferent things change as infrequently as possible). Most things we work on we work until they are "complete". That is, they rarely change after that. We have components that have been in production for 3 years and haven't been substantially touched aside from Ruby upgrades and the like. They are still perfectly in control. The list goes on and on. Here are a few articles, feel free to poke around the others and ask any follow up questions. Partitions and Compositions: https://github.com/aaronjensen/software-development/blob/master/partitions-and-compositions.md https://github.com/aaronjensen/software-development/blob/mas... The Mythical Monolith: https://github.com/aaronjensen/software-development/blob/master/monoliths.md https://github.com/aaronjensen/software-development/blob/mas...
8n4vidtmkvmk 3y ago
Standing up a service across the globe is non-trivial even with world class tools. You still run into issues like services not being properly co located with other services they need to call so you end up paying for inter continental hops. And cold connections because the new service doesn't have much traffic yet. And maybe DB leaders are only in North America so Asia has terrible perf.

roenxi 3y ago

> I'd love to be able to separate myself and my team from others by an agreed-upon interface. Don't doubt it, but that is what the grandparent was going to - you don't want to be held accountable for the success of the overall product, you want a defined contract that you maintain and the value-add of that is someone else's problem. The dysfunction of a business is made up of individuals doing what makes the most sense for them locally. It isn't unreasonable to want microservices, it just happens that the incentives that make developers like microservices also tend towards systems that lack accountability for achieving system-level outcomes.

jpk 3y ago
Isn't this precisely what leadership is for? If you've implemented an architecture that mirrors your org chart, then the interfaces aren't just in software, they exist between managers as well. System-level success depends on the managers and the directors above them to steer the boat in a way that places accountability on the teams and components of the architecture necessary to ship a feature, or whatever.
- kaashif 3y ago
  > If you've implemented an architecture that mirrors your org chart In my experience this is almost invariably what happens. I would replace the if with a when. Wow, you're telling me our three teams want to write three separate services? No way. And the teams in New York want to use one technology while the teams in San Francisco want to use something else? Say it ain't so!
- onion2k 3y ago
  Isn't this precisely what leadership is for? If your devs are just implementing whatever system design is coming from your tech leads, and having no opportunity to have an input or their own, you'll find you have very unhappy devs who don't hang around very long. That churn will make it impossible to build a decent system.
  - jpk 3y ago

Nextgrid 3y ago

> I'd love to be able to separate myself and my team from others by an agreed-upon interface. My experience suggests that speccing out such an interface well is a tremendous amount of work, and if you get it wrong (you most likely will) or if it needs to evolve it will generate lots of work since now it's set in stone and very hard to change so you're likely going to pay "interest" by working around its shortcomings for the foreseeable future.

the_gipsy 3y ago
The interface will evolve, and in fact that is the hard part of software development.
twhitmore 3y ago
Difficulty & cost evolving the interface between microservices can seriously hinder improving the actual product. I've seen the misguided adoption of microservices affect a couple of companies & products.