6 ms·

I would like to propose a new variant on an old maxim: Premature microservice-izing is the root of all evil. I've worked with an app that had been factored in

I would like to propose a new variant on an old maxim: Premature microservice-izing is the root of all evil. I've worked with an app that had been factored into half a dozen microservices. Each of these needed two EC2 instances for redundancy. They weren't using docker or any other container infrastructure and were treating EC2 instances as pets instead of cattle, which meant that a LOT of effort was spent maintaining the dozen or so instances. Each of these instances was running at < 1% load average! So in this case the microservice architecture was also costing a significant amount of money. My takeaway: don't build an app from scratch using microservices, especially if all you're splitting apart is web handling which is almost never going to be where your application spends a significant amount of time. Instead, build apps in the straightforward monolithic way, and only split things out into microservices when there is a very clear reason that has been identified from performance monitoring or some other metric gathering.

vidarh 10y ago
My takeaway from your example is not to not use microservices, but to get proper operational processes in place. At most I'd stretch to conceding that they should have done that first.
- pbreit 10y ago
  No. You have to spend as much energy as possible on end user features of which dev ops is the opposite.
  - nine_k 10y ago
    Reliability and speed are features, and often the features that make users stay or go. If your devops is inadequate, so that you have low visibility into your product's performance, can't degrade gracefully, and have trouble deploying things without significant downtime, investing in proper devops infrastructure and processes may directly help your bottom line.
    - pbreit 10y ago
      That stuff is table stakes and relatively easy to deliver in 2016. Compared to achieving product/market fit and getting people to pay for your service, comparatively simple.
      - vidarh 10y ago
        Comparatively simple, but without getting at least the basics right you'll be moving comparatively slowly as you end up with time-suck after time-suck to compensate for lacking processes.
    - eropple 10y ago
      As somebody who's actually paid to implement devops practices for many clients at scale, this is nuts. The only performance argument for a service-oriented architecture (and let's be really real, the overhead of HTTP means that microservices rarely have a performance argument at all) is when and only when you have specific requirements that cannot be solved by horizontally scaling a monolithic system. Until you have an actual, metrically verified demonstration of the failure of horizontally scaling your monolithic architecture, sticking with it will in almost every instance outperform your "microservice architecture". Reliability, too, is a joke argument; building a single application (whether or not you internally segment the code with an eye towards later service segmentation) means that you have N instances of every component, rather than N of some, M of others, and so on. And nothing related to deploying your very bloggable microservice should differ from a reasonably developed monolithic system in the first place. The set of companies that have any sort of reliability or performance-oriented need a "microservice architecture" is three, maybe four orders of magnitude smaller than the set of companies building web applications. They--and you--have been sold on a trend that is actively antithetical to the goals of delivering a product. Build your application. Use an auto-scaling group and an ELB. You have more important things to do until you reach the scale where you can pay someone (hi) to do them.
      - nine_k 10y ago
        I did not try to defend microservices. I tried to note that devops is not an opposite to customer-facing value. I think most of your points are valid (wherever my experience is enough to judge).
  - vidarh 10y ago
    If you don't have proper devops processes in place, you'll be spending a progressively larger proportion of your time handling manually what should be automated, and firefighting problems that shouldn't even occur, as your system grows more complex. A pre-requisite to "spend as much energy as possible on end user features" is to have some sort of foundation to build on that can grow with your platform, or you might as well build on quicksand.
origami777 10y ago
This approach makes a lot of sense. The only other scenario where I think it makes to go microservice is when development progress is impacted by organizational structure. That seems to be a huge reason why corps like amazon broke things down into microservice teams; so productivity could stay high in extremely complex and high headcount environments.
BinaryIdiot 10y ago
While the architecture of the app you had to deal with is absolutely insane I'm not sure I'd start with monolithic either. Typically I've seen people go monolithic and when they start realizing they need to break things up everything is glued together so tightly they basically need to completely rewrite to get to microservices. I think there is a middle ground. Especially if you can start writing common libraries for the other components to use.
- pbreit 10y ago
  If you manage to get to the point where you truly need to micro-ize, it is only at that point that you can actually afford to.
- solatic 10y ago
  Tight vs loose coupling is not a new concept. You can write a monolith with bounded-context components which are loosely coupled to each other with an eye towards possibly putting those bounded contexts in different containers in the future. Unfortunately, this requires an actual architect to think about the design of the monolith ahead of time, and most startups either have trouble recruiting anyone above the junior level, confuse the responsibilities of the project manager (who, in the early days, is typically the CEO/CTO) with the responsibilities of the architect, or simply cannot afford to put an architect on payroll.
baconforce 10y ago
What you are referring to is the Monolith First approach: http://martinfowler.com/bliki/MonolithFirst.html http://martinfowler.com/bliki/MonolithFirst.html. The key is to identify service boundaries early on and keep them relatively clean so that separating them into microservices at a later point is easier.
mr337 10y ago
"and were treating EC2 instances as pets instead of cattle" There is sooo much truth to this statement. Never be afraid to put down an instance, and if so your doing it wrong!
- deleted 10y ago
  [deleted]
- hueving 10y ago
  Where do you store your data? Would you be able to terminate all of your instances and not lose data?
pbreit 10y ago
That is exactly my company's situation. And at least 1/2 energy spent on "infrastructure" instead of end-user features. And the whole thing could easily run monolithic on one small instance.
morgante 10y ago
I'm amazed that your takeaway from that case was about microservices. The problem was obviously outdated operations approaches (not using containers, hand-tuning each server, etc.).
- mightybyte 10y ago
  Newer operations approaches may have helped make the state of affairs tractable, but it does not at all argue for that microservice architecture. There was literally no value gained from running a dozen web servers instead of two. And that being the case, the straightforward but "outdated" way of managing one or two servers manually would almost definitely be cheaper than learning and implementing the more modern approach.
- hueving 10y ago
  If you think everyone who isn't using containers is outdated, you are living in an unrealistic bubble.
  - morgante 10y ago
    I don't think containers are necessary for every use case, but for the described one they definitely are.
ktRolster 10y ago
> Each of these needed two EC2 instances for redundancy. Doesn't EC2 already have redundancy built in?
- ec109685 10y ago
  No, vms can be terminated if the parent host crashes.
solipsism 10y ago
This is not a very good example. These people were just doing it wrong. If they had been running something like AppEngine, Heroku, GKE, ECS, etc, they wouldn't have had to worry about maintaining VMs, replicating app instances, proper VM sizing, etc. It's not the microservice architecture that was costing them money, it was their poor technical choices.

vidarh 10y ago

My takeaway from your example is not to not use microservices, but to get proper operational processes in place. At most I'd stretch to conceding that they should have done that first.

pbreit 10y ago
No. You have to spend as much energy as possible on end user features of which dev ops is the opposite.
- nine_k 10y ago
  Reliability and speed are features, and often the features that make users stay or go. If your devops is inadequate, so that you have low visibility into your product's performance, can't degrade gracefully, and have trouble deploying things without significant downtime, investing in proper devops infrastructure and processes may directly help your bottom line.
  - pbreit 10y ago
    That stuff is table stakes and relatively easy to deliver in 2016. Compared to achieving product/market fit and getting people to pay for your service, comparatively simple.
    - vidarh 10y ago
      Comparatively simple, but without getting at least the basics right you'll be moving comparatively slowly as you end up with time-suck after time-suck to compensate for lacking processes.
  - eropple 10y ago
    As somebody who's actually paid to implement devops practices for many clients at scale, this is nuts. The only performance argument for a service-oriented architecture (and let's be really real, the overhead of HTTP means that microservices rarely have a performance argument at all) is when and only when you have specific requirements that cannot be solved by horizontally scaling a monolithic system. Until you have an actual, metrically verified demonstration of the failure of horizontally scaling your monolithic architecture, sticking with it will in almost every instance outperform your "microservice architecture". Reliability, too, is a joke argument; building a single application (whether or not you internally segment the code with an eye towards later service segmentation) means that you have N instances of every component, rather than N of some, M of others, and so on. And nothing related to deploying your very bloggable microservice should differ from a reasonably developed monolithic system in the first place. The set of companies that have any sort of reliability or performance-oriented need a "microservice architecture" is three, maybe four orders of magnitude smaller than the set of companies building web applications. They--and you--have been sold on a trend that is actively antithetical to the goals of delivering a product. Build your application. Use an auto-scaling group and an ELB. You have more important things to do until you reach the scale where you can pay someone (hi) to do them.
    - nine_k 10y ago
      I did not try to defend microservices. I tried to note that devops is not an opposite to customer-facing value. I think most of your points are valid (wherever my experience is enough to judge).
- vidarh 10y ago
  If you don't have proper devops processes in place, you'll be spending a progressively larger proportion of your time handling manually what should be automated, and firefighting problems that shouldn't even occur, as your system grows more complex. A pre-requisite to "spend as much energy as possible on end user features" is to have some sort of foundation to build on that can grow with your platform, or you might as well build on quicksand.

origami777 10y ago

This approach makes a lot of sense. The only other scenario where I think it makes to go microservice is when development progress is impacted by organizational structure. That seems to be a huge reason why corps like amazon broke things down into microservice teams; so productivity could stay high in extremely complex and high headcount environments.

BinaryIdiot 10y ago

While the architecture of the app you had to deal with is absolutely insane I'm not sure I'd start with monolithic either. Typically I've seen people go monolithic and when they start realizing they need to break things up everything is glued together so tightly they basically need to completely rewrite to get to microservices. I think there is a middle ground. Especially if you can start writing common libraries for the other components to use.

pbreit 10y ago
If you manage to get to the point where you truly need to micro-ize, it is only at that point that you can actually afford to.
solatic 10y ago
Tight vs loose coupling is not a new concept. You can write a monolith with bounded-context components which are loosely coupled to each other with an eye towards possibly putting those bounded contexts in different containers in the future. Unfortunately, this requires an actual architect to think about the design of the monolith ahead of time, and most startups either have trouble recruiting anyone above the junior level, confuse the responsibilities of the project manager (who, in the early days, is typically the CEO/CTO) with the responsibilities of the architect, or simply cannot afford to put an architect on payroll.

baconforce 10y ago

What you are referring to is the Monolith First approach: http://martinfowler.com/bliki/MonolithFirst.html http://martinfowler.com/bliki/MonolithFirst.html. The key is to identify service boundaries early on and keep them relatively clean so that separating them into microservices at a later point is easier.

mr337 10y ago

"and were treating EC2 instances as pets instead of cattle" There is sooo much truth to this statement. Never be afraid to put down an instance, and if so your doing it wrong!

deleted 10y ago
[deleted]
hueving 10y ago
Where do you store your data? Would you be able to terminate all of your instances and not lose data?

pbreit 10y ago

That is exactly my company's situation. And at least 1/2 energy spent on "infrastructure" instead of end-user features. And the whole thing could easily run monolithic on one small instance.

morgante 10y ago

I'm amazed that your takeaway from that case was about microservices. The problem was obviously outdated operations approaches (not using containers, hand-tuning each server, etc.).

mightybyte 10y ago
Newer operations approaches may have helped make the state of affairs tractable, but it does not at all argue for that microservice architecture. There was literally no value gained from running a dozen web servers instead of two. And that being the case, the straightforward but "outdated" way of managing one or two servers manually would almost definitely be cheaper than learning and implementing the more modern approach.
hueving 10y ago
If you think everyone who isn't using containers is outdated, you are living in an unrealistic bubble.
- morgante 10y ago
  I don't think containers are necessary for every use case, but for the described one they definitely are.

ktRolster 10y ago

> Each of these needed two EC2 instances for redundancy. Doesn't EC2 already have redundancy built in?

ec109685 10y ago
No, vms can be terminated if the parent host crashes.

solipsism 10y ago

This is not a very good example. These people were just doing it wrong. If they had been running something like AppEngine, Heroku, GKE, ECS, etc, they wouldn't have had to worry about maintaining VMs, replicating app instances, proper VM sizing, etc. It's not the microservice architecture that was costing them money, it was their poor technical choices.