9 ms·

Microservices Architecture on Google Cloud

70 points by hliyan 5y ago

qaq 5y ago
Paying MS overhead is basically justifiable if you will need to scale your product development team for a single product to some ungodly size so not really applicable to 99.99% of the projects. The original hard constraint of not having a scalable enough RDBMS no longer exist you can use Spanner, CockroachDB etc.
newprint 5y ago
Microsoft published FAR better free book - .NET Microservices: Architecture for Containerized .NET Applications https://docs.microsoft.com/en-us/dotnet/architecture/microservices/ https://docs.microsoft.com/en-us/dotnet/architecture/microse... It is excellent reading for designing microservices in general, despite the fact that it targets .NET
bigus 5y ago
The comparison between microservices vs monolith was too simplistic and should have mentioned how much harder it is to deal with inconsistencies and distribution when working microservices. You want to be a certain company size before going all in on microservices.
- brickbrd 5y ago
  Can I find a better reading somewhere else on this?
  - bigus 5y ago
    Martin Fowler has some great comparisons about the two styles: https://www.martinfowler.com/articles/microservice-trade-offs.html https://www.martinfowler.com/articles/microservice-trade-off...
    - throwaway984393 5y ago
      That's the microservice take; they should also read Monolith First: https://www.martinfowler.com/bliki/MonolithFirst.html https://www.martinfowler.com/bliki/MonolithFirst.html
- throwaway984393 5y ago
  I don't believe that company size has anything to do with whether microservices work. That's the line people are sold, but companies large and small both suck at implementing microservices, and companies large and small can both implement monoliths successfully. I think it really has to come down to the individual case of a given product/project. One product may work way better as one architecture versus another, regardless of how many parts it may involve, due to the context, customers, use cases, organizational model, support model, regulations, etc. As a general rule, we tend to use the org chart to define our architectures (Conway's Law), but many org charts are just fucked up. We should absolutely fight an architectural design that is mimicking a poor organizational model.
  - danjac 5y ago
    Another problem with Conway's Law is that even if you create a perfect microservices architecture based on your org chart, all it takes is a couple reorgs and/or acquisitions before that architecture is no longer based on reality.
stevefan1999 5y ago
The more services you have, the higher the latency, to a point it is unbearable. So you know what the engineers do? They have to go back and make some services monolithic again. This is also applicable to microkernel model, where everything is almost processes intercommunicating either locally or remotely, but their performance is so bad they are either academically significant only or are simply abandoned. There are also critical services that bottlenecks the entire service plane. When one of them critical services (such as authentication service, which brought down Google recently) dies, it still have the cascading failures that monolithic services usually have. And sometimes it is impossible to properly make microservices when data consistency is highly important. These services (such as database and transaction based services) are critical by nature and is almost impossible to scale, and have to either using leader-election, which in a nutshell, distributed locks where the losers are merely standbys wasting resources on whether they can obtain the lock again, and will be highly susceptible to deadlock unless a reliable transactional memory source is involved (such as etcd leader lock in Kubernetes). Or replicate only. They are also what bugger up microservices to have degenerated into its monolithic equivalent or have to pay a great toll to get it scalable to survive in the microservice world. Well, to be honest, microservices in practice are just multiple monolithic services strapped together; We need to redesign all our current infrastructure to remove these critical bottlenecks. That's how you do microservices right.
- rackjack 5y ago
  Microkernels are not a good example. The assertion that they have worse performance in a way that can be meaningfully measured is a rumor, especially since we now have Fuchsia, not even mentioning other microkernels like seL4 and QNX.
  - aliswe 5y ago
    Yes Fuchsia, also why the downvotes? strange!
- dilyevsky 5y ago
  Consensus based systems aren’t wasting resources if data replication is desired and at least with raft and similar the replicas aren’t continuously trying to obtain the lock.
recursivedoubts 5y ago
pet theory: microservices is a psyop by big tech to make deploying and maintaining software so insanely difficult that future potential competitors are too tied up trying to keep the cloud equivalent of "hello world" afloat to present any real threat
- dboreham 5y ago
  There's a simpler explanation : Conway's Law.
  - adamors 5y ago
    Yeah, I don't understand why HN commenters refuse to understand this. The real advantages of microservices (I would say the only advantage but YMMV) is that it greatly simplifies development work, CI etc. when your organisation reaches a certain point. Even at 200 devs, working on just one monolith has a lot of friction. Open a pull request for something trivial and you'll have to wait hours for the CI to finish.
- analyst74 5y ago
  Some companies adopt micro-services to turn impossible problems into merely hard problems; some companies adopt micro-services to turn simple problems into hard problems.
- Aqueous 5y ago
  ever tried to scale monolithic development beyond 3 teams? the point of micro services is simple: to make technical infrastructure resemble the human organization it is designed to support. that’s it.
  - destitude 5y ago
    And I've been at a company where they did exactly this and turned out to be a complete mess and nearly bankrupted the company (it was even a unicorn).. Without the proper efficient layout and oversight of how a microservice architecture should work for your system you are better off reducing the engineers working on the monolith before telling everyone to switch it over to microservices. Otherwise if you can guarantee the above criteria then yes, I'd agree.
gravypod 5y ago
I highly recommend this topic: https://www.youtube.com/watch?v=-UKEPd2ipEk https://www.youtube.com/watch?v=-UKEPd2ipEk
throwaway984393 5y ago
A lot of the products I've seen built that attempt to be made of microservices are actually just a really inefficient monolith. A lot of different "services" that are all completely inter-dependent, or all use one database, or use libraries of the other "services", or use the exact same patterns/tools/templates/build systems/frameworks, or hard-code configuration values from other "services", or are actually stateful. My favorite thing is when one team can't deploy a change because they're waiting on another team to deploy a change.
- dustintrex 5y ago
  The "shared database microservices" antipattern is particularly pernicious. I've seen this a couple of times at enterprises where a mandate comes down from up high to microservice all the things, so they just split the frontend side of monolith into a couple of notionally separate apps but keep the old backend for all state. Ta-dah, microservices!
latch 5y ago
I don't think you can have a serious conversation about microservices without talking about poison messages and ordering guarantees (both with respect to each other and not). What does Pub/Sub provider to deal with those? (If you're interviewing somewhere and they say they have microservices, ask how they deal with poison messages. The answer can be telling).
- aliswe 5y ago
  Whats a good answer? I want more information ...
  - latch 5y ago
    Tongue-in-cheek answer (but really what I believe): Don't use microservices. Short answer: It's extremely difficult and, there isn't always a good solution. Different approaches are used on a _per consumer_ basis. That's the gist of what you want to hear. Long answer: In some cases ordering and lost messages don't matter so much, so any failed messages can just be thrown out. For some queues lost messages matters, but not ordering. So you can put the message at the back of the queue (possibly delayed if that's support), so you don't block other messages. Partitioning helps isolated poison messages to a vertical (e.g., customer_id). Always adding id and timestamp to each message can let consumers behave more smartly (e.g. throwing away out of order or already processed messages). But when none of that helps and ordering and delivery matter (which I think will make up most cases for any non-trivial system)...I'd love to know too. Good alerts to wake up developers? Then there's how to scale this and making it reliable What's the ordering guarantee across N consumers? What if a consumer nack's a message. What if there's prefetching?
- thatsamonad 5y ago
  I’m not sure if this fully addresses your comment but, similar to AWS SQS, Google Cloud Pub/Sub supports both message ordering[0] and dead letter topics[1]. [0](https://cloud.google.com/pubsub/docs/ordering#receiving_messages_in_order https://cloud.google.com/pubsub/docs/ordering#receiving_mess...) [1](https://cloud.google.com/pubsub/docs/handling-failures#setting-a-dead-letter-topic https://cloud.google.com/pubsub/docs/handling-failures#setti...)
westoque 5y ago
> It is extremely difficult to change a monolith’s technology or language or framework because all components are tightly coupled and dependent on each other. As a result, even relatively small changes can require lengthy development and deployment times. I disagree with this so much. I have personally worked with Rails application monoliths and Node.js microservices and I can tell you that making changes on the monolith is muliple times easier mostly depending on the code structure. I would take a properly structured monolith any day. This not only includes code/features but also deployments. Adding more services introduces more complexity in the deployment architecture as well. A good example of this is just by looking at the GitLab codebase https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab, it's a monolith but has good abstractions/structure vs say the Google Microservices Demo app https://github.com/GoogleCloudPlatform/microservices-demo https://github.com/GoogleCloudPlatform/microservices-demo which is not tightly coupled but introduces more complexity from implementation to deployment.
- qaq 5y ago
  It's obvious BS driven by cloud marketing because in monolith case you pay them for way fewer services. In Micro Services case a single client request will generate a number of downstream requests to multiple services driving the cloud provider's profits up. You also have a higher chance of using their highest margin tracing/telemetry offerings.
  - da39a3ee 5y ago
    Not sure whether you just meant that as humorous cynicism, but I think it has more to do with pain experienced by teams working on monoliths and them thinking “there has to be a better way”.
    - qaq 5y ago
      It's orders of magnitude easier to architect modular monolith than to build, test, deploy and manage a distributed system (e.g. microservices).
dikei 5y ago
I think many developers like microservices partly because they get to do green-field projects constantly: just start a fresh service when you need a new functionality, etc. Of course, in time, this might spiral into an uncontrollable mess without discipline, but it feels good at the beginning.
jillesvangurp 5y ago
Google does make dealing with dockerized services pretty easy. I'm CTO for a small startup. I actually know how to use Kubernetes, Terraform and have used a lot of configuration languages like puppet, ansible, and chef in the past fifteen years as well. However, what these have in common is that using them properly can more or less become a full time job for a few months. It's never simple or easy. I've been on multiple teams where somebody was doing that stuff full time for months on end. I don't have that kind of spare time. Either I do product development or I do devops but I can't do both and I certainly can't take two months out of my schedule for this stuff. So, I tend to look for solutions that minimize the amount of devops I need to do to the absolute bare minimum. Additionally, we are a bootstrapped setup, which means I also look for cost savings. Google cloud run is awesome for this. Last year I wanted to setup CI/CD for a simple service and have it run in Google cloud so we could point our web app at it. This took around fifteen minutes from start to finish. My starting point was a git repository that already had a Docker file in it. It took me a few mouse clicks to get cloud run to generate a cloud build and deploy the first version of that in an autoscaling cloud run environment. Awesome. Love it. The best part is that this setup cost us close to 0$/month while we were developing for the next six months. It had so few requests that we stayed below the freemium tier. It even does websockets now. We actually applied to get access to the beta of that. Later we ran into some limitations with background threads with cloud run. It aggressively throttles running containers when they are not serving a request. So, I decided to spin up a vm with the same docker container. Unlike AWS, spinning up a vm that runs a docker container is stupidly easy. I just grabbed the same docker container that we used for cloud run and passed that to the vm configuration (in the UI) and it launched in one go with a container optimized OS and our container came up after a few seconds. Every UI screen in Google cloud has a "copy this as a gcloud command line" as a well; so that became the basis for our CD (Github actions). We simply defined a github action that updates the vm. Prototype what you want in a UI and then copy and adapt the command for automation. Great stuff. Fast forward a year and I had a need to finally make this a bit more proper production environment. So, I bought a wildcard certificate, created a load balancer with it, defined an instance group with an instance template similar to the vm I prototyped earlier and I ended up with a nice auto-scaling service that we can deploy with zero down time. The hardest part was a bit of trial and error to figure out what I needed to do exactly. Took me about a day to get this right. Again, we have a github action that updates this; so full CI/CD. If you run monoliths, this stuff goes a long way. There's more to our setup of course but mostly I manage to not spend most of my days on devops topics. We will at some point out grow our current setup and hire a full time devops person to scale our setup a bit more responsibly. But actually, this setup already ticks most of my boxes. It's simple enough that I don't actually care about automating how it is created. It's flexible enough that it is easy to tweak. And it has things like logging, health checks, monitoring, alerting, etc. that is relatively easy to manage as well although maybe a little bare-bones. If/when we move to kubernetes, we'll end up roughly quadrupling our cost. Right now that would not really solve a problem I have. But it's probably a valid next step to take. Until then, monoliths and KISS are what we do.
joij2oi332 5y ago
microservices should be considered harmful in the most cases. one large company contacted me regarding a freelance work in a large team to rewrite their microservices back to monolith. The damage microservice architecture did to them is huge.

70 points by hliyan 5y ago

qaq 5y ago

Paying MS overhead is basically justifiable if you will need to scale your product development team for a single product to some ungodly size so not really applicable to 99.99% of the projects. The original hard constraint of not having a scalable enough RDBMS no longer exist you can use Spanner, CockroachDB etc.

newprint 5y ago

Microsoft published FAR better free book - .NET Microservices: Architecture for Containerized .NET Applications https://docs.microsoft.com/en-us/dotnet/architecture/microservices/ https://docs.microsoft.com/en-us/dotnet/architecture/microse... It is excellent reading for designing microservices in general, despite the fact that it targets .NET

bigus 5y ago

The comparison between microservices vs monolith was too simplistic and should have mentioned how much harder it is to deal with inconsistencies and distribution when working microservices. You want to be a certain company size before going all in on microservices.

brickbrd 5y ago
Can I find a better reading somewhere else on this?
- bigus 5y ago
  Martin Fowler has some great comparisons about the two styles: https://www.martinfowler.com/articles/microservice-trade-offs.html https://www.martinfowler.com/articles/microservice-trade-off...
  - throwaway984393 5y ago
    That's the microservice take; they should also read Monolith First: https://www.martinfowler.com/bliki/MonolithFirst.html https://www.martinfowler.com/bliki/MonolithFirst.html
throwaway984393 5y ago
I don't believe that company size has anything to do with whether microservices work. That's the line people are sold, but companies large and small both suck at implementing microservices, and companies large and small can both implement monoliths successfully. I think it really has to come down to the individual case of a given product/project. One product may work way better as one architecture versus another, regardless of how many parts it may involve, due to the context, customers, use cases, organizational model, support model, regulations, etc. As a general rule, we tend to use the org chart to define our architectures (Conway's Law), but many org charts are just fucked up. We should absolutely fight an architectural design that is mimicking a poor organizational model.
- danjac 5y ago
  Another problem with Conway's Law is that even if you create a perfect microservices architecture based on your org chart, all it takes is a couple reorgs and/or acquisitions before that architecture is no longer based on reality.

stevefan1999 5y ago

The more services you have, the higher the latency, to a point it is unbearable. So you know what the engineers do? They have to go back and make some services monolithic again. This is also applicable to microkernel model, where everything is almost processes intercommunicating either locally or remotely, but their performance is so bad they are either academically significant only or are simply abandoned. There are also critical services that bottlenecks the entire service plane. When one of them critical services (such as authentication service, which brought down Google recently) dies, it still have the cascading failures that monolithic services usually have. And sometimes it is impossible to properly make microservices when data consistency is highly important. These services (such as database and transaction based services) are critical by nature and is almost impossible to scale, and have to either using leader-election, which in a nutshell, distributed locks where the losers are merely standbys wasting resources on whether they can obtain the lock again, and will be highly susceptible to deadlock unless a reliable transactional memory source is involved (such as etcd leader lock in Kubernetes). Or replicate only. They are also what bugger up microservices to have degenerated into its monolithic equivalent or have to pay a great toll to get it scalable to survive in the microservice world. Well, to be honest, microservices in practice are just multiple monolithic services strapped together; We need to redesign all our current infrastructure to remove these critical bottlenecks. That's how you do microservices right.

rackjack 5y ago
Microkernels are not a good example. The assertion that they have worse performance in a way that can be meaningfully measured is a rumor, especially since we now have Fuchsia, not even mentioning other microkernels like seL4 and QNX.
- aliswe 5y ago
  Yes Fuchsia, also why the downvotes? strange!
dilyevsky 5y ago
Consensus based systems aren’t wasting resources if data replication is desired and at least with raft and similar the replicas aren’t continuously trying to obtain the lock.

recursivedoubts 5y ago

pet theory: microservices is a psyop by big tech to make deploying and maintaining software so insanely difficult that future potential competitors are too tied up trying to keep the cloud equivalent of "hello world" afloat to present any real threat

dboreham 5y ago
There's a simpler explanation : Conway's Law.
- adamors 5y ago
  Yeah, I don't understand why HN commenters refuse to understand this. The real advantages of microservices (I would say the only advantage but YMMV) is that it greatly simplifies development work, CI etc. when your organisation reaches a certain point. Even at 200 devs, working on just one monolith has a lot of friction. Open a pull request for something trivial and you'll have to wait hours for the CI to finish.
analyst74 5y ago
Some companies adopt micro-services to turn impossible problems into merely hard problems; some companies adopt micro-services to turn simple problems into hard problems.
Aqueous 5y ago
ever tried to scale monolithic development beyond 3 teams? the point of micro services is simple: to make technical infrastructure resemble the human organization it is designed to support. that’s it.
- destitude 5y ago
  And I've been at a company where they did exactly this and turned out to be a complete mess and nearly bankrupted the company (it was even a unicorn).. Without the proper efficient layout and oversight of how a microservice architecture should work for your system you are better off reducing the engineers working on the monolith before telling everyone to switch it over to microservices. Otherwise if you can guarantee the above criteria then yes, I'd agree.

gravypod 5y ago

I highly recommend this topic: https://www.youtube.com/watch?v=-UKEPd2ipEk https://www.youtube.com/watch?v=-UKEPd2ipEk

throwaway984393 5y ago

A lot of the products I've seen built that attempt to be made of microservices are actually just a really inefficient monolith. A lot of different "services" that are all completely inter-dependent, or all use one database, or use libraries of the other "services", or use the exact same patterns/tools/templates/build systems/frameworks, or hard-code configuration values from other "services", or are actually stateful. My favorite thing is when one team can't deploy a change because they're waiting on another team to deploy a change.

dustintrex 5y ago
The "shared database microservices" antipattern is particularly pernicious. I've seen this a couple of times at enterprises where a mandate comes down from up high to microservice all the things, so they just split the frontend side of monolith into a couple of notionally separate apps but keep the old backend for all state. Ta-dah, microservices!

latch 5y ago

I don't think you can have a serious conversation about microservices without talking about poison messages and ordering guarantees (both with respect to each other and not). What does Pub/Sub provider to deal with those? (If you're interviewing somewhere and they say they have microservices, ask how they deal with poison messages. The answer can be telling).

aliswe 5y ago
Whats a good answer? I want more information ...
- latch 5y ago
  Tongue-in-cheek answer (but really what I believe): Don't use microservices. Short answer: It's extremely difficult and, there isn't always a good solution. Different approaches are used on a _per consumer_ basis. That's the gist of what you want to hear. Long answer: In some cases ordering and lost messages don't matter so much, so any failed messages can just be thrown out. For some queues lost messages matters, but not ordering. So you can put the message at the back of the queue (possibly delayed if that's support), so you don't block other messages. Partitioning helps isolated poison messages to a vertical (e.g., customer_id). Always adding id and timestamp to each message can let consumers behave more smartly (e.g. throwing away out of order or already processed messages). But when none of that helps and ordering and delivery matter (which I think will make up most cases for any non-trivial system)...I'd love to know too. Good alerts to wake up developers? Then there's how to scale this and making it reliable What's the ordering guarantee across N consumers? What if a consumer nack's a message. What if there's prefetching?
thatsamonad 5y ago
I’m not sure if this fully addresses your comment but, similar to AWS SQS, Google Cloud Pub/Sub supports both message ordering[0] and dead letter topics[1]. [0](https://cloud.google.com/pubsub/docs/ordering#receiving_messages_in_order https://cloud.google.com/pubsub/docs/ordering#receiving_mess...) [1](https://cloud.google.com/pubsub/docs/handling-failures#setting-a-dead-letter-topic https://cloud.google.com/pubsub/docs/handling-failures#setti...)

westoque 5y ago

> It is extremely difficult to change a monolith’s technology or language or framework because all components are tightly coupled and dependent on each other. As a result, even relatively small changes can require lengthy development and deployment times. I disagree with this so much. I have personally worked with Rails application monoliths and Node.js microservices and I can tell you that making changes on the monolith is muliple times easier mostly depending on the code structure. I would take a properly structured monolith any day. This not only includes code/features but also deployments. Adding more services introduces more complexity in the deployment architecture as well. A good example of this is just by looking at the GitLab codebase https://gitlab.com/gitlab-org/gitlab https://gitlab.com/gitlab-org/gitlab, it's a monolith but has good abstractions/structure vs say the Google Microservices Demo app https://github.com/GoogleCloudPlatform/microservices-demo https://github.com/GoogleCloudPlatform/microservices-demo which is not tightly coupled but introduces more complexity from implementation to deployment.

qaq 5y ago
It's obvious BS driven by cloud marketing because in monolith case you pay them for way fewer services. In Micro Services case a single client request will generate a number of downstream requests to multiple services driving the cloud provider's profits up. You also have a higher chance of using their highest margin tracing/telemetry offerings.
- da39a3ee 5y ago
  Not sure whether you just meant that as humorous cynicism, but I think it has more to do with pain experienced by teams working on monoliths and them thinking “there has to be a better way”.
  - qaq 5y ago
    It's orders of magnitude easier to architect modular monolith than to build, test, deploy and manage a distributed system (e.g. microservices).

dikei 5y ago

I think many developers like microservices partly because they get to do green-field projects constantly: just start a fresh service when you need a new functionality, etc. Of course, in time, this might spiral into an uncontrollable mess without discipline, but it feels good at the beginning.

jillesvangurp 5y ago

Google does make dealing with dockerized services pretty easy. I'm CTO for a small startup. I actually know how to use Kubernetes, Terraform and have used a lot of configuration languages like puppet, ansible, and chef in the past fifteen years as well. However, what these have in common is that using them properly can more or less become a full time job for a few months. It's never simple or easy. I've been on multiple teams where somebody was doing that stuff full time for months on end. I don't have that kind of spare time. Either I do product development or I do devops but I can't do both and I certainly can't take two months out of my schedule for this stuff. So, I tend to look for solutions that minimize the amount of devops I need to do to the absolute bare minimum. Additionally, we are a bootstrapped setup, which means I also look for cost savings. Google cloud run is awesome for this. Last year I wanted to setup CI/CD for a simple service and have it run in Google cloud so we could point our web app at it. This took around fifteen minutes from start to finish. My starting point was a git repository that already had a Docker file in it. It took me a few mouse clicks to get cloud run to generate a cloud build and deploy the first version of that in an autoscaling cloud run environment. Awesome. Love it. The best part is that this setup cost us close to 0$/month while we were developing for the next six months. It had so few requests that we stayed below the freemium tier. It even does websockets now. We actually applied to get access to the beta of that. Later we ran into some limitations with background threads with cloud run. It aggressively throttles running containers when they are not serving a request. So, I decided to spin up a vm with the same docker container. Unlike AWS, spinning up a vm that runs a docker container is stupidly easy. I just grabbed the same docker container that we used for cloud run and passed that to the vm configuration (in the UI) and it launched in one go with a container optimized OS and our container came up after a few seconds. Every UI screen in Google cloud has a "copy this as a gcloud command line" as a well; so that became the basis for our CD (Github actions). We simply defined a github action that updates the vm. Prototype what you want in a UI and then copy and adapt the command for automation. Great stuff. Fast forward a year and I had a need to finally make this a bit more proper production environment. So, I bought a wildcard certificate, created a load balancer with it, defined an instance group with an instance template similar to the vm I prototyped earlier and I ended up with a nice auto-scaling service that we can deploy with zero down time. The hardest part was a bit of trial and error to figure out what I needed to do exactly. Took me about a day to get this right. Again, we have a github action that updates this; so full CI/CD. If you run monoliths, this stuff goes a long way. There's more to our setup of course but mostly I manage to not spend most of my days on devops topics. We will at some point out grow our current setup and hire a full time devops person to scale our setup a bit more responsibly. But actually, this setup already ticks most of my boxes. It's simple enough that I don't actually care about automating how it is created. It's flexible enough that it is easy to tweak. And it has things like logging, health checks, monitoring, alerting, etc. that is relatively easy to manage as well although maybe a little bare-bones. If/when we move to kubernetes, we'll end up roughly quadrupling our cost. Right now that would not really solve a problem I have. But it's probably a valid next step to take. Until then, monoliths and KISS are what we do.

joij2oi332 5y ago

microservices should be considered harmful in the most cases. one large company contacted me regarding a freelance work in a large team to rewrite their microservices back to monolith. The damage microservice architecture did to them is huge.