8 ms·

Adopting Microservices at Netflix: Lessons for Architectural Design

207 points by davidkellis 12y ago

davidw 12y ago
My comment, slightly edited, from the previous posting of this, at https://news.ycombinator.com/item?id=9106813 https://news.ycombinator.com/item?id=9106813 > One kind of coupling that people tend to overlook as they transition to a microservices architecture is database coupling, where all services talk to the same database and updating a service means changing the schema. You need to split the database up and denormalize it. That sounds like a decision you wouldn't want to take lightly; the kind of thing you might do once your company is already big. I wouldn't want to start out that way though, it sounds like a recipe for a mess.
- anonymousDan 12y ago
  Yes I just came in here to write a comment along those lines. I mean surely you're opening a whole can of worms in terms of consistency etc. I get the feeling Netflix happens not to have use cases where strong consistency is a requirement. I'd be interested to get more detail about how they went about the transition - even just pointers to more on the meta-data management tools they use.
  - davidkellis 12y ago
    The consistency problem is an open question in my mind. I definitely don't like the idea of having some data synchronization tool to fix the inconsistent data across services problem. I wonder what the best practice is for maintaining data consistency across services. Does anyone know?
    - shanemhansen 12y ago
      Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.
      - davidkellis 12y ago
        But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Person gets deleted? In that case, we've got a consistency problem, even though each service owned its data. Is the best practice to not use entity services?
        cgh 12y ago
        The "each service owns its data" scenario shouldn't take precedence over cases like this, where consistency is aligned with obvious business rules. If Person gets deleted, then "on delete cascade" should take care of that Person's Address and Billing records. For updates and maybe reads, it's a different story.
        davidkellis 12y ago
        cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?
        cgh 12y ago
        Sorry, I should have been more clear. I'm assuming a shared database. So cascading deletes would be defined in the table's schema. Let's pretend we're using Postgresql: \d Person [A bunch of table schema stuff] Referenced by: TABLE "Billing" CONSTRAINT "billing_id_fk" FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE (I typed that off the type of my head so it might not be quite correct.)
        dsp1234 12y ago
        The original position being argued though was "... where all services talk to the same database... You need to split the database up and denormalize it.". So the basic premise is that there is no shared database, and thus having the database enforce cascading deletes is not an option.
- BerislavLopac 12y ago
  If the two (the database schema and the microservices API) are designed and maintained separately, this assertion (that "updating a service means changing the schema") is not necessarily true. They have separate responsibilities -- the database is persisting the data, and the services provide some business logic, and while they might need to change together this is not always the case.
- nl 12y ago
  Database-as-an-integration-layer is a well known anti-pattern. It's tempting because it's easy, and at first glance it seems to solve lots of problems (consistency, communication etc). It's a big mistake. If multiple service are reading and writing from the same DB it rapidly becomes impossible to change things. Things like input validation changes suddenly have to be implemented in multiple places (which is hard), and done simultaneously (which often becomes hard enough to stop work being done). In a microservice based approach, the data flows though shared services, and so changes can be done in fewer places. (Note that read-only, reporting-style databases are separate. I think there is a good case for these being shared)
  - dragonwriter 12y ago
    Multiple services reading from the same DB can -- and should -- be using different accounts that have access to different objects that abstract the underlying data model from the various clients and keep them loosely coupled. This is almost as old of a recommendation as relational DBs themselves. You can create a tightly coupled design with a shared DB, but there is nothing inherent with shared DB integration requires that.
jfoutz 12y ago
If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.
- deleted 12y ago
  [deleted]
- ryderm 12y ago
  So you have short timeouts and retries that are load balanced to different nodes. But ideally your services are fast even in their 99 percentiles so this isn't an issue. This is much easier to achieve in a small service than a huge complex one.
  - jfoutz 12y ago
    Um. maybe. 5 machines behind a load balancer. Normal case, load is even, 100 requests to each server. One server starts running into trouble, exceeding timeouts. your load is now ~125 per server, because each client retries frequently. Is 125 enough to push over a "slow" threshold on the others? This will further magnify the load. The load balancer will spin up more machines, so now you have 10 machines leaning on whatever the back end is. Yes, your approach is great - but you really have to understand the failure modes - if you're living on the edge, you could have a pretty un fun cascading error.
    - sarnowski 12y ago
      Thats what circuit breakers are for to not have cascading errors.
- samspot 12y ago
  I don't find that performance runs at % chance scale. When things perform poorly, they are often very consistent. With this architecture you can rapidly iterate & scale your poor performing core services, while the non-core services can run slowly w/o causing a big deal. The usual alternative is redeploying and scaling the entire app to iterate on performance, which is a much slower process. If performance is a concern, microservices should be a big win.
exacube 12y ago
This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity? This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?
- MichaelGG 12y ago
  Exactly. Even in the late 90s "3 tier" development pushed the idea of having a separate layer and sticking to APIs. This is just saying that instead of making big layers, make smaller ones. Which isn't bad, it's just that most of the time you don't need this. Add a new service, like "GetRecommendationFromFacebook?" OK, go ahead and deploy, because you haven't changed the rest of the services in your layer, like "LoginUser". And there was a whole hubbub of discovery what with UDDI and DISCO and all that jazz I never really understood. It'd be nice if they gave some solid examples of the "micro" part to distinguish it from the general SOA idea.
- newobj 12y ago
  When it comes down to it, n-tier, SOA, and microservices are all expressions of the same basic ideas.
- brown9-2 12y ago
  Yes, a lot of organizations already design their backends in this way without necessarily thinking to use a brand new term to describe it. Cockcroft defines a microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts. To me this just reads as "service-oriented architecture in a sensible way". No one thinks to intentionally build a SOA with tightly coupled components with poor boundaries.
nawitus 12y ago
A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which would mean that updating one microservice would break another. Perhaps that is acceptable before the first "stable release" of a microservice.
- bigdubs 12y ago
  Honestly, I've been following along with these trends a lot and it seems like the answers for 1&2 come down to team size. If you have a bigger team, with more development inertia microservices can seem amazing and the tradeoffs are worth it (repeated work vs. development tempo). To a small team that doesn't have the inertial issues to generate the benefits of microservices, it seems like they are nice in theory but have too much overhead to supplant monolithic approaches.
- jacques_chester 12y ago
  > How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.
  - HeyLaughingBoy 12y ago
    I really wish the team I'm on could understand this!
  - mercurial 12y ago
    Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.
pbrettb 12y ago
building and rebuilding and maintaining applications is hard enough. Now we have whitepapers from a company that wants to sell us on a new paradigm which -- cough cough -- they just happen to have software to support.
polynomeal 12y ago
My team recently added a microservice to support our fairly monolythic backend service. The big challenge we found was that it takes a lot of effort to make a new (micro)service. We need to create a system of alarms (instead of relying on existing catch all defaults). We need its own test environment, we need to find ways to send traffic to pre-prod. We needed to figure out how to bootstrap the new service into the companies infrastructure. We needed to think about how the dependent service would authenticate against the new service. All of that on top of the core feature work. All these things are good. You want isolated, focused test environments. You want tightly defined alarms. However we underestimated how long creating a new service would take. In the end we ended up pushing features out when they were ready but before the operational work was complete. Unsurprisingly we saw the issues we knew we wanted to protect against. Better microservice franeworks that match the companies infrastructure would be helpful. Make building microservices cheap by building tools to speed up the process.
- cam- 12y ago
  Front End teams tend not like micro services, there is too much overhead in getting too little data. As an example we integrate with one micro service where get back a boolean and and a date. We have the overhead of an http call and all the error handling that goes with it for two pieces of data which would be better aggregated into another service. We story point an integration with a new service as an 8, but adding a new field (or two) in an existing API data structure is a 1. I hope micro services is not just a new fashion in software and is actually useful ten years from now.
  - doktorn 12y ago
    It is by no means a new fashion. In an interview from 2006, Werner Vogels (CTO & VP of Amazon) talks about it. http://queue.acm.org/detail.cfm?id=1142065 http://queue.acm.org/detail.cfm?id=1142065
- alxndr 12y ago
  I've had the same experience lately. Coordinating a number of staging environments for an evolving SOA backend has been challenging for both developers and ops. In addition to the services, each one can have many other things to worry about: monitoring, error collection, which version is deployed to which environment, replicating the complexity locally to work on it...

207 points by davidkellis 12y ago

My comment, slightly edited, from the previous posting of this, at https://news.ycombinator.com/item?id=9106813 https://news.ycombinator.com/item?id=9106813 > One kind of coupling that people tend to overlook as they transition to a microservices architecture is database coupling, where all services talk to the same database and updating a service means changing the schema. You need to split the database up and denormalize it. That sounds like a decision you wouldn't want to take lightly; the kind of thing you might do once your company is already big. I wouldn't want to start out that way though, it sounds like a recipe for a mess.

anonymousDan 12y ago
Yes I just came in here to write a comment along those lines. I mean surely you're opening a whole can of worms in terms of consistency etc. I get the feeling Netflix happens not to have use cases where strong consistency is a requirement. I'd be interested to get more detail about how they went about the transition - even just pointers to more on the meta-data management tools they use.
- davidkellis 12y ago
  The consistency problem is an open question in my mind. I definitely don't like the idea of having some data synchronization tool to fix the inconsistent data across services problem. I wonder what the best practice is for maintaining data consistency across services. Does anyone know?
  - shanemhansen 12y ago
    Ideally you don't have to sync the data because one service owns that data. Other services request that data via api. In a RESTful world those api requests are cacheable.
    - davidkellis 12y ago
      But what about the situation where you have an entity service that owns the data for one piece of the domain, for example a People service, and then other services, like the Address service and the Billing service, reference a particular person. In that scenario, I can imagine the Address service and the Billing service would have a foreign key referencing a person in the People service. Then, what happens if the Person gets deleted? In that case, we've got a consistency problem, even though each service owned its data. Is the best practice to not use entity services?
      - cgh 12y ago
        The "each service owns its data" scenario shouldn't take precedence over cases like this, where consistency is aligned with obvious business rules. If Person gets deleted, then "on delete cascade" should take care of that Person's Address and Billing records. For updates and maybe reads, it's a different story.
      - davidkellis 12y ago
        cgh, I've hit the reply limit, but I wanted to ask how you'd implement the cascade deletes thing over services? Would the People service have to emit events describing that a person was deleted that the Address and Billing services would be expected to subscribe to in order to handle that the person was deleted?
        cgh 12y ago
        Sorry, I should have been more clear. I'm assuming a shared database. So cascading deletes would be defined in the table's schema. Let's pretend we're using Postgresql: \d Person [A bunch of table schema stuff] Referenced by: TABLE "Billing" CONSTRAINT "billing_id_fk" FOREIGN KEY (id) REFERENCES Person(id) ON DELETE CASCADE (I typed that off the type of my head so it might not be quite correct.)
        dsp1234 12y ago
        The original position being argued though was "... where all services talk to the same database... You need to split the database up and denormalize it.". So the basic premise is that there is no shared database, and thus having the database enforce cascading deletes is not an option.
BerislavLopac 12y ago
If the two (the database schema and the microservices API) are designed and maintained separately, this assertion (that "updating a service means changing the schema") is not necessarily true. They have separate responsibilities -- the database is persisting the data, and the services provide some business logic, and while they might need to change together this is not always the case.
nl 12y ago
Database-as-an-integration-layer is a well known anti-pattern. It's tempting because it's easy, and at first glance it seems to solve lots of problems (consistency, communication etc). It's a big mistake. If multiple service are reading and writing from the same DB it rapidly becomes impossible to change things. Things like input validation changes suddenly have to be implemented in multiple places (which is hard), and done simultaneously (which often becomes hard enough to stop work being done). In a microservice based approach, the data flows though shared services, and so changes can be done in fewer places. (Note that read-only, reporting-style databases are separate. I think there is a good case for these being shared)
- dragonwriter 12y ago
  Multiple services reading from the same DB can -- and should -- be using different accounts that have access to different objects that abstract the underlying data model from the various clients and keep them loosely coupled. This is almost as old of a recommendation as relational DBs themselves. You can create a tightly coupled design with a shared DB, but there is nothing inherent with shared DB integration requires that.

jfoutz 12y ago

If your app depends on lots of of them, it's only going to run as fast as the slowest dependency. 1% chance of poor performance isn't to bad, the joint distribution of 20 microservices each with a 1% chance, well that gets pretty ugly. In the normal case, everything is great, but the failure modes of each service become a much bigger deal. It's a great architecture, but fan out of dependencies is a real risk.

deleted 12y ago
[deleted]
ryderm 12y ago
So you have short timeouts and retries that are load balanced to different nodes. But ideally your services are fast even in their 99 percentiles so this isn't an issue. This is much easier to achieve in a small service than a huge complex one.
- jfoutz 12y ago
  Um. maybe. 5 machines behind a load balancer. Normal case, load is even, 100 requests to each server. One server starts running into trouble, exceeding timeouts. your load is now ~125 per server, because each client retries frequently. Is 125 enough to push over a "slow" threshold on the others? This will further magnify the load. The load balancer will spin up more machines, so now you have 10 machines leaning on whatever the back end is. Yes, your approach is great - but you really have to understand the failure modes - if you're living on the edge, you could have a pretty un fun cascading error.
  - sarnowski 12y ago
    Thats what circuit breakers are for to not have cascading errors.
samspot 12y ago
I don't find that performance runs at % chance scale. When things perform poorly, they are often very consistent. With this architecture you can rapidly iterate & scale your poor performing core services, while the non-core services can run slowly w/o causing a big deal. The usual alternative is redeploying and scaling the entire app to iterate on performance, which is a much slower process. If performance is a concern, microservices should be a big win.

exacube 12y ago

This seems like it's just taking the idea of decoupling your service and talking to them via APIs a little further by saying.. decouple them to an even smaller granularity? This has always been the generally accepted way to scale out software services. Is there a novel idea being discussed here, or just that they've been doing this at Netflix?

MichaelGG 12y ago
Exactly. Even in the late 90s "3 tier" development pushed the idea of having a separate layer and sticking to APIs. This is just saying that instead of making big layers, make smaller ones. Which isn't bad, it's just that most of the time you don't need this. Add a new service, like "GetRecommendationFromFacebook?" OK, go ahead and deploy, because you haven't changed the rest of the services in your layer, like "LoginUser". And there was a whole hubbub of discovery what with UDDI and DISCO and all that jazz I never really understood. It'd be nice if they gave some solid examples of the "micro" part to distinguish it from the general SOA idea.
newobj 12y ago
When it comes down to it, n-tier, SOA, and microservices are all expressions of the same basic ideas.
brown9-2 12y ago
Yes, a lot of organizations already design their backends in this way without necessarily thinking to use a brand new term to describe it. Cockcroft defines a microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts. To me this just reads as "service-oriented architecture in a sensible way". No one thinks to intentionally build a SOA with tightly coupled components with poor boundaries.

nawitus 12y ago

A few questions: a) How do you prevent technical debt? It seems to be more difficult due to APIs which shouldn't have breaking changes. In theory you could always version up the APIs and serve both versions or just add a new API for a breaking change, but these solutions seems awkward. b) How do you start developing multiple microservices at the same time? I would expect APIs to change a lot in the beginning, which would mean that updating one microservice would break another. Perhaps that is acceptable before the first "stable release" of a microservice.

bigdubs 12y ago
Honestly, I've been following along with these trends a lot and it seems like the answers for 1&2 come down to team size. If you have a bigger team, with more development inertia microservices can seem amazing and the tradeoffs are worth it (repeated work vs. development tempo). To a small team that doesn't have the inertial issues to generate the benefits of microservices, it seems like they are nice in theory but have too much overhead to supplant monolithic approaches.
jacques_chester 12y ago
> How do you start developing multiple microservices at the same time? Same as any other project: Develop from the outside in. In practice, trying to develop in the "optimal order' leads to speculative development that will be wasted.
- HeyLaughingBoy 12y ago
  I really wish the team I'm on could understand this!
- mercurial 12y ago
  Nothing prevents you from having multiple microservices sharing the same codebase. That said, the "it's just like the web" model doesn't sound fantastic to me. It sounds like your app now depends on contracts which are only enforced by good practices, not by something strongly typed you can check at compile time, unless you use something like protocol buffers to generate the boilerplate.

pbrettb 12y ago

building and rebuilding and maintaining applications is hard enough. Now we have whitepapers from a company that wants to sell us on a new paradigm which -- cough cough -- they just happen to have software to support.

polynomeal 12y ago

My team recently added a microservice to support our fairly monolythic backend service. The big challenge we found was that it takes a lot of effort to make a new (micro)service. We need to create a system of alarms (instead of relying on existing catch all defaults). We need its own test environment, we need to find ways to send traffic to pre-prod. We needed to figure out how to bootstrap the new service into the companies infrastructure. We needed to think about how the dependent service would authenticate against the new service. All of that on top of the core feature work. All these things are good. You want isolated, focused test environments. You want tightly defined alarms. However we underestimated how long creating a new service would take. In the end we ended up pushing features out when they were ready but before the operational work was complete. Unsurprisingly we saw the issues we knew we wanted to protect against. Better microservice franeworks that match the companies infrastructure would be helpful. Make building microservices cheap by building tools to speed up the process.

cam- 12y ago
Front End teams tend not like micro services, there is too much overhead in getting too little data. As an example we integrate with one micro service where get back a boolean and and a date. We have the overhead of an http call and all the error handling that goes with it for two pieces of data which would be better aggregated into another service. We story point an integration with a new service as an 8, but adding a new field (or two) in an existing API data structure is a 1. I hope micro services is not just a new fashion in software and is actually useful ten years from now.
- doktorn 12y ago
  It is by no means a new fashion. In an interview from 2006, Werner Vogels (CTO & VP of Amazon) talks about it. http://queue.acm.org/detail.cfm?id=1142065 http://queue.acm.org/detail.cfm?id=1142065
alxndr 12y ago
I've had the same experience lately. Coordinating a number of staging environments for an evolving SOA backend has been challenging for both developers and ops. In addition to the services, each one can have many other things to worry about: monitoring, error collection, which version is deployed to which environment, replicating the complexity locally to work on it...