5 ms·

That's a lot of microservices. How do you handle changing the interface of a service in a backward incompatible way? Or updating a dependency across all servi

That's a lot of microservices. How do you handle changing the interface of a service in a backward incompatible way? Or updating a dependency across all services?

chuhnk 11y ago
We use protobuf rpc which allows us to maintain some form of backwards compatibility. New fields are ignored by old version of a service. We can also run multiple versions of a service in parallel and send a percentage of traffic to each. Also have the ability to do label based routing so new features are only enabled where a v1.1 header or something of that nature is seen. Updating dependencies across all services is somewhat of an anti pattern in microservices, this chain of dependencies shouldn't exist. Microservices are loosely coupled with a bounded context. In the case of bug fixes based on core libraries we have roll out procedures to update services over time but in a microservice world there's never a point where you look to update a block of services all at once. You should be able to deploy new versions of a service without breaking the system. Tradeoffs are made for a microservice architecture patterns but when organisations scale to 100+ people we deem these tradeoffs necessary as what we lose in convenience in monolithic tradition, we gain in speed of execution and a highly availably fault tolerant globally scaling system. You can learn more about our journey on our blog https://sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-1/ https://sudo.hailoapp.com/services/2015/03/09/journey-into-a... And slides from former platform tech lead Matt Heath https://speakerdeck.com/mattheath https://speakerdeck.com/mattheath Also from current platform automation lead Boyan Dimitrov http://www.slideshare.net/nathariel http://www.slideshare.net/nathariel Various talks can also be found on youtube.
- nulltype 11y ago
  Thanks for all the info! Very informative. "Updating dependencies across all services is somewhat of an anti pattern in microservices, this chain of dependencies shouldn't exist." But what about, say, your tracing library. It seems like that would be shared across all your services, right?
  - chuhnk 11y ago
    Very early on we had a lot of critical updates in core libraries which meant rebuilding everything and releasing but at the time the number of services were quite low and we were still in the R&D phase. Overtime that maturity meant changes going out were mostly feature based and we could afford to lag those things out. You're right though, certain things require rebuilding everything but we try to approach that pragmatically. On the platform team we'd have ownership of say 20-30 services so over a period of a week we can push those through a staging and load test environment then to production. We have 4 or 5 other teams that do the same. There are tools to ensure we can keep track of which libraries are in production and if anything is out dated. Like I mentioned before. There are definitely tradeoffs to microservices and they don't make sense for every use case but if you look at the companies that adopted this architecture pattern you'll see the common journey they all went on. Monolithic architecture for the first few years, scaled by brute force, money and people. Eventually stalling in development and organisational speed of execution. Taking a step back to reevaluate and then determining a migration path to a new decomposed service oriented architecture.

That's a lot of microservices. How do you handle changing the interface of a service in a backward incompatible way? Or updating a dependency across all services?

chuhnk 11y ago

We use protobuf rpc which allows us to maintain some form of backwards compatibility. New fields are ignored by old version of a service. We can also run multiple versions of a service in parallel and send a percentage of traffic to each. Also have the ability to do label based routing so new features are only enabled where a v1.1 header or something of that nature is seen. Updating dependencies across all services is somewhat of an anti pattern in microservices, this chain of dependencies shouldn't exist. Microservices are loosely coupled with a bounded context. In the case of bug fixes based on core libraries we have roll out procedures to update services over time but in a microservice world there's never a point where you look to update a block of services all at once. You should be able to deploy new versions of a service without breaking the system. Tradeoffs are made for a microservice architecture patterns but when organisations scale to 100+ people we deem these tradeoffs necessary as what we lose in convenience in monolithic tradition, we gain in speed of execution and a highly availably fault tolerant globally scaling system. You can learn more about our journey on our blog https://sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-1/ https://sudo.hailoapp.com/services/2015/03/09/journey-into-a... And slides from former platform tech lead Matt Heath https://speakerdeck.com/mattheath https://speakerdeck.com/mattheath Also from current platform automation lead Boyan Dimitrov http://www.slideshare.net/nathariel http://www.slideshare.net/nathariel Various talks can also be found on youtube.

nulltype 11y ago
Thanks for all the info! Very informative. "Updating dependencies across all services is somewhat of an anti pattern in microservices, this chain of dependencies shouldn't exist." But what about, say, your tracing library. It seems like that would be shared across all your services, right?
- chuhnk 11y ago
  Very early on we had a lot of critical updates in core libraries which meant rebuilding everything and releasing but at the time the number of services were quite low and we were still in the R&D phase. Overtime that maturity meant changes going out were mostly feature based and we could afford to lag those things out. You're right though, certain things require rebuilding everything but we try to approach that pragmatically. On the platform team we'd have ownership of say 20-30 services so over a period of a week we can push those through a staging and load test environment then to production. We have 4 or 5 other teams that do the same. There are tools to ensure we can keep track of which libraries are in production and if anything is out dated. Like I mentioned before. There are definitely tradeoffs to microservices and they don't make sense for every use case but if you look at the companies that adopted this architecture pattern you'll see the common journey they all went on. Monolithic architecture for the first few years, scaled by brute force, money and people. Eventually stalling in development and organisational speed of execution. Taking a step back to reevaluate and then determining a migration path to a new decomposed service oriented architecture.