4 ms·
I joined Uber in 2016, right around when on every conference you'd hear a talk along the lines on "Lessons learned at Uber on scaling to thousands of microservi
by gregdoesit 7y ago
I joined Uber in 2016, right around when on every conference you'd hear a talk along the lines on "Lessons learned at Uber on scaling to thousands of microservices" [1].
After a year or two, those talks stopped. Why? Turns out, having thousands of microservices is something to flex about, and make good conference talks. But the cons start to weigh after a while - and when addressing those cons, you take a step back towards fewer, and bigger services. I predict Monzo will see the same cons in a year or two, and move to a more pragmatic, fewer, better-sized services approach that I've seen at Uber.
In 2020, Uber probably has fewer microservices than in 2015. Microservices are fantastic for autonomy. However, that autonomy also comes with the drawbacks. Integration testing becomes hard. The root cause of most outages become parallel deployments of two services, that cause issues. Ownership becomes problematic, when a person leaves who owned a microservice that was on the critical path. And that no one else knew about. Oncall load becomes tough: you literally have people own 4-5 microservices that they launched. Small ones, sure, but when they go down, they still cause issues.
To make many services work at scale, you need to solve all of these problems. You need to introduce tiering: ensuring the most ciritical (micro)services have the right amount of monitoring, alerting, proper oncall and strong ownership. Integration testing needs to be solved for critical services - often meaning merging multiple smaller services that relate to each other. Services need to have oncall owners: and a healthy oncall usually needs at least 5-6 engineers in a rotation, making the case for larger services as well.
Microservices are a great way to give more autonomy for teams. Just beware of the autonomy turning into something that's hard to manage, or something that burns people out. Uber figured this out - other companies are bound to follow.
[1] http://highscalability.com/blog/2016/10/12/lessons-learned-from-scaling-uber-to-2000-engineers-1000-ser.html http://highscalability.com/blog/2016/10/12/lessons-learned-f...
- imtringued 7y ago>And that no one else knew about. Yeah exactly. Having two people dedicated to the "Account" service that has 10 features sound much better than having 2 people responsible for 5 microservices each. You might end up with a reset credentials service, a register customer service, a delete account service without any coherent overarching design strategy instead of having just having a plain CRUD service with 6 extra operations. I can't blame them for having 160 services because a lot of enterprise tier organizations are truly that complicated (I work at one) but I do blame them for having 1600.
- lykr0n 7y ago> And that no one else knew about. Yep. I've encountered a production issue that was traced down to a Location service- you pass in some info, and get location information back- that had been running for 3 years without an owner. The developers had all left, and the team was dissolved. Not an inherent fault of microservices, but having 1000s of them running around will cause some to slip through the cracks.