17 ms·
Does Uber really need 4000 microservices?
by x86x87 3y ago
Does Uber really need 4000 microservices?
- tight-ship 3y agoDoes it matter how they organize their services? Your experience and environment will be different in so many ways that I doubt it's comparable.
- nine_zeros 3y agoHow else would engineers demonstrate "impact" for promotions? /s
- zrail 3y agoYou jest but also you're not wrong.
- nine_zeros 3y agoMy own company has 800+ microservices. I am very familiar with the politics of microservices.
- 0xblinq 3y ago[flagged]
- thakoppno 3y agoMy personal microservice fiefdom is about ten. My company probably has 1000. Is this ratio normal?
- nine_zeros 3y agoAny more than 1 microservice per engineer (as in, 7 microservices for a 7 engineer team) is too much for engineers to handle during on-call incidents. If management values business SLAs that is.
- mooreds 3y agoI worked at a SV startup (series A) for a while and an EM once mentioned struggling to keep the number of microservices under the number of engineers.
- aitchnyu 3y agoIs there a compelling article about the ideal microservice to engineer ratio (ie less than 1.0)?
- mooreds 3y agoI don't know of one, but this thread has some interesting discussion: https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_company_has_2_microservices_per_engineer/ https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_...
- wiseowise 3y agoDysfunctional organizations lead to absurd solutions to absurd problems.
- ninja3925 3y ago(Worked at Lyft) Our number of active micro services was small in comparison. 4,000 is likely a overblown number to highlight the accomplishment possibly counting inactive ones
- 0xblinq 3y ago[flagged]
- drexlspivey 3y agoIsn’t Lyft US only? Uber operates in 80+ countries
- zht 3y agoLyft also doesn’t do food delivery…
- latchkey 3y agoWorked at Grab. They had a ton of micro services. It was their way of partitioning databases so that they didn't have to deal with joins. Yes, it caused a lot of problems.
- danpalmer 3y agoA different (better?) question is, does Uber need 4000 API contracts? The answer to that is probably yes. APIs let us split work across systems/people/teams/regions, and provide a way for both sides of a split to work together. Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently. Sometimes those APIs make sense to package as microservices.
- davidw 3y agoI wonder what setting up a local dev instance is like for anything involving more than one or two of those.
- threeseed 3y agoI've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particular scenario to test.
- plonk 3y agoIs there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.
- nurple 3y agoI've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions. I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.
- belter 3y agoApparently they started at 1000 and went from there... "What I Wish I Had Known Before Scaling Uber to 1000 Services" - https://youtu.be/kb-m2fasdDY https://youtu.be/kb-m2fasdDY
- barbazoo 3y agoMaybe they meant instances.
- 0xblinq 3y agoWhat would be the engineers doing otherwise? You get bored if you don’t.
- dkarl 3y agoIt's insane to hear attrition used as an excuse for architectural decisions, but I've seen it firsthand.
- x86x87 3y agoSo we're going to get to the point where everyone gets their own microsevice, right?
- BigJono 3y agoThat is unironically a better way to do it than anywhere I've worked that does microservices. My experience with microservice shops is you have one macromonolith with 50 people working on it (which has all the problems of a monolith and none of the benefits), 5 actual decent microservices with a team or individual that properly maintains them, and 100 random utility micro"services" that are like 3 lines of code, used by exactly one other service, and you need 40 loc and a network call to interact with them. I'll take everyone has their own service any day of the week. At least when I need to interface with 12 different things I can have 12 different people to roast for not properly documenting their API. And tbh literally the only positive I can come up with for microservices is the ability to neatly fire one into the sun and rewrite it from scratch.
- scarface_74 3y agoIsn’t that basically what happens when you split an API out to the different methods and write a Lambda for each one?
- whynotmaybe 3y agoThere's quite a sizing range between monolith and microservice. If all their It needs are behind micro "micro" services, that figure is understandable. Outside of the map, taxi, food, payments, onboarding, they also have monitoring, deployment, HR, billing, legal, taxes, internationalized stufd, and the usual "..." for what I'm missing. If you just take a standard ERP, you could easily split it in dozens even hundreds of microservices.
- bcrosby95 3y ago> If all their It needs are behind micro "micro" services, that figure is understandable. I call them nano services.
- whynotmaybe 3y agoAnd you're pinpointing to the problem with such news. Everybody knows what is a monolith but nobody really knows what is the size of a "micro" service. Just for taxes, do you make one service for taxes or one for each recipient of taxes? (In the EU, is it one for each country, in US, one for each state + federal ) with a different team managing each service?
- mirekrusin 3y agoBut how does your "..." sum up to 4000?
- threeseed 3y agoUber is a global company (70+ countries) operating Uber and Uber Eats. So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.
- id00 3y agoNo, that's not correct (mostly). Services are written in the way to support global operations. But that scale introduces a lot of complexity so you can't just have "one service for onboarding drivers"
- whatwhaaaaat 3y agoDo you have knowledge of this? I find this hard to believe given the regulations from some of the larger countries requiring, by law, customer data be processed in country.
- joneholland 3y agoDeployments != unique codebases.
- whatwhaaaaat 3y agoTry but read the full comment chain again to understand the context of our discussion.
- joneholland 3y agoI did. Your concern about data needing to be isolated due to regulation doesn’t require you to make a complete copy of the code.
- whatwhaaaaat 3y ago>“So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.” Responding comment says no they are not and the services are built to handle global traffic. I respond and say I doubt that due to on soil laws. You can argue two regions with different configurations but the same code bases are different services but that’s not what we’re talking about here.
- talent_deprived 3y agoI've seen at least one place with many more than that in recent years. If you have one microservice "listener" per queue and another for the database processing and persistence (business logic) and another providing an API for one or more frontend UI's related to it then the microservice tally goes up very fast. It's kind of surprising to read so many comments indicating HN readers weren't aware of this.
- herval 3y agosounds like a massive nightmare
- donutshop 3y agoMassive scale*
- isbvhodnvemrwvn 3y agoMost are likely limited to some subdomain, with limited communication between domains.
- speedgoose 3y agoIt reminds me this thread about Netflix, with insane amounts of events and logs compared to active users. https://news.ycombinator.com/item?id=30635369 https://news.ycombinator.com/item?id=30635369
- ilrwbwrkhv 3y agoYup Pornhub serves much more video than Netflix and they do so without that insane amount of complexity.
- mattsan 3y agoHonestly Pornhub's stack is genuinely impressive. More start-ups should just use PHP and get shit done
- Nextgrid 3y agoI think one of the reasons behind PornHub's tech stack is that their industry doesn't really lend itself to VC, so building an engineering playground for PH would be a waste of money as no amount of complexity would net them VC money (nor an invite to a cloud provider conference), where as most startups live and die based on the VC funding their complexity and buzzwords allow them to grift.
- plaguuuuuu 3y agoI think the takeaway was that choice of language might be less important than engineering strategy, because PH are successful despite their choice of stack :)
- x86x87 3y agoPornhub is not about waving the engineering flag up and down to signal how cool their infra is though.
- Roark66 3y agoIsn't pornhub free, their only monetization being ads? Also does Pornhub have personal recommendations, per country and region libraries, an android, ios and android TV apps? Probably not. That is a much easier business model and a lot lower level of complexity than Netflix. I imagine running pornhub is essentially running a large website that hosts video. Probably just the billing side of Netflix is more complicated than the entirety of Pornhub operation.
- 0xDEF 3y agoYes, there are specific business rules for each nation, region/state, and city.
- bastawhiz 3y agoUber has a really liberal definition of a micro service. Every web UI or dashboard is a service (of which there are many hundreds). Every application anyone builds across their many thousands of engineers is a service. It's rare, I think, for services to have fewer then a few thousand lines of code. In my experience, most companies would have a monolith that serves multiple UIs from the same service. Uber instead ships that monolith as a library which is a framework for building individual UIs. It has its pros and cons but I quite liked how they did it.
- justapassenger 3y agoFrom experience working at big tech I’m willing to take a guess. Maybe a couple of dozens will be actual more complex and meaningful services. Then few dozens more services that are somewhat more unique. And then majority of the long tail will be mostly cookie cutter services, doing X, but for lots of different use cases, where each of use cases is separate deployment counting as a service (for example - systems to process streams of logs related to business logic).
- lokar 3y agoThe same binary with a different configuration
- jpalomaki 3y agoThere's an an interesting HN comment[1] from 2020 by former Uber engineer, which discusses the complexity a bit. It's more about UI, but the thread discusses the backend as well. In brief something that may look super simple for the user (like handling payments) is actually quite complicated when you cover all the market, different payment types etc. And all this carries to the backend as well. [1] https://news.ycombinator.com/item?id=25376346 https://news.ycombinator.com/item?id=25376346
- lamontcg 3y agoDo all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?
- scarface_74 3y agoAnd then every time you had a change, you would have to deploy everything and your surface of failure is greater. What would a monolith buy you?
- jgraettinger1 3y agoNot having to evolve or understand or staff 4,000 micro services. An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.
- scarface_74 3y agoYou still have to understand your boundaries when you have a large monolith unless you have one big ball of mud. Even with a well constructed monolith, you need to have well defined “services” with contractual interfaces. You don’t have to understand 4000 services to make one change anymore than I need to understand the entire boto3 library when I am building on top of it. https://boto3.amazonaws.com/v1/documentation/api/latest/index.html https://boto3.amazonaws.com/v1/documentation/api/latest/inde... You can’t just change your interface in a monolith either without breaking other parts of it.
- deathanatos 3y agoThere's no way that number isn't fiction; Occam's razor say's its out of the range of believable. That's ~2 per eng according to Google. That's absurd. (That eng headcount is also a bit … high.) This sounds like a figure from someone who sees a signle microservice running across 100 pods/instances, and counted that as 100 "microservices".
- scarface_74 3y agoS3 alone is built on top of 300 micro services. I don’t find it unbelievable that Uber needs a lot of them.
- zo1 3y agoI find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?
- scarface_74 3y agoA random engineer at any company is not going to divulged non public information about the inner working of their company without permission. We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.
- x86x87 3y agoyou see someone telling you how many microservices s3 has right above.
- scarface_74 3y agoYes that “someone” was me - a former employee who couldn’t remember whether that information was public or whether it was something I was exposed to from the inside. It in fact was public. https://aws.amazon.com/blogs/storage/how-automated-reasoning-helps-us-innovate-at-s3-scale/#:~:text=S3%20is%20built%20from%20over,form%20a%20distributed%20storage%20system https://aws.amazon.com/blogs/storage/how-automated-reasoning.... I verified that before I posted that little tidbit.