10 ms·
Uber migrates microservices to multi-cloud platform running Kubernetes and Mesos
- x86x87 3y agoDoes Uber really need 4000 microservices?
- tight-ship 3y agoDoes it matter how they organize their services? Your experience and environment will be different in so many ways that I doubt it's comparable.
- nine_zeros 3y agoHow else would engineers demonstrate "impact" for promotions? /s
- zrail 3y agoYou jest but also you're not wrong.
- nine_zeros 3y agoMy own company has 800+ microservices. I am very familiar with the politics of microservices.
- 0xblinq 3y ago[flagged]
- thakoppno 3y agoMy personal microservice fiefdom is about ten. My company probably has 1000. Is this ratio normal?
- nine_zeros 3y agoAny more than 1 microservice per engineer (as in, 7 microservices for a 7 engineer team) is too much for engineers to handle during on-call incidents. If management values business SLAs that is.
- mooreds 3y agoI worked at a SV startup (series A) for a while and an EM once mentioned struggling to keep the number of microservices under the number of engineers.
- aitchnyu 3y agoIs there a compelling article about the ideal microservice to engineer ratio (ie less than 1.0)?
- mooreds 3y agoI don't know of one, but this thread has some interesting discussion: https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_company_has_2_microservices_per_engineer/ https://www.reddit.com/r/ExperiencedDevs/comments/x1p5gj/my_...
- wiseowise 3y agoDysfunctional organizations lead to absurd solutions to absurd problems.
- ninja3925 3y ago(Worked at Lyft) Our number of active micro services was small in comparison. 4,000 is likely a overblown number to highlight the accomplishment possibly counting inactive ones
- 0xblinq 3y ago[flagged]
- drexlspivey 3y agoIsn’t Lyft US only? Uber operates in 80+ countries
- zht 3y agoLyft also doesn’t do food delivery…
- latchkey 3y agoWorked at Grab. They had a ton of micro services. It was their way of partitioning databases so that they didn't have to deal with joins. Yes, it caused a lot of problems.
- danpalmer 3y agoA different (better?) question is, does Uber need 4000 API contracts? The answer to that is probably yes. APIs let us split work across systems/people/teams/regions, and provide a way for both sides of a split to work together. Uber has a lot of teams, a lot of engineers, and so it makes sense that there are a lot of API boundaries to allow them to work together more efficiently. Sometimes those APIs make sense to package as microservices.
- davidw 3y agoI wonder what setting up a local dev instance is like for anything involving more than one or two of those.
- threeseed 3y agoI've worked on a couple of extremely large micro services projects. And the thing is that nobody ever needs to run the entire stack other than end to end tests which get run in the cloud. You just checkout the services you need and because they are designed to be isolated the dependencies will usually be automatically stubbed out. So it's just a matter of running them or chaining them together if you have a particular scenario to test.
- plonk 3y agoIs there typically a “cold restart” plan to rebuild the whole infra from scratch? I’m thinking of things like circular dependencies when services boot.
- nurple 3y agoI've found that the eventual consistency provided by orchestrators like k8s solve this problem rather well. If the services are written using the right paradigms to handle such a situation, they will also be much more resilient to platform disruptions. I like to view k8s a lot like erlang's OTP, if something isn't right with the state of a service, I advocate calling 'exit()' and letting the restart with exponential backoff handle the transient.
- belter 3y agoApparently they started at 1000 and went from there... "What I Wish I Had Known Before Scaling Uber to 1000 Services" - https://youtu.be/kb-m2fasdDY https://youtu.be/kb-m2fasdDY
- barbazoo 3y agoMaybe they meant instances.
- 0xblinq 3y agoWhat would be the engineers doing otherwise? You get bored if you don’t.
- dkarl 3y agoIt's insane to hear attrition used as an excuse for architectural decisions, but I've seen it firsthand.
- x86x87 3y agoSo we're going to get to the point where everyone gets their own microsevice, right?
- BigJono 3y agoThat is unironically a better way to do it than anywhere I've worked that does microservices. My experience with microservice shops is you have one macromonolith with 50 people working on it (which has all the problems of a monolith and none of the benefits), 5 actual decent microservices with a team or individual that properly maintains them, and 100 random utility micro"services" that are like 3 lines of code, used by exactly one other service, and you need 40 loc and a network call to interact with them. I'll take everyone has their own service any day of the week. At least when I need to interface with 12 different things I can have 12 different people to roast for not properly documenting their API. And tbh literally the only positive I can come up with for microservices is the ability to neatly fire one into the sun and rewrite it from scratch.
- scarface_74 3y agoIsn’t that basically what happens when you split an API out to the different methods and write a Lambda for each one?
- whynotmaybe 3y agoThere's quite a sizing range between monolith and microservice. If all their It needs are behind micro "micro" services, that figure is understandable. Outside of the map, taxi, food, payments, onboarding, they also have monitoring, deployment, HR, billing, legal, taxes, internationalized stufd, and the usual "..." for what I'm missing. If you just take a standard ERP, you could easily split it in dozens even hundreds of microservices.
- bcrosby95 3y ago> If all their It needs are behind micro "micro" services, that figure is understandable. I call them nano services.
- whynotmaybe 3y agoAnd you're pinpointing to the problem with such news. Everybody knows what is a monolith but nobody really knows what is the size of a "micro" service. Just for taxes, do you make one service for taxes or one for each recipient of taxes? (In the EU, is it one for each country, in US, one for each state + federal ) with a different team managing each service?
- mirekrusin 3y agoBut how does your "..." sum up to 4000?
- threeseed 3y agoUber is a global company (70+ countries) operating Uber and Uber Eats. So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.
- id00 3y agoNo, that's not correct (mostly). Services are written in the way to support global operations. But that scale introduces a lot of complexity so you can't just have "one service for onboarding drivers"
- whatwhaaaaat 3y agoDo you have knowledge of this? I find this hard to believe given the regulations from some of the larger countries requiring, by law, customer data be processed in country.
- joneholland 3y agoDeployments != unique codebases.
- whatwhaaaaat 3y agoTry but read the full comment chain again to understand the context of our discussion.
- joneholland 3y agoI did. Your concern about data needing to be isolated due to regulation doesn’t require you to make a complete copy of the code.
- whatwhaaaaat 3y ago>“So almost certainly they are duplicating their entire stack per-country if only to get around the vastly different regulatory environments.” Responding comment says no they are not and the services are built to handle global traffic. I respond and say I doubt that due to on soil laws. You can argue two regions with different configurations but the same code bases are different services but that’s not what we’re talking about here.
- talent_deprived 3y agoI've seen at least one place with many more than that in recent years. If you have one microservice "listener" per queue and another for the database processing and persistence (business logic) and another providing an API for one or more frontend UI's related to it then the microservice tally goes up very fast. It's kind of surprising to read so many comments indicating HN readers weren't aware of this.
- herval 3y agosounds like a massive nightmare
- donutshop 3y agoMassive scale*
- isbvhodnvemrwvn 3y agoMost are likely limited to some subdomain, with limited communication between domains.
- speedgoose 3y agoIt reminds me this thread about Netflix, with insane amounts of events and logs compared to active users. https://news.ycombinator.com/item?id=30635369 https://news.ycombinator.com/item?id=30635369
- ilrwbwrkhv 3y agoYup Pornhub serves much more video than Netflix and they do so without that insane amount of complexity.
- mattsan 3y agoHonestly Pornhub's stack is genuinely impressive. More start-ups should just use PHP and get shit done
- Nextgrid 3y agoI think one of the reasons behind PornHub's tech stack is that their industry doesn't really lend itself to VC, so building an engineering playground for PH would be a waste of money as no amount of complexity would net them VC money (nor an invite to a cloud provider conference), where as most startups live and die based on the VC funding their complexity and buzzwords allow them to grift.
- plaguuuuuu 3y agoI think the takeaway was that choice of language might be less important than engineering strategy, because PH are successful despite their choice of stack :)
- x86x87 3y agoPornhub is not about waving the engineering flag up and down to signal how cool their infra is though.
- Roark66 3y agoIsn't pornhub free, their only monetization being ads? Also does Pornhub have personal recommendations, per country and region libraries, an android, ios and android TV apps? Probably not. That is a much easier business model and a lot lower level of complexity than Netflix. I imagine running pornhub is essentially running a large website that hosts video. Probably just the billing side of Netflix is more complicated than the entirety of Pornhub operation.
- 0xDEF 3y agoYes, there are specific business rules for each nation, region/state, and city.
- bastawhiz 3y agoUber has a really liberal definition of a micro service. Every web UI or dashboard is a service (of which there are many hundreds). Every application anyone builds across their many thousands of engineers is a service. It's rare, I think, for services to have fewer then a few thousand lines of code. In my experience, most companies would have a monolith that serves multiple UIs from the same service. Uber instead ships that monolith as a library which is a framework for building individual UIs. It has its pros and cons but I quite liked how they did it.
- justapassenger 3y agoFrom experience working at big tech I’m willing to take a guess. Maybe a couple of dozens will be actual more complex and meaningful services. Then few dozens more services that are somewhat more unique. And then majority of the long tail will be mostly cookie cutter services, doing X, but for lots of different use cases, where each of use cases is separate deployment counting as a service (for example - systems to process streams of logs related to business logic).
- lokar 3y agoThe same binary with a different configuration
- jpalomaki 3y agoThere's an an interesting HN comment[1] from 2020 by former Uber engineer, which discusses the complexity a bit. It's more about UI, but the thread discusses the backend as well. In brief something that may look super simple for the user (like handling payments) is actually quite complicated when you cover all the market, different payment types etc. And all this carries to the backend as well. [1] https://news.ycombinator.com/item?id=25376346 https://news.ycombinator.com/item?id=25376346
- lamontcg 3y agoDo all of those need to be microservices or you could you instead have one monolithic payment service that handled all those use cases?
- scarface_74 3y agoAnd then every time you had a change, you would have to deploy everything and your surface of failure is greater. What would a monolith buy you?
- jgraettinger1 3y agoNot having to evolve or understand or staff 4,000 micro services. An ability to easily change the boundaries of your conceptual components, because they WILL be wrong now or in the future.
- scarface_74 3y agoYou still have to understand your boundaries when you have a large monolith unless you have one big ball of mud. Even with a well constructed monolith, you need to have well defined “services” with contractual interfaces. You don’t have to understand 4000 services to make one change anymore than I need to understand the entire boto3 library when I am building on top of it. https://boto3.amazonaws.com/v1/documentation/api/latest/index.html https://boto3.amazonaws.com/v1/documentation/api/latest/inde... You can’t just change your interface in a monolith either without breaking other parts of it.
- deathanatos 3y agoThere's no way that number isn't fiction; Occam's razor say's its out of the range of believable. That's ~2 per eng according to Google. That's absurd. (That eng headcount is also a bit … high.) This sounds like a figure from someone who sees a signle microservice running across 100 pods/instances, and counted that as 100 "microservices".
- scarface_74 3y agoS3 alone is built on top of 300 micro services. I don’t find it unbelievable that Uber needs a lot of them.
- zo1 3y agoI find it highly disturbing that so far I've seen Zero Uber devs on this thread adding any sort of context/info or just confirmation. Wtf is this, the NSA? KGB? Can't they just list/dump the names of said 4000 services, or is that somehow some sort of secret-sauce?
- scarface_74 3y agoA random engineer at any company is not going to divulged non public information about the inner working of their company without permission. We had to sign something at AWS not to divulge internal tooling like what we used for our internal account factory that we used to create AWS accounts. Literally tens of thousands of people know what this tool is.
- x86x87 3y agoyou see someone telling you how many microservices s3 has right above.
- scarface_74 3y agoYes that “someone” was me - a former employee who couldn’t remember whether that information was public or whether it was something I was exposed to from the inside. It in fact was public. https://aws.amazon.com/blogs/storage/how-automated-reasoning-helps-us-innovate-at-s3-scale/#:~:text=S3%20is%20built%20from%20over,form%20a%20distributed%20storage%20system https://aws.amazon.com/blogs/storage/how-automated-reasoning.... I verified that before I posted that little tidbit.
- dzlobin 3y ago[flagged]
- eatonphil 3y agoThis article is a recap of the original engineering article by the quoted developer and manager at Uber. https://www.uber.com/en-GB/blog/up-portable-microservices-ready-for-the-cloud/ https://www.uber.com/en-GB/blog/up-portable-microservices-re...
- dang 3y agoOk, we've changed the URL to that from https://www.infoq.com/news/2023/10/uber-up-cloud-microservices/ https://www.infoq.com/news/2023/10/uber-up-cloud-microservic.... Thanks!
- deleted 3y ago[deleted]
- jbotdev 3y agoIt seems like they’ve gotten to the “holy grail” of deployment where developers don’t have to worry about infrastructure at all in theory. I’ve seen many teams go for simple/leaky abstractions on top Kubernetes to provide a similar solution, which is tempting because it’s easy and flexible. The problem is then all your devs need to be trained in all the complexities of Kubernetes deployments anyway. Hopefully Uber abstracted away Kubernetes and Mesos enough to be worthwhile, and they have a great infra team to support the devs.
- threeseed 3y agoKubernetes works great for larger projects when combined with ArgoCD or similar. They all use GitOps which means all infra deployments and changes are tracked and easily able to be rolled back on any issues. And the complexity is nothing compared to having to manage your own cloud resources using Terraform etc which used to be the case. And these days every developer needs to be on board with DevOps and so there are no real old-school infra teams supporting anyone.
- Nextgrid 3y agoThe other "leak" in these abstractions that arises from physical limits is performance, especially when it comes to IO. This is a major problem for databases and ultimately makes database "portability"/fault tolerance tricky since they work best with direct-attach storage that's inherently bound to a single physical machine.
- pm90 3y agoNot to mention there is all sorts of other limits that you can hit at scale just on the compute layer itself (eg max pids, file descriptors etc). I don’t know if we can truly abstract away the underlying system. The best we can do is give a best effort approximation that works in most cases, but explicitly call out the limits when they are reached. I suspect that this is just the bubbling up of the underlying physics limitation of having limited resources where compute is run.
- zeroCalories 3y agoIt's not clear to me that being completely unaware of your infrastructure is a good thing. I don't think it's too much trouble to ask an engineer to understand k8s and think about where their service will live, even if it's a ci system that actually deploys. Furthermore, many layers of abstraction, especially in-house abstraction, just mean you have more code to maintain, another system for people to learn, and existing knowledge that you can't leverage anymore.
- m3kw9 3y ago[flagged]
- mbrumlow 3y agoIn 3 years… “Uber saved cost by migrating their micro service to their own colo.” followed by “Uber simplified operations by migrating their micro service platform to a monolith”.
- EduardoBautista 3y ago[flagged]
- weakwire 3y agoYes, no one said they did that.
- EduardoBautista 3y ago[flagged]
- arthurcolle 3y agoThey were making a joke about a hypothetical post in another three years
- yardstick 3y agoThe “in 3 years” was meant as a forward looking statement. The post was being sarcastic that Uber will in 3+ years time claim another victory, based on abandoning the cloud.
- gremlinunderway 3y agoThanks, I also misunderstood "in 3 years" meant forward-looking (i.e. "3 years from now...") rather than backward-looking ("for over 3 years...")
- politelemon 3y agoIn 5 years... "We've discovered a new paradigm for efficiently carving up and distributing computational units for our application. We call it, nanofunctions."
- this_user 3y agoFor a company that is basically a taxi service, they seem to invest an awful lot in constant rebuilds of their extremely complex infrastructure, which raises the question of whether that is even remotely necessary or just an exercise in pretending that they are a tech company.
- gedy 3y ago> "Basically a taxi service" Not defending their tech stack, but I mean that is a lot of realtime data that needs to be accurate - this is not your typical SaaS crud app.
- zht 3y agoJust the taking payments and reconciliation in dozens of currencies and payment methods alone would be so kind numbing difficult and complex
- bogota 3y agoThis is just such a bad take that it makes everything else you say after it null. And google is just a search engine they only need like 20 engineers……………
- ttul 3y ago“Basically a taxi service,” except that Uber spans hundreds of cities, coordinates millions of drivers - none of whom work on a fixed schedule - and its only interface with customers is an app that has to be fast, accurate, and reliable at all times.
- matwood 3y agoEven a single taxi event is complex. Tell Uber you want to go to the ATL airport and it'll ask which terminal. If you're catching one from the ATL airport, it'll map and walk you to the rather distant pick up spot. And we haven't even touched on payment yet...
- zht 3y ago
- CodeNest 3y ago[flagged]
- abbadadda 3y ago“Microservices” https://m.youtube.com/watch?v=y8OnoxKotPQ https://m.youtube.com/watch?v=y8OnoxKotPQ
- voz_ 3y agoUber Microservices were such an inefficient PITA. There was buzzword soup of a bunch of half baked infra pieces and they were always migrating. Every part of the stack was rotten. Udeploy, xterra, tchannel, schemaless, etc etc. My peak “wtf” moment was when we had a SEV because two services that should communicate actually used different versions of thrift, both hard forked by Uber, with different implementations for sets. Passing a set from one service to another caused everything to break.
- activescott 3y ago> In preparation for the move to the cloud, the company spent two years working towards making all stateless microservices portable so that their placement in zones and regions can be managed centrally without any involvement from the service engineers I'd like to hear more about how Uber organized the engineering teams over two years to make "stateless microservices portable". How many teams? What were the requirements to each team? What was the timeline? How did they know it was completed? How was it prioritized along other business priorities of the teams? How long did they think it would take originally? Was it worth it?
- s3p 3y agoMaybe direct these questions to a C-level employee at Uber who could potentially answer them for you?
- echelon 3y agoThere are no doubt lots of Uber employees that post here. This is an appropriate forum to ask.
- scarface_74 3y agoAnd why do you think they could answer you with any details without going through comms?
- deleted 3y ago[deleted]
- fragmede 3y agobecause this is the Internet and anyone can make an anonymous account via VPN, if someone were so inclined.
- scarface_74 3y agoYes and I would break my NDA to answer a random question on HN for what personal gain?
- digger495 3y ago[flagged]
- lowbloodsugar 3y agoLook forward to the future write -up of how a Zookeeper issue nuked their entire Mesos stack.
- scarface_74 3y ago> The team used existing tooling to move services between zones in order to ensure they were portable. Firstly, they allowed services to be moved back to the original zone to resolve any portability issues, but once resolved, services would be moved periodically to validate portability and prevent regressions. This is something that most companies don’t do when they say they want to do $x to “prevent lock in”. Uber actually is testing for portability along the way.
- dehrmann 3y agoIt's probably more cost effective to negotiate a long-term max price with your cloud provider with a force majeure clause.
- scarface_74 3y agoUnless you’re crazy enough to work with GCP, the “my cloud provider is going to lock me in and then raise prices” doesn’t happen. AWS has only raised prices in a few very obscure cases ever. One of which is putting a price on HEAD requests in S3 (?). AWS already gives long term price discounts/guaranteed prices for reserve pricing and Big customers already have negotiated contracts.
- Narkov 3y agoPrice increases are just one way you can get screwed. You can also lose out when your provider doesn't drop prices or pick up operating efficiencies that other providers have.
- scarface_74 3y agoAnd when has that happened with respect to either GCP, AWS or Azure at a level that it’s worth migrating? Even if you have done everything in a “cloud agnostic” way, “infrastructure has weight”. Any large migration isn’t just technical , it involves project management, organization training, regression testing, compliance testing, security testing, architecture review boards, vendor negotiations, firewall changes, coordination with third parties who may only allow list certain IP addresses, data migration, etc. Heck they often have multiple physical network connections to the cloud provider (Direct Connect) Anyone who thinks they can run everything on K8s and they have “cloud agnosticism” has never done a very large scale migration. You would be amazed how long it takes to do a bog standard lift and shift of a hundreds of plain jane VMs and VM hosted databases. You can’t get anymore cloud agnostic than that. source: I’ve done a few over the years in both the “real world” and working in the cloud consulting department at AWS (Professional Services). I no longer work at AWS and have no specific loyalty to AWS.
- kosolam 3y agoCongrats to the UP team. The platform sounds good. I especially liked the Balancer component.
- jiveturkey 3y agoTo save you the deep deep dive: on OCI and GCP.
- opportune 3y agoI believe the dollar amount savings figures, they’re big and worthy of a congratulations to the engineers involved! IMO, engineering man hour savings are a lot less trustable. This may eliminate or simplify some engineering processes but IME massive migrations like this simply replace them with a different set of processes; because they’re different and theoretically addressable they’re not counted against the hours saved as they can be bucketed into bugs/to be addressed by the roadmap/legacy behavior migrated from the old system (which is now dangerously-fragile-legacy and not ol-reliable-legac). Eventually someone will come along and decide this too is an inherently flawed platform that needs to be entirely replaced at great expense, and the circle of life continues. This is still a massive undertaking not just from an engineering perspective but from an organizational/process one though. Whoever pulled this off essentially had to coordinate (or figure out how to simplify/explain things well enough to skip coordination) with almost every engineer and likely almost every production service in a company with thousands of engineers. Those in startups may balk about this kind of thing taking two years, but having done my own two year projects (at a smaller but comparable scale) in a big company I can say two years is what I’d consider a highly optimistic and unlikely outcome for a project of this magnitude.
- jvans 3y ago> This may eliminate or simplify some engineering processes but IME massive migrations like this simply replace them with a different set of processes Yes > because they’re different Now I have to learn an entire new set of tools/processes etc that are more useful to someone else but not helpful for me. The old one had its quirks but I knew it inside out and now the whole org has to re-learn how to do everything we did before.
- locustmostest 3y agoI couldn't find any explanation of where the data would be found. Are they splitting data across clouds, and constantly "porting" that data from cloud to cloud as part of their portability? Orchestrating the application layer across clouds is interesting, but how does their data layer work?
- fbnbr 3y agoThe title is misleading. I don’t see Mesos mentioned ones in the article. I got so excited about reading for Mesos helping in the multi cloud world, potentially as the hypervisor for running k8s
- xyst 3y agoI dislike the Uber business itself (horrible treatment of drivers, poor customer service, poor safety controls, bullying of small businesses with Uber Eats, shitty executive level team with questionable ethics). But the underlying technology which carried them to this point is a fascinating read.
- intunderflow 3y agoOh hey, this is the thing I work on. We're giving a talk about this at KCD Denmark on the 14th of November "Keynote: Uber - Migrating 2 million CPU cores to Kubernetes" if anyone is in the area and has any particular interest in this.