20 ms·
What I Wish I Had Known Before Scaling Uber [video]
- tiborsaas 10y ago"Uber is most reliable over the weekends when engineers don't change it" :)
- kenrose 10y agoWe see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning. It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.
- asimuvPR 10y agoI've come to question releasing often as a result.
- jon-wood 10y agoI can see the argument that if releasing causes things to break then don't release so frequently, but in practice the end result of that is lots of things breaking at once and having to unpick everything. Debugging is much easier if you're debugging a single change fresh in your mind.
- bryanlarsen 10y agoYeah, debugging 2 bugs that are shadowing each other often takes several orders of magnitude longer to debug than just a single issue.
- Thrillington 10y agoWhile having tooling for tight debugging loops allows short release cycles, it leaves that decision up to the business
- asimuvPR 10y agoYou are right. Making small changes that can be isolated for debugging purposes is a good approach. What I mean is that we should always question "best" practices and how we apply them to our development process. These days there is a tendency to drink the kool-aid (guilty of this as well). DevOps is something that we are still learning and developing as a profession.
- dozzie 10y ago> DevOps is something that we are still learning and developing as a profession. Not really. DevOps is simply a modernish label for system administration that we have for dozens years already.
- the_other 10y agoIsn't that just "Ops"?
- dozzie 10y agoMy point exactly.
- aurelianito 10y agoYes, but now we actually try to automate in a systematic way; as opposed to a single admin hacking some custom shell scripts. That's what the dev in devop mean. And that's new, at least it was not a common activity in the XX century.
- dozzie 10y agoIf by "systematic way" you mean a group of programmers clueless about system administration hacking together some custom Ansible scripts, then I wouldn't call it progress. Sysadmins have tools to automate their work for a long, long time (cfengine, bcfg2, even Puppet and Chef predates DevOps hype). DevOps didn't bring anything new to the table.
- zaroth 10y agoIt seems like so many "best practices" are really thinly veiled attempts at exploding complexity with only tenuous potential business advantages. We create ourselves so many of the problems we are paid to solve.
- mLuby 10y ago"We create ourselves so many of the problems we are paid to solve" Sounds lucrative ;)
- jessaustin 10y agoIt works for doctors and lawyers; why shouldn't software developers try it?
- thomaslee 10y ago> It seems like so many "best practices" are really thinly veiled attempts at exploding complexity I'll bite. :D It's a bit more nuanced than that IMO, the "deploy often" mantra is only as good as the process around release + deploy. If you half-ass testing and push to production without without a process for verification -- or if, say, your deploy process is half-baked, or your staging environment is worthless -- you can probably expect "interesting" production deploys on a pretty regular basis. As much as we'd like to pretend we're all good engineers, this happens more often than you'd expect -- even with good engineers: at some point a company transitions from scrappy startup to a shambling beast, and the things that used to work for a scrappy startup (like skimping on testing and dealing with failures in production) are insufficient when you've got more eyes on the product. Further, the engineering culture remains stuck in "scrappy startup" mode long after the shambling has begun. And all that's ignoring the fact that less frequent deploys with more changes have their own set of problems. We actually got to a point with deploys of a certain distributed system such that we were terrified if we had more than a few days worth of changes queued up. So many things that could go wrong! :) > We create ourselves so many of the problems we are paid to solve. This, on the other hand, I completely 100% agree with: if not us, then who? :) EDIT: minor formatting change
- cortesoft 10y agoThe breakage rate per new feature is fairly constant; if you release 7 new features once a week or 1 new feature once a day, you will have the same number of issues. The question then is; is it easier to deal with all the issues at once, or a smaller number of issues every day?
- danek 10y agoIn addition to that, I would also worry about interactions between issues. I tend to lean towards spreading out the issues over time to make the eventual diagnosis easier. Occasionally we'll have a problem where we cannot deploy to production for several days (normally it's once a day). A massive inventory of ready-to-deploy features builds up. When we do finally deploy, this deluge of features and fixes creates new issues that force a rollback, delaying the deploy even longer...
- msoad 10y agoYou can roll back a small release that broke the world. A
- AYBABTME 10y agoOr the opposite: embrace failure, practice resiliency, work with it instead of against it!
- awj 10y agoFunny, I've taken it as justification for releasing often. If I have a hard time changing one thing without breaking the service, it's nearly impossible to change a hundred things without breaking something. Since it's a given that I will have to change things, I'll try to stick to a scope where I stand a chance of doing so successfully.
- selckin 10y agoDo they have the same load/users during the weekend?
- jweir 10y agoNo, in the video he states that the weekend is their busiest period.
- DuskStar 10y agoAbsolute numbers of alerts are probably a lot less useful than alerts/use, or alerts/users. After all, if people use the services PagerDuty covers less often on the weekend then you might see a lower alert volume even if issues are relatively more common.
- ex_amazon_sde 10y ago> most production issues are caused by people It's a well known fact, both for systems and networks.
- Yhippa 10y agoAfter working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.
- jsprogrammer 10y agoThe user experience is pleasant as well. Nothing better than your dependency breaking at 4PM on Friday with no support until Monday.
- dnautics 10y agoPretty sure the user volume for Uber is higher over the weekends than during the week. Particularly system stressful times are 1:30-2:15am PT on Saturday and Sunday mornings.
- kasparsklavins 10y agoThere is less commuting on weeekends.
- rsanek 10y agoIn the talk, he specifically mentions that weekends are the busiest.
- chejazi 10y agoWhile a lot of people go out on Friday and Saturday night, I don't think the total volume surpasses commute volume. The surge in pricing you experience late at night is largely due to the number of drivers on the road, which would be less given the time of day. Edited out consideration: traffic in your area might be a good proxy for Uber volume. Traffic late at night is generally low.
- dnautics 10y agoYour statement is true in San Francisco, but Pacific time zone also encompasses Oakland, south bay, los Angeles, and san Diego, where relatively few people commute using Uber compared to going out using Uber. From personal experience, having driven for Lyft and Uber I'm two cities (Sf, sd) surge in SF does get acutely high in the morning, more broadly in the evening (work departure times are spread out over a wide range of hours), and acute around 2am. In California, bar closures statewide are 2am at the latest, so that accounts for unified departure times in multiple markets.
- anezvigin 10y agoWould PagerDuty consider publishing some anonymized aggregate statistics?
- employee8000 10y ago90% of outages are caused by configuration changes which is why change management was so hot for enterprise software.
- ChuckMcM 10y agoThat conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.
- hinkley 10y agoSo I should fire all my developers is what you're saying... Man, I'm gonna save so much money.
- ryandrake 10y agoYou laugh, but a lot of time the best thing for a software product is to not make so many damn changes. People tend to over-estimate the value of features and tweaks and under-estimate the risk of things going wrong.
- hinkley 10y agoOh I know. I've been in a meeting with several bosses where I tell them flat out "don't give me more developers, give me more hardware and tools". They usually look like I have an extra arm growing out of the top of my head.
- ChuckMcM 10y agoThat does save a lot of money :-) The trick is managing the rate of change and the risk of disruption. If you manage it to no risk you end up changing too slowly, if you manage it to close to the risk you end up with unexpected downtime and other customer impacting events. Understanding where you are between no risk and certain doom really only comes with experience.
- hinkley 10y agoI'm bad at optimism, but maybe the takeaway here is that we're finally at the point where it's not the hardware or systems that are the biggest problem. The automation is actually working well enough that it's not the tall tent pole.
- Aeolun 10y agoThat doesn't surprise me at all. If you're not changing software, you can't create any additional bugs.
- pbhjpbhj 10y agoBet I can!;o)
- di4na 10y agoQuestion is : people at each level. Here most of the alerts happens at night... due to batchs... Most of the other alerts are due to network issues, moving servers due to hardware intervention that change the topology of the network, etc etc.
- mathattack 10y agoCouldn't this also just as easily be a change in usage patterns?
- gjolund 10y ago"Move fast and break things" is the worst thing to happen to software development that I can remember.
- enraged_camel 10y agoI think it's OK to move fast and break things when you're in the initial product development stage, where you need to develop features quickly and try lots of new stuff to see what sticks. Once you have paying customers though, it may be time to slow down and start caring about stability, at least for your main branch.
- chetanahuja 10y agoIt's also ok to "move fast and break things" when the most critical thing you're breaking is your annoying high school "friend"s ability to share the latest racist meme with you. There's a reason facebook made this phrase popular.
- adekok 10y agoI suspect that the issue isn't the engineers. It's the changes. A system running just fine will usually continue to run just fine. Until it's changed.
- fail2fail2ban 10y agoOr until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.
- softawre 10y ago> crazy monkey Literally lol'd. It's "chaos monkey".
- jsprogrammer 10y agoChaos monkey is a simulation, I believe.
- bch 10y agoIt's not a simulation, it's real-world production technology. https://en.wikipedia.org/wiki/Technical_details_of_Netflix https://en.wikipedia.org/wiki/Technical_details_of_Netflix
- rrdharan 10y agoAnd it's OSS: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- s_kilk 10y agoI think they meant that Netflix Chaos Monkey is a "simulation" of the real-deal, which is sending a shaved-ape into a physical server room and pulling out physical network cables. I didn't read it as derogatory.
- mshenfield 10y agoFacebook noticed the same thing. http://www.darkcoding.net/software/facebooks-code-quality-problem http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"
- jackweirdy 10y agoHospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.
- dsacco 10y agoThis is interesting. Do you have a source for this?
- jackweirdy 10y agoI believe I read it here, though there are several similar studies and this might not be the one I originally read. https://www.ncbi.nlm.nih.gov/pubmed/25531231 https://www.ncbi.nlm.nih.gov/pubmed/25531231
- huac 10y agoThat study gave no causal reasons. In particular, there was no effect on number of surgeries done. It's just as possible that research-oriented doctors go to the conference, and those doctors are worse at performing surgeries.
- SilasX 10y agoIsn't that because a surgery, if it fails, will result in the patient dying immediately, rather than some random time later? If you do more surgeries, you will increase both the failed and the successful surgeries, whether your success rate is 10 or 90 percent. Compare: "bomb explosions tend to increase around the time when we send in the bomb squad. I guess bomb defusing is pointless." IOW: Surgeries and bomb defusing tend to move forward a lot of the death-probability-mass (while, in theory, destroying some of it).
- samstokes 10y ago"A ship in harbor is safe — but that is not what ships are built for."
- crpatino 10y agoAre you implying that software purpose is to provide developers with puzzles to tinker with... in detriment of their paying customers?
- samstokes 10y agoI see how one could interpret it that way. I meant to imply rather that software generally needs to be regularly changed in order to continue providing value to its paying customers. At minimum, you've got keeping up with security patches, library/framework deprecations etc. Software which has literally not been changed in years is often an insecure timebomb, with unpatched vulnerabilities, and as soon as you do need to change it, you're stuck because all its dependencies are deprecated. Besides, you also need to keep up with the market, competitors, customers' usage patterns changing, etc.
- hueving 10y agoIt's a bad analogy because they don't upgrade ships while they are in use. People aren't saying software is most stable when it's not used, they are saying it's most stable when people stop changing it.
- crpatino 10y agoOk, that's a reasonable position, though not one I completely agree with. It would be nice if we as customers had the freedom to pick up which upgrades to install, and to limit ourselves to security patches and (maybe, some) library deprecations. Windows upgrades let you do just that, even if it is a pain to manage. In my previous job, we held monthly meetings to discuss which windows patches to install in our dozens of business critical Win2k3 servers. For most software, upgrades are opaque. In the best case, you are given a binary choice: install now or install later. In many cases you dont even get that, either the product stops working until you apply the patch or the patch gets automatically applied without your explicit consent. Which would itself not be a problem if most upgrades did what they are ment to do. Instead, as the GP suggest, patches introduce new (sometimes critical) bugs all the time. With pure repair patches, the balance is mostly positive siince they fix more bug than they break, but with every new feature (which usually is required by a minority of very vocal customers) comes a risk that some other more important feature is going to stop working as expected. Industry wide, we lack the discipline to change things in a responsible way. Change management is today what source version control was back in the 90s: something that most people has heard good things about, but most people is not doing correctly, and a significan minority not at all.
- kaspm 10y agoDoes anyone who has worked in software for any period of time not realize this? Computers are dumb they (generally) repeat the same tasks reliably _compared to humans_. Occasionally there are system failures but most failure are caused by code or configuration changes, especially when they interact with inputs and systems in untested ways.
- deleted 10y ago[deleted]
- amelius 10y agoI didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.
- xapata 10y agoSome cities are bigger than others. That's a heavily imbalanced sharding key.
- amelius 10y agoWhy is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.
- kowdermeister 10y agoThat's like a post office way of thinking about the internet, we already past that point. Plus you haven't solved the problem of managing 800 microservices and 2000 programmers, but now you have a deployment problem also.
- jon-wood 10y agoWith 800 services and 2000 programmers you already have a deployment problem.
- eeZah7Ux 10y agoWith 800 services and 2000 programmers you already have a problem.
- new_hackers 10y agoWith programers you already have a problem.
- fitzwatermellow 10y agoLink to Slides here: https://gotocon.com/dl/goto-chicago-2016/slides/MattRanney_WhatIWishIHadKnownBeforeScalingUberTo1000Services.pdf https://gotocon.com/dl/goto-chicago-2016/slides/MattRanney_W... Any speculation as to why Uber doesn't just want to use something like Netflix Eureka / Hystrix instead?
- mranney 10y agoThe Netflix stack has a lot of assumptions about Java, and most of Uber is written in languages other than Java.
- buzzdenver 10y agoDid I miss what WIWIK stands for ?
- paukiatwee 10y agoJust to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)? If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA). Maybe uber only have 200 "active" microservice app running at the same time where each microservices have N running instances. I just cant imagine running 1000 different microservices (e.g different docker images, not docker instances) at the same time.
- employee8000 10y agoYes, and much more than 2 per instance of a service.
- deleted 10y ago[deleted]
- mranney 10y agoOur number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.
- jrochkind1 10y agothat sounds nightmarish
- dastbe 10y agoWhy? I'm assuming their highish engineer/service ratio is because their services do less individually. Anecdotally, I've worked on services that ran tens of thousands of instances across the world. You build the tools to manage them and it works very well.
- softawre 10y agoIt's not the number of instances that he's talking about, it's the number of images. Microservices can go to far. I'm very thankful for this video.
- andrewvijay 10y agoHighly recommended video. Lots of stuff he spoke are very relatable. Like having many repos , storing configs as a separate repo, politics by people, having a tracking system
- babyrainbow 10y agoSorry, don't think so. I went through the slides and didn't find anything really interesting.
- softawre 10y agoYou went through the slides but didn't watch the video? If so, you can't really make the argument that it isn't interesting.
- andrewvijay 10y agoI'm recommending the video explicitly. Not the slides!
- gtrubetskoy 10y agoI think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to be using high level dynamic languages like Python and Ruby, and camp 2 will be more on the strongly typed side e.g. C/C++, Go, Java).
- KayEss 10y agoI still kind of think you can have great REST and JSON and make full use of a great RDBMS. REST/JSON with strong schemas (and loose where that makes sense), using Postgres and C++.
- fail2fail2ban 10y agoREST/JSON makes sense between a human backed browser and a middle tier; between machines, SOAP is the best contract first technology, or at least a custom XSD backed XML schema.
- iamnothere 10y agoIs your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas. Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.
- mtberatwork 10y ago> Sure, in the long term things should be refactored, structured, and optimized. How often does that really happen though? Once you've amassed enough technical/data debt, resistance to refactoring increases until it never happens at all. Having well defined, coherent data models and schemas from the start will pay off in the long run. Applications begin and end with data, so why half-ass this from the get go?
- dorianm 10y agoI got told by an Uber engineer that's because of their hyper-hyper growth their tech is basically the biggest mess possible. That's also why that makes it an interesting place to work at and helped them achieve this growth. Personnaly I think this should be made into a global app with no geo-fencing (e.g. Available everywhere basically).
- StreamBright 10y agoAbsolutely amazing to watch. I think most of the big companies (Amazon, Google) already have solutions for these issues like: limited number of languages, SLAs between services, detailed performance metrics and the ability to trace microservices.
- Roritharr 10y agoI'm not a big green-IT guy, but always pushing your systems close to its load maxima and then backing off with the test traffic as real traffic comes in feels like an enormous waste of electricity.
- taeric 10y agoAlmost as much as the overhead of splitting everything to a thousand microservices...
- petetnt 10y agoReally enjoyed this talk. Our services don't quite (yet :)) run at that scale, but many of the issues mentioned have already peaked at some point. It's also good to have (more) validation to some choices we have made in the past, are currently making or are thinking about making in short term future.
- agentultra 10y agoWhat I Wish Small Startups Had Known Before Implementing A Microservices Architecture: Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes. Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still not going to gain much. There are so many well-known strategies for coping with scale that I think the main take-away here for non-Uber companies is to start up-front with some performance characteristics to design for. Set the upper bound on your response times to X ms, over-fill data in order to keep the bound on API queries to 1-2 requests, etc. Know your data and the program will reveal itself is the rule of thumb I use.
- rando832 10y agoI've never heard the term over-fill data, and google seems to have no idea of it in reference to software.
- agentultra 10y agoAh... it's almost the reason for GraphQL. Basically if you're building a hypermedia REST API you return an entity or collection of entities whose identifiers allow you to fetch them from the service like so: {"result": "ok!" "users": ["/users/123", "/users/234"] } The client, if interested, can use those URLs to fetch the entities from the collection that it is interested in. This poses a problem for mobile clients where you want to minimize network traffic... so you over-fill your data collection by returning the full entity in the collection. {"result": "ok!" "users": [{"id": 123, "name": "Foo"}, {"id": 234, "name": "Bar"}] } The trade off is that you have to fetch the data for every entity in the collection, the entities they own, etc; and ship one really large response. The client would receive this giant string of bytes even if the client was only interested in a subset of the properties in the collection. GraphQL does away with this problem on the client side rather elegantly by allowing the client to query for the specific properties they are interested in. You don't end up shipping more data than is necessary to fulfill a query. Nice! ... but the tradeoff there is that you lose the domain representation in your URLs since there are none.
- admiralhack_ 10y agoGreat video. I went in expecting it to cover mostly the technical side of things. Instead Matt gave a great overview of the team / engineering organization dynamics to watch out for when adopting microservices. (I particularly liked that he pointed out how developers may write new code / services instead of dealing with team politics.)
- eternalban 10y agoI think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field. Microservices address that gap. And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".
- majormajor 10y agoDo you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on ("running locally, with low volume, everything works together great!")? Personally, I vastly prefer debugging and building on non-microservice architectures than on something split willy-nilly into dozens of services (because most implementations I've seen don't do microservices with clean conceptual boundaries - it's more political/organizational divisions that determine what lands where, not architectural concerns).
- user5994461 10y ago>>> Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on That's one way to scale development projects. Have multiple teams of dev who work on separate stuff. They can develop in their corner however they want and that makes them happy. The final thing is a clusterfuck of services with little cohesion [YET they ALWAYS manage to put the shit together in production in a mostly working state]. The alternative implies to have a consistent and cohesive view of the components. For that, you'll need to have people with experience in architecturing scalable and sane production systems [so they understand what are the consequences and tradeoffs of ALL their decisions]. I know of very few people on the planet who can design systems. (We're talking past unicorn-rarity here). Plus, the developers must actually listen AND care about the long term maintenance AND the people involved (i.e. not want to do shit that will hit either them in 6 months or the team in the next office). The amount of collaboration AND communication AND skills required to operate this strategy is beyond the realm of most people and organisation. There are very few individuals who can execute at this level.
- mstade 10y agoI quite enjoyed watching this. My takeaway isn't so much that this is a critique or endorsement of microservices, but rather just a series of observations. Not even lessons learned in many cases, just honest pragmatic observations. I like how he doesn't seem to judge very much – he obviously has his own opinion of several of these topics, but seem to let that more or less slide in order to not get in the way of these observations. Good talk, will likely watch again.
- nichochar 10y agoHonestly I don't think the problem is microservices. I mean everything he brings up is true, but it's more of a "how you do microservices" issue. I used to work in startups, and overall was impressed with velocity. Then I joined a big valley tech company, and now I understand. It's because they hire smart and ambitious people, but give them a tiny vertical to work on. On a personal level, you WANT to build something, so you force it. I think you solve this by designing your stack and then hiring meticulously with rules (like templates for each micro service), instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)
- inthewoods 10y agoI found this video so super interesting and yet frustrating for completely personal reasons: the company I work for used to sell a tracing product that was specifically designed for the distributed tracing problem and handled all of the issues he highlighted - trace sampling, cross-language/framework support built in, etc. It was/is based on the same tech as Zipkin but is production ready. Sadly, he and his team must have spent a huge amount of time rolling their own rather than ever learning about our product. Now, it still might not have been a good match, but man, the problems he mentions were right in the sweet spot of what our product did really, really well.
- dllthomas 10y agoSure, but was it Invented Here?
- inthewoods 10y agoYes, I think that is the primary problem - they invested in Zipkin. At the end of his talk, he mentions the idea that you should prioritize core functions and use vendors for everything else. But he's right that some person created that code - and thus it becomes difficult to get them to switch. And instead it becomes a huge source of technical debt for a company that is non-core to their mission.
- jasonshen 10y agoThis is why sales and marketing is so so important and yet overlooked/undervalued. You might have a great product that's perfect for your customer, but you also have to convince them to use it and pay for it.
- inthewoods 10y agoTrue - and marketing to developers is very challenging - a lot of noise to get above.
- gb123 10y agoHey, Matt Ranney, I used your node-pcap library to learn how to parse PCAP :) Did not know you worked for Uber, thanks!
- iamleppert 10y agoIt amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing? When did we loose our heads and think such an architecture is sane? The UNIX philosophy is do one thing and do it well, but that doesn't mean be foolish about the size of one said thing. Doing one thing means solving a problem, and limited the scope of said problem so as to have a cap on cognitive overhead, not having a notch in your "I have a service belt". We don't see the LS command divided into 50 separate commands and git repo's.....
- venantius 10y agoWhile I'd concede that 1,700 services is a lot, and probably a function of the fact that they're hiring too many engineers too quickly (there's no possible way to indoctrinate people into sane and standardized engineering practices at this sort of hiring growth), I don't actually think what he's describing is that unusual for a company at Uber's scale or experiencing Uber's rate of growth. I work at Airbnb and while we have many fewer engineers and services than Uber does, most of the issues he talked about in the talk resonated with me to one degree or another.
- iamspoilt 10y agoI couldn't agree more.
- willejs 10y agoSaying something like '...they have very poor engineers working for them.' is pretty unfounded, and naive. It's easy to say that having 1700 services is overkill from our point of view, but we don't know the complete architecture, problems being solved and environment that they operate in. One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestration, common libraries, standards and deployment. Deploying and operating new services becomes relatively straight forward. Saying that, 1700 is a lot, which makes me intrigued to see inside uber.