27 ms·

A former Uber engineer's disaster story

• deleted 6y ago
  [deleted]
• bfrog 6y ago
  This to me reads of Swift being pushed hard well before it was truely ready. Polynomial linker time? Wow. Ouch. No thank you.
  • twic 6y ago
    In version 2.x, no less! What was it in 1.x, exponential?
    • jonplackett 6y ago
      Swift 2 was so, so horrible. I really loved it and continue to from v3 onwards though.
  • saagarjha 6y ago
    Linkers must run in polynomial time for certain things; this is a problem independent of Swift.
    • lgg 6y ago
      That is true, but I am pretty sure this is misattributed. At the time we had two major issues which slowed down loading these sorts of dylibs. One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur). The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it). IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.
• pavlov 6y ago
  Apple's botched introduction of Swift is easily responsible for billions of dollars in lost productivity across the industry since 2014.
  • saagarjha 6y ago
    "Botched introduction, really? Perhaps Apple could have been a bit more upfront about the state of the tooling (which, of course, they never will be) but I think they did a rather good job considering that they moved development out from "total secrecy" to "community-involved" within two years, all the while shipping new features and improvements to the experience at a fairly decent pace.
    • pavlov 6y ago
      > “which, of course, they never will be” That’s the problem. Apple knows by now that their developer community laps up absolutely everything they put out and regularly gets burned. (I’m old enough to have rewritten code for Cocoa Bindings in 2004. It’s the next big thing! And garbage collection in 2008. It will fix everything! And GCD... and...) For such an enormous change as Swift, they should have switched off the reality distortion field, and gone with a more sober, more academic approach.
      • Apocryphon 6y ago
        What's wrong with GCD?
        • pavlov 6y ago
          It's a great library for a lot of things. When originally introduced it was also heavily oversold by Apple as a magic bullet for concurrency. Here's one account: https://tclementdev.com/posts/what_went_wrong_with_the_libdispatch.html https://tclementdev.com/posts/what_went_wrong_with_the_libdi...
          • astrange 6y ago
• perryizgr8 6y ago
  What's this "cellular limit"? Is it some apple specific oddity?
  • tass 6y ago
    The App Store doesn’t allow downloading apps over a certain size unless connected to wifi.
    • hundchenkatze 6y ago
      They changed it with iOS 13. It will warn you now, and you can still download over cellular if you want. https://www.wandera.com/ios-13-app-download-cap-removed/ https://www.wandera.com/ios-13-app-download-cap-removed/
    • astrange 6y ago
      Note that some of the dropping platforms advice doesn’t really help here, or only helps coincidentally, because the App Store serves you a device-specific app binary.
  • hundchenkatze 6y ago
    Yep, it was to prevent you from accidentally blowing through your cellular data cap. They no longer prevent the download, just warn you that the app is over 200MB.
• linuxftw 6y ago
  The biggest engineering disaster Uber had was when it intentionally disabled the automatic safety brakes on their 'self driving' car and killed a person.
  • gambiting 6y ago
    I don't know why you're getting downvoted. Making a crappy, slow app is nothing compared to conciously deciding to switch off the built-in autonomous breaking system in lieu of your own(which then doesn't work and kills a person). That's an engineering disaster, not a taxi going to the wrong city block a few times.
    • jcranmer 6y ago
      Considering that the engineer describes his story as "the biggest engineering disaster I’ve ever had the misfortune of being involved in," I don't think it is relevant to his story that the self-driving pedestrian collision occurred. He isn't claiming--and even from the thread title, I wouldn't assume he is claiming--that this is the biggest engineering disaster ever, or even the biggest engineering disaster Uber faced.
      • gambiting 6y ago
        Yeah, I suppose that's fair enough.
    • jonny_eh 6y ago
      They bloated the app so that it wouldn't be slow. The whole twitter thread is about that.
  • deeviant 6y ago
    Less a engineering disaster and more a human one.
  • wbxp 6y ago
    I was working at Uber ATG when Elaine Herzberg was killed. They kept the details secret from the other employees, citing fears of "too many cooks". I recall being offered grief counseling. Uber disabled Volvo's automatic emergency braking system, which probably would have saved Elaine, to develop their system on public streets. I left the company and took a pay cut to work on ethical robotics instead. https://www.washingtonpost.com/local/trafficandcommuting/pedestrian-in-self-driving-uber-collision-probably-would-have-lived-if-braking-feature-hadnt-been-shut-off-ntsb-finds/2019/11/05/7ec83b9c-ffeb-11e9-9518-1e76abc088b6_story.html https://www.washingtonpost.com/local/trafficandcommuting/ped...
• gazab 6y ago
  I want to hear the Android side of this story!
  • nobunaga 6y ago
    Android was a much easier story, Still had some binary size and method count issues but we were able to overcome them thanks to another brilliant engineer we had in the team in SF who worked on Android.
    • lern_too_spel 6y ago
      The Android apk is 56.9 MB for my device, while the iOS app is 331.3 MB. What accounts for the 6x difference?
      • bsimpson 6y ago
        Courage.
      • saagarjha 6y ago
        I assume that the constraints placed on the Android team are much higher, as that market seems like it would be much more sensitive of app sizes.
        • lern_too_spel 6y ago
          That is true, but what specifically is the iOS app getting in that extra quarter gigabyte? What couldn't they throw away to get under 200 MB at all?
          • saagarjha 6y ago
            I honestly have not been provided with a good answer beyond "we just didn't spent resources to do it".
      • pandanomic 6y ago
• tyingq 6y ago
  The Otto due diligence has to be up there as well.
• MontagFTB 6y ago
  Computing is always susceptible to shiny object syndrome. Just because it’s new doesn’t make it better. I wonder if things would be different with a more nuanced approach to Swift adoption, knowing what they knew of its limitations already, and being realistic that the language is not a panacea (none are!)
• nwsm 6y ago
  Great thread. > So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going. > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs This would scare me too. > I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out. This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.
  • curiousllama 6y ago
    This isn't sunk cost; the exec chain just has different incentives. They're hired for outcomes. It's their job to make it happen, not make it make sense. > A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks. When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.
  • ehostunreach 6y ago
    >> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs > This would scare me too. FWIW, I made it clear that this was also a great way to break the app :)
  • busterarm 6y ago
    Uber also has a habit of solving problems by hiring a manager externally and building a brand new team of all new hires for them. There's some upsides to that, but basically even though you're a W-2 employee, you're effectively a consultant in that arrangement. Your job is not secure.
  • 6y ago
• app4soft 6y ago
  > “biggest engineering disaster” I just supposed that this was a story about Uber Elevate ElectricVTOL (eCRM) design.[0,1] [0] https://twitter.com/TheWoracle/status/995107989886353408 https://twitter.com/TheWoracle/status/995107989886353408 [1] https://github.com/OpenVSP/OpenVSP/tree/main/examples/vspaero_ex/Uber_eCRM-002 https://github.com/OpenVSP/OpenVSP/tree/main/examples/vspaer...
  • deleted 6y ago
    [deleted]
• gambiting 6y ago
  Were all of these.....tweets originally? Or does the author write in this crazy "few sentences per paragraph then break" style? So hard to read.
  • fjni 6y ago
    nailed it: https://twitter.com/StanTwinB/status/1336890442768547845 https://twitter.com/StanTwinB/status/1336890442768547845
  • nwsm 6y ago
    Yes, threadreader.app concatenates a thread of tweet replies, and each section is clickable, linking to the original tweet.
  • stuff4ben 6y ago
    Yep, and it's why Twitter as a blogging platform will never take off. I don't understand why people do that. Just find some place to blog and post a link to that in your twitter feed. Gah! It's so frustrating when people abuse a platform like that...
    • kgwgk 6y ago
      Or write it in word and post a screenshot in your twitter feed :-)
• nobunaga 6y ago
  I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it. Problem here with Swift came down to - Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple) - Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here) Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired. Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.
  • nwsm 6y ago
    Appreciate your insight, and I'm sorry to hear about the AMS team. The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years". Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal. Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?
    • nobunaga 6y ago
      I was not part of the initial analysis team so cannot really comment if objc was more suitable as it would require to look at objc at that time and do tests etc but I suspect we would have had a more smoother experience. Though the the code at the time was bad and indeed something needed to happen to ensure future scalability of the teams and product/code, but I think there were more sane paths we could have taken. It would have cost the company more time/money and at that time the company was battling Grab, GoJek, didi, Lyft etc so we did not have time to lose. Lots of factors to take in to consideration. No real easy answer I havent dabbled in mobile developer platform tooling for the last few years now so I cannot comment on the iOS side, all i remember is that back then it was a shitshow :) Mobile is still playing catch up to how backend infra and tooling operates, I suspect its better but probably still got a long way to go.
• lxe 6y ago
  Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here. Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.
  • nobunaga 6y ago
    Management had very different incentives while engineers had different motives. The two were not aligned, and TK culture was still very prominent back then (work hard, dont go home early, we are going to be a 1 trillion dollar company etc etc) and Thuan definitely helped him spread that culture. So definitely not great days but I think the engineers took away a lot from their time there. Lots of lessons learned
  • macNchz 6y ago
    Yeah, I imagine this pressure from the top had something to do with the "90 hour weeks" cited in the thread...I've never worked in an org as large as Uber, but personally I have made some very bad engineering choices when in crunch mode, especially on the doubling-down-on-bad-decisions front. When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.
• jamesmarkewen 6y ago
  Who's making the version of this post for Netflix?
• staplung 6y ago
  I've lost track of the number of "biggest engineering disaster" stories I've read that begin "so we decided to rewrite it..." The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going. Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
  • coldcode 6y ago
    When you write code like shit, the language doesn't matter. What Uber did was write a monstrosity with too many engineers and too many features without considering that was beyond their ability to deliver reliably. The product decision making was also clearly deficient and didn't help. Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.
    • scoutt 6y ago
      It can be quality code written by good engineers, but there are issues you won't see until you deal with big numbers. The problem here is that Swift seemed to be a cool language that it might work for a "Hello world" app. You won't notice any problem until the thing grows up to a certain point. For example, I work in embedded and I am proposed with a new "X brand" Chinese Modem to integrate in our devices. I don't recommend it but since it costs less I'm pushed to evaluate it. I am handed 3 samples and they all work fine, so management convinced themselves that Brand X from China is the way to go. Once in production the modems started to fail at a 10% rate (10 devices over 100 won't work). And that is a problem you only see only when manufacturing a certain quantity of devices. Now there are delays, an ongoing battle with the manufacturer of modems and a unhappy customer (that's why I didn't recommended it).
• aaronbrethorst 6y ago
  It was pretty obvious when Swift 1.0 launched that it should’ve been called Swift 1.0 beta 1. Not a knock against anyone who worked on the project, other than senior management at Apple. Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet. I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’
  • saagarjha 6y ago
    Large Swift apps are easy to find; companies would invariably use it as a marketing point on their engineering blog. Even if not, people open up apps all the time, any large app that uses Swift would be found and discussed.
    • aaronbrethorst 6y ago
      Large Swift apps are easy to find Sure, if they exist. I'm not talking about the world as it is today in December 2020 with Swift 5.3. I'm talking about 2016.
      • saagarjha 6y ago
        My point was that there was no need to ask Chris Lattner back in 2016: you could just look around the landscape and see that there were not many, as they would be easy to see if they did exist.
        • aaronbrethorst 6y ago
          Lots of companies don’t speak publicly about their product strategies, like certain former employers of his.
          • 6y ago
• xnx 6y ago
  Definitely thought this was going to be a story about when their car killed someone: https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverles...
  • ericlewis 6y ago
    I doubt any engineer who worked on that would be even remotely allowed to talk about it.
• XorNot 6y ago
  The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point. Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!" Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.
  • asimpletune 6y ago
    Yeah it’s kind of bizarre that after writing everything leading up to that, the author would still frame the ObjC folks in that way. Like, this or that eng. was so brilliant with their hack, but all those obj-c eng. who advocated boring but stable solutions all along were just curmudgeons with no helpful advice.
    • quasse 6y ago
      The problem from the author's perspective is that the Obj-C "curmudgeons" were proving the right solution to save the app (drop Swift and cut losses + go back to a functioning language) but not the careers of the people who had gotten them into this terrible situation.
      • andrewprock 6y ago
        This perspective is reflexive. The "ObjC curmudgeons" didn't feel like the adventures that the "Swift zealots" were generating helped their careers in any way.
    • jonny_eh 6y ago
      Sounded like a "both sides" analysis, where the assumption is that both sides in a disagreement must share blame.
      • entropicdrifter 6y ago
        The Middle Ground Fallacy
• justapassenger 6y ago
  My mind is constantly blown away that Uber app needs to be so big.
  • bhupy 6y ago
    Does it blow your mind that Google Maps is ~190MB?
    • jonny_eh 6y ago
      Contains more functionality than Uber at a smaller size.
      • bhupy 6y ago
        Uber app contains very similar functionality, namely real-time map view that needs to be able to render any part of the globe (or at least serviceable area), a location search engine, the ability to render drivers around you in real-time. Just like Google Maps lets you select point A-B directions across cars, walking, public transit, Uber gives you multiple products to select (UberX, UberPOOL), and renders the path on the app accordingly. The Uber app also does a lot of extra computation at the app side to solve some of the business problems that the OP mentioned in the thread. For example (from the Twitter thread): > Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience. Uber's approach to solving this involves some novel computation that adds to the binary size -> https://eng.uber.com/rethinking-gps/ https://eng.uber.com/rethinking-gps/ Uber app also tries to resolve and render points-of-interest, so that you don't have to search for just an address as your destination, you can also search for "Dolores Park". That not only powers the search on the backend, it's also rendered on the frontend so that when the map tile for Dolores Park is rendered, the app knows to drop a tree-shaped icon on top of it labeled "Dolores Park". Uber has been using their own maps for a lot of this, and has an entire Maps org that does in-house map display, routing, and search. And that's just the before-trip functionality. There's extra stuff that happens during the trip, including showing your current location, being able to share your trip with friends, real-time 911 calling for safety, etc etc. And this is all just the similarities with Google Maps. Uber does extra stuff on top of that, including: Onboarding flows for signup/signin, including the 3rd party identity services. Payments: the app needs to support payment linking flows so that you can add and verify your credit card. Sometimes your credit card declines, and there's a whole arrears flow that needs to be built out to support that. And credit cards aren't the only way to pay for trips in Uber, you also have proprietary systems like Venmo, and PayPal that you can link. And on top of that, there are myriad such proprietary wallets all over the world that Uber has supported since at least 2015, including PayTM (in India) and Airtel Money (India and Africa), just to name a couple examples. And in most of the world outside of the US, you can pay for Uber trips with paper cash, and that's a whole flow on the frontend during onboarding / arrears settlement. This needs to all be packed into the same binary because you don't download a different version of the app when you go to a different country. They also bundle UberEats into the app for some people, and there are probably hundreds of other random features that are prototyped and hidden behind feature flags, or variable experiences that are selectively rolled out in certain geographies. They all need to be packaged into the same binary for everyone. Throw in other random product features like their Spotify integration, and it all adds up. Uber isn't just a dumb CRUD app. You can also see the same in other comparable apps, Lyft's iOS app is 290 MB, Ola is 205.3 MB, Grab is 289.6 MB, GoJek is 236.2 MB, Yandex Go is 145.4 MB, etc etc. EDIT: there's an even better written summary somewhere up-thread of the sheer amount of business complexity that needs to be baked into the app -> https://news.ycombinator.com/item?id=25376346 https://news.ycombinator.com/item?id=25376346
• flareback 6y ago
  > So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions. I really like this point.
  • munificent 6y ago
    This is a good point, but it's important to be clear about it. There is no universally superior language. There are a number of competing factors which means that optimizing for one sacrifices others. So instead of a single point in space that maximizes all of them, there is instead a volume bounded by a curve and going farther in one direction requires lower values on other axes. But, there is a lot of space nowhere near that trade-off optimizing boundary. You can absolutely make languages worse in ways that are strictly negative for all users. For historical reasons, all languages contain some amount of that dead weight. So the absense of a perfect language does not imply that all existing languages are right at the edge of the optimizing boundary.
  • GVIrish 6y ago
    I think not clearly understanding the tradeoffs was a huge factor here, but I would go one step deeper into that decision-making process. One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and took a cursory look at build times for a small pilot, but failed to look deep enough to find some of the crippling shortcomings. Related to the first point, a lot of the time with a new technology, the show stopper bugs/shortcomings/issues haven't been discovered yet. If you're choosing a battle-tested technology, most of the nasty edge cases and limitations have been discovered at some point. But with something new, there's a lot more uncertainty with what you'll run into once you put it into production. That risk of surprises is higher with some new tools and technologies than others. But it's a big wildcard when you're betting your billion dollar business on it. Maybe that bet pays off, but you need to do some sober analysis on that risk of unknown unknowns and have some idea of how and when you'd make the decision to cut your losses.
• mandevil 6y ago
  One company I worked for a few years ago, I joined right as a just such a ground-up rewrite of their core system was coming into place. Everyone in orientation was telling me "You'll be spending most of your working life dealing with <new project codename>" but they were actually training me on the existing system. About a month out of orientation there is a company wide. They canceled <new project codename> because it wasn't meeting it's technical specs. It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.
• nineplay 6y ago
  Not a mobile developer - Was the move to Swift inevitable and they started too early or should they have stayed with - and still be using - Objective-C?
  • donmcc 6y ago
    I think it was clear early on that Swift is the future of iOS development. But if you’ve been around long enough, you’ll have observed that it takes about 10 years for a new language and associated tooling to reach “maturity”. What “maturity” means is debatable, but one aspect is that the ecosystem grows to handle the extreme edge cases like very large scale development. And boy was Uber pushing the edges of what was possible on iOS at the time. So by my way of thinking, they moved at least a couple years too early. (The Swift project started internally in 2010, first public release in 2014, Uber’s rewrite started in 2016.)
  • cellularmitosis 6y ago
    If they had stuck with the Swift ban they would certainly be facing a hiring problem today.
• dap 6y ago
  Great thread. Thanks for sharing! It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that. I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?
  • saagarjha 6y ago
    I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same. People saw the issues that were placed on a national stage, then looked inwards and saw similar issues much closer to home, and these things started coming out as people were no longer OK with keeping quiet about it. For the other half of your comment: I personally think many companies have issues focusing on non-technical problems. From the thread, the issues were clearly that they were writing far too much code, it seems like they were not testing enough, they lacked a person who would tell them how their changes would be perceived by the public. This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others. And you can see the solutions: while highly technical and "awesome", they are obvious band-aids. It's just that it's much easier to slap on technical band-aids than fix the root, non-technical problem.
    • dap 6y ago
• tantalor 6y ago
  Readable version: https://threadreaderapp.com/thread/1336890442768547845.html https://threadreaderapp.com/thread/1336890442768547845.html
  • andrewprock 6y ago
    Thank you.
  • pauljurczak 6y ago
    Readable and accessible to ex-twitter users.
  • epaga 6y ago
    Weird, that was what I originally posted but someone (dang?) changed it to Twitter.
• jimbob45 6y ago
  Seems like the lesson here is to not adopt new technologies on your flagship products (instead of first testing the waters on smaller products).
  • cellularmitosis 6y ago
    > There’s a video somewhere in one of our talks of an Uber engineer typing a single line statement in Xcode and then waiting 45 seconds for the letter to appear in the editor slowly, one-by-one. I'm not sure if you are familiar with Xcode, but this indicates an off-the-charts LOC count. Something isn't adding up -- the user-visible features and screens in the app don't necessitate a codebase this large. My guess would be the intersection of several compounding factors: - a product team endlessly pushing special-case features which aren't core to the user experience (the twitter thread talks about the rate of new code being added as if it were a foregone conclusion the app would continue to grow without bound), - an A/B testing framework which leads to the deployed codesize being much larger than what the typical user actually interacts with (worse if they are lax about culling the vestigial A's and B's), - and reaching a "thermal runaway" point of too many devs x too many lines of code, where code reuse stops happening. Once you reach the point where it takes less time for a dev to write a given feature in a from-scratch fashion rather than doing some codebase archaeology to find existing code structures which can be reused, you've reached thermal runaway. The Facebook iOS app revelation of "18,000 classes" indicates they were likely suffering from this as well https://news.ycombinator.com/item?id=10066338 https://news.ycombinator.com/item?id=10066338 Was Swift up to the task? Perhaps not. But does a ride-hailing app really necessitate this level of codebase complexity? Uber's iOS app is now over 300MB...
    • jimbob45 6y ago
      It’s funny you quote that specific line because that one line stood out to me too. I agree that something extremely fishy was going on that couldn’t have been solely explained by “too many linked libraries”. However, choosing a more mature technology like Java or C# or even Golang would have certainly alleviated a great deal of stress from their process.
• secondcoming 6y ago
  > Some of my ex-Twitter colleges called on me to quit such an evil company that would track you like this. Twitter owns an company called MoPub that broadcasts user location information to the adtech industry.
• draw_down 6y ago
  > The Zealots were in denial about the problems that Swift created. The curmudgeons complained about everything you could imagine without providing much in the way of solutions. I can sympathize with someone caught between these two factions, but I really dislike this rhetorical move where people say "well you're not providing solutions". Particularly if they were the one who advocated making the mess in the first place. Those people are just galling, the solution would be to fire their fucking ass. Everything about this originated from people being hotshots trying to use underbaked tech and trying to get promoted (which worked by the way). The "solution" is not running your company/engineering org that way. The way this ends, in an equanimous "who can say if it was worth it" is totally bonkers to me. Think of the human cost alone. So much of this turmoil could have been avoided by not just simply going balls to the wall. This story is upsetting.
• overgard 6y ago
  I know C++ has its warts, but almost every mobile project I've worked on has been C++ with a thin layer of objective c and java interop. IMO it's really not worth it to get so tied up in a single companies tooling even if the developer experience is superficially nicer. C++ has all sorts of awful, but it's familiar awful with mature tooling. Admittedly the things I've worked on has been games and CAD systems so the internal data models and simulation are a lot more complicated than the UI, but I think that holds for a lot of apps.
  • georgiecasey 6y ago
    The vast majority of apps are UI, CRUD, business logic and other things where native code (as in compiled C/C++) is just overkill and makes things more complicated. The few exceptions seem to be the fields you listed.
• pjc50 6y ago
  This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens. I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies. (The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)
  • dreamcompiler 6y ago
    Follow-on tweets address this somewhat: Every airport has different pickup locations, every region has different payment rules including cash, etc. Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.
    • lock-free 6y ago
      There are a lot of ways to frame these problems as interesting algorithms but it doesn't really strike me as something that is troubled by scale. I've worked on similar problems where "everything is a special case" in some vein and it takes a few very clever minds to construct the algorithms and datastructures that can come to a good solution. But that's just it, it's the work of a few engineers and not thousands writing if-else blocks for each airport. I still don't understand what Uber does that requires so many people getting paid so much money, the only thing that makes sense to me is they don't want them working for Lyft. It would be helpful for people to shed more light on how engineering resources are spent and what particular systems require so many hands to realize, but that's too valuable information to leak I'd imagine.
      • Justsignedup 6y ago
• lisper 6y ago
  Am I the only one who wants to scream when encountering long-form content on Twitter?
  • forgot-my-pw 6y ago
    Twitter is such an awful use-case for this. Just write a blog post!
• owenversteeg 6y ago
  Wow. The real lesson here is that all this "engineering" and doing stuff the supposedly elegant and better way was a colossal disaster.
  • vp8989 6y ago
    There is no cohesion or elegance at those kinds of head counts. There are just too many cooks, too many different perspectives, most toolchains allow you to solve problems in too many different ways. IMO and contrary to what people say, making good software is not a team sport. You could try to offset that by some combination of keeping head count low, optimizing for technical homogeneity when hiring and/or using intentionally spartan toolchains (Golang for example) but the odds are still against you. Also, US business culture is maximalist and land-grabby in nature so no one is going to let you actually apply any of those constraints. The people who write checks don't actually care if the software is "good" because in almost all cases it doesn't have to be.
• 11thEarlOfMar 6y ago
  I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches. I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations[0][1]. [0] http://read.pudn.com/downloads196/ebook/923902/ShowStopper!-The%20Breakneck%20Race%20To%20Create%20WindowsNT%20And%20The%20Next%20Generation%20At%20Microsoft.pdf http://read.pudn.com/downloads196/ebook/923902/ShowStopper!-... [1] http://www.proudlyserving.com/archives/2007/09/dave_cutler_win.html http://www.proudlyserving.com/archives/2007/09/dave_cutler_w...
  • saagarjha 6y ago
    I remember that some engineers on the iPhone project went through divorces as a result of working on it.
  • munificent 6y ago
    > I expect that's hyperbole, but there is a real cost to these death marches. I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.
    • anonacct38 6y ago
      I went through a pretty intense crunch time at a fortune 10. We had one person go to the hospital with tachycardia and 3 people who were vomiting. There were only 10 people on the team.
  • sixstringtheory 6y ago
    I believe it. During my last stretch of unemployment I started tracking the days I couldn't get out of bed in my spreadsheets.
  • Bayart 6y ago
• alexggordon 6y ago
  If I could read a book of these stories I'd stop everything and read it now. Absolutely incredible. I love stories like these, but I also understand these hells are probably reserved for the special world of startups. The interesting takeaway from this for me is the question "Was the rewrite successful?". As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me. Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story. The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.
  • aliceryhl 6y ago
    In that case I can recommend the book the phoenix project.
• bobobob420 6y ago
  Posting stories as Twitter threads personally for me make me loose all respect for you as an engineer. I mean what in the hell is the abysmal way of reading? How could any person allow themselves to post something like this?
  • andrewprock 6y ago
    As with Swift vs. ObjC, this may be an instance of "fashion over function".
  • astrange 6y ago
    I mean, I’ve seen people post things on Twitter that would make me not hire them[1], but it was usually the actual words and not the way they’re posted. [1] anyone remember that weird thread about Beau Brummell inventing toxic masculinity?
• newbie578 6y ago
  I wonder if you tried to do the same thing today, but with different tools, let's say Flutter, how it would turn out? If you think how the Uber app looks and works, it should be a major relief doing it in Flutter, I don't see any major breakpoints.
• deleted 6y ago
  [deleted]
• CyberRabbi 6y ago
  Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.
  • wankerrific 6y ago
    This 100%. From the tweets it sounds like the problem they were trying to solve was to make the app more modular so it would scale better to more devs. There were/are well established patterns for doing so in ObjectiveC. The Swift rewrite was not necessary and probably took a very difficult project and turned it into the death march that the OP describes. They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift. This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.
  • deckard1 6y ago
    I shudder at the thought of hundreds or thousands of Uber engineers banging out a rewrite together. I've been through a number of rewrites, and every single time it always meant the same thing: get your resume ready and look for the exit. It's going to be a multi-year shitshow.
  • mpweiher 6y ago
    I've spent most of my career doing rewrites, each time reluctantly, and each time it was a resounding success. So easy with the absolutes :-)
    • CyberRabbi 6y ago
      Point taken but something tells me we might have different ideas about what it means for a rewrite to succeed.
      • mpweiher 6y ago
• dreamcompiler 6y ago
  Are webview apps still possible on IOS? Given that the Uber app has to work online anyway, it seems like a lot of this headache could have been avoided by keeping most of the logic on the server.
  • saagarjha 6y ago
    Yes, they are. The experience would be quite poor for many things, however.
• carapace 6y ago
  I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it. > I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out. > With only a week left we decided eat the 8 figures and drop support for iOS 8. I'll say this: If I were an Uber investor I would be pretty angry. And the story just gets worse from there. This is a story of gross incompetence. It's not an engineering disaster, it's a management disaster. This is classic "inmates running the asylum". ( https://www.goodreads.com/book/show/44098.The_Inmates_Are_Running_the_Asylum https://www.goodreads.com/book/show/44098.The_Inmates_Are_Ru... ) This is the story of some hackers who got in good at a big company with lots of money and sturm und drang so they had no adult supervision while they partied while pretending to be actual engineers. (That they were pretending to themselves too doesn't make it any better.) The writing was on the wall, but rather than own up and take responsibility they double down and keep messing things up for months and years. > This is when the real brilliant engineers started to shine. This is when they get pulled from other tasks to clean up your mess! (cf. Broken window fallacy.) Those real brilliant engineers should be doing other better things than dark rituals to appease the Swift compiler gods. Look, we all make mistakes. There's a blog post from an engineer at a company I used to work for complaining about the pointless middleware I wrote years earlier. (I feel ya bro, I told them at the time that it was pointless.) Here's the thing: I got out of that place when I realized that the folks running it weren't really competent. (And, sure enough, they got pushed out by the investors and now the company is a "Wallstreet darling". So... Yay?) I've burned enough investor money in my career that I don't think it's funny anymore. (I never really did think it was funny, I don't like waste, but I was young. REDACTED was only my second official job, and in hindsight it was really crazy. I mean, they built a skating half-pipe in the office! At the time, everybody thought that was real cool.)
  • Apocryphon 6y ago
    One wonders if this is common everywhere at this scale. Take Facebook's massively bloated iOS app: https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_on_earth_is_the_facebook_app_size_so_large/dluf4rc/ https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_... https://news.ycombinator.com/item?id=14903228 https://news.ycombinator.com/item?id=14903228 And their Android app patching Dalvik at runtime to deal with the huge number of Java methods: https://www.facebook.com/notes/facebook-engineering/under-the-hood-dalvik-patch-for-facebook-for-android/10151345597798920 https://www.facebook.com/notes/facebook-engineering/under-th... And they're not the only ones known for bloated apps, either. Take Amazon's apps for instance: https://news.ycombinator.com/item?id=14902875 https://news.ycombinator.com/item?id=14902875
• throwaheyy 6y ago
  The biggest disaster about this story is having to read it 2 lines at a time. Too broken up, didn't read.
• quux 6y ago
  Is this where RIBs came from? My company adopted that and ran into smaller versions of all the same problems. We moved on to something leaner after a couple of months. Lots of interesting ideas in RIBs though.
  • pandanomic 6y ago
    Yes
  • phink 6y ago
    On top of the discussed complexity and featuritis, I suspect over-engineering, NIH syndrom and "architecture" must have played a role. Misunderstanding, misapplying and blaming MVC to end up with twice the code in 3 or more times more classes.
• ChrisArchitect 6y ago
  anyone else think it's ridiculous that this is in a giant tweet thread? :\
  • ChrisArchitect 6y ago
    oh gosh thanks https://news.ycombinator.com/item?id=25375799 https://news.ycombinator.com/item?id=25375799
• skoocda 6y ago
  Here's the Threader (i.e. Twitlonger) thread for anyone who wants to read the whole thing: https://threader.app/thread/1336890442768547845 https://threader.app/thread/1336890442768547845
• marcod 6y ago
  https://threadreaderapp.com/thread/1336890442768547845.html https://threadreaderapp.com/thread/1336890442768547845.html
• Eduard 6y ago
  Why not release the Swift app version as an additional app, e.g. "Uber Next" while keeping the original version alive until all problems are fixed? This story screams of so many self-made issues.
  • cbrauchli 6y ago
    Because Apple would not have approved it.
  • noway421 6y ago
    They'd want to keep the existing install base.
• Apocryphon 6y ago
  This story at Uber is a fascinating example of the growing pains involved with transitioning to a new language. LinkedIn adopted Swift back in 1.2. I would love to read some war stories of how things there have progressed over the years. Source: https://www.youtube.com/watch?v=X9waDi787uo https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37) At an interview with them a few years ago, I was told they had been moving towards rewriting the app in Objective-C.
  • georgiecasey 6y ago
    the difference is people actually need to use the Uber app /s
• deleted 6y ago
  [deleted]
• torcete 6y ago
  I understand that the fast pace development of the first app versions was also related to have to be faster than uber's rivals.
• valzam 6y ago
  In some other thread someone who regularly attended meetups hosted by Uber in NY said this (paraphrasing): "You could rely on Uber coming up with the most complicated solution to problems they create for themselves. But the food was good" Pretty much sums up anything I have ever read about uber engineering.
• hardwaregeek 6y ago
  I wonder why there wasn't a push to move a lot of the logic onto servers? That's obviously an oversimplified solution but presumably there should be some way to take stuff like localization, payments, etc. and push them onto a server application? This isn't a rhetorical question. I imagine there's a well thought out reason why that wasn't considered.
• rsp1984 6y ago
  The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discovered that our compiled code size was growing a rate of 1.3 mb a week ... But the real problem was the growth curve. It was always eating our winnings back. This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible. The larger community benefited from our learnings. No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen. So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions. No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.
  • rossdavidh 6y ago
    While I think your tone is a bit more critical than it needs to be, as many very intelligent people have made the same mistake, I think the sentiment is correct. But 'standing up to management' is literally not possible if they don't want it to be. They get to make the decisions if they want to, more or less by definition of 'management'. He did say 'we need to bail' to his boss at one point. But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.
• xiphias2 6y ago
  Binary size should have been treated like a limited resource: set a hard limit in the test suite, and let the engineers who commit code in the organization fight it out how to delete unused code when they create new one for a new feature. I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules): https://github.com/geohot/tinygrad https://github.com/geohot/tinygrad Here's the GPU ops part: https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_gpu.py https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_...
• nathanmock 6y ago
  Fun fact: I was able to access a bunch of employee admin screens in early 2015 because of the client side architecture of the app. I noticed around a year later the architecture had changed and always wondered if my article had a role in the transition http://nathanmock.com/archives/how-i-accessed-employee-settings-on-ubers-app http://nathanmock.com/archives/how-i-accessed-employee-setti...
• xfer 6y ago
  He talks about having swift compiler engineers in team. Was swift not opensource back then, so they could optimize the compiler for size instead of reordering optimization passes using simulated annealing?
• otterley 6y ago
  "Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience. So to improve location pickup we changed the location permission to collect signal in the background so we could send the drivers to your current location. People freaked out." This doesn't make much sense. When you open the app to schedule a pickup, and you have location tracking enabled while you use the app, Uber should be apprised as to your current location within seconds. Where you were before you opened the app should not matter. It would be useful if the author could clarify this. As one of the people who started a petition to Apple to disallow developers from disabling location tracking only while using the app (i.e. all the time or never, which Uber tried to force at the time), I am partly responsible for the outcome. But from what I can tell, no customers were worse off as a result. If Uber has data showing otherwise, that would be interesting.
  • taeric 6y ago
    Sounds like a type of kalman filter on the location. Any given reading is likely crap, but combined, you can get fairly accurate. So, the idea is probably too keep a decent sized history of readings so that your could just add the current to that.
  • adamjb 6y ago
    GPS fixes improve over time as the ephemeris gets downloaded and updated so what he claims makes sense. A full ephemeris takes 30 seconds to broadcast from each satellite.
  • StanTwinB 6y ago
    It’s not about collecting info before you open the app. What happens with most people is they open the app, request a ride, then close the app again and do something else until the driver has arrived notification. Location services start out with a wider range and narrow in on you over time, so a longer collection window gives more precision.
• nojvek 6y ago
  I’ve seen many rewrite stories go well. I’ve seen many go south. This just seems like awful management and not spending time early de-risking things. That being said it’s a common problem in growth startup. Prioritization is hard. Too many engineers doesn’t mean problems get solved. They need to be able to talk to each other and execute with what’s essential to users. Growth is a curse and a blessing.
• deleted 6y ago
  [deleted]
• nojvek 6y ago
  Easy to bash on Uber. Partly a fault of Apple releasing swift and calling it 1.0. It’s clear swift didn’t scale for Uber and it should have been in beta. Apple could have made a more rock solid product.
• scottlamb 6y ago
  > As a consequence of all these problems, there began to be a growing movement across all levels of the org that was rallying around the idea of “rewriting the app from scratch” The general sentiment was that the architecture was slowing us down, starting over would be faster. I've read plenty of these stories where a rewrite was a total disaster. (Most famously, Netscape.) I've seen and/or lived through some too. (One where a major distributed system was completely changing its storage model, and it was thought refactoring was impossible. In hindsight, it was achievable and would have saved _years_.) Anyone have examples of regretting a refactoring to the same extent?
• Traubenfuchs 6y ago
  The uber app was never fast and snappy. The uber app crashes regularly, shows fake cars flying across the map, misaligns your drivers car with the black line of the car will take, has spinning cars and countless other bugs. He ends his "twitter thread" explaining the results of them never properly testing their new UI & UX. Horrible. Another billion dollar company just throwing expensive engineers at narrow, single issue problems. No proper engineering strategy, user centered design or connected thinking visible. Just an overengineered mess having hundreds of megabytes of bloat. I rant, but could my company or me do it better? No.
• neop1x 6y ago
  Gone are the days where Twitter body limit was understood as the reason to post short tweets instead of long tales...
  • aembleton 6y ago
    Works well in ThreadReader though: https://threadreaderapp.com/thread/1336890442768547845.html https://threadreaderapp.com/thread/1336890442768547845.html
  • thinkingemote 6y ago
    I know right. I was even more disheartened when someone who I thought might be mature enough and understand the modern internet also used Twitter instead of their own blog to write a series of tweets as a blog, Cory Doctorow. I refuse to go to Twitter to read such stuff (threadreader is not a justification, it's a crappy workaround) and fully endorse the communication of the idea that it sucks.
• junon 6y ago
  Former Uber employee. I can corroborate much of what they say (not having been on those sorts of teams). The depiction of the company culture is accurate.
• steve76 6y ago
  Waiving down a taxi costs a billion dollars. Now I know where my bailout money went. Anyways, I can solve all your problems: Market crashes. Not just Apple and Uber go down, but their investors, and where they have their deposits. All the bad stuff that ruins your life, the hoops you need to jump through, the idea of have to pay money first to work a job in the gig economy, goes with them. Everything costs nothing, or they pay you to take it. We get all the good stuff. .taxi is just there. The local name servers are just there. No one thinks about messing around with it, or using it to hurt people. If they do no bullying attorneys are there to protect them. If you need a ride, and can't make a data request to .taxi in 2020, you are effectively banished from society and deserve what you get. Walk home. People have been doing it for millennia.