4 ms·

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Appl

by rsp1984 6y ago

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discovered that our compiled code size was growing a rate of 1.3 mb a week ... But the real problem was the growth curve. It was always eating our winnings back. This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible. The larger community benefited from our learnings. No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen. So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions. No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.

rossdavidh 6y ago
While I think your tone is a bit more critical than it needs to be, as many very intelligent people have made the same mistake, I think the sentiment is correct. But 'standing up to management' is literally not possible if they don't want it to be. They get to make the decisions if they want to, more or less by definition of 'management'. He did say 'we need to bail' to his boss at one point. But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.
cactus2093 6y ago
What you're ignoring is that some significant percentage of that "bloat" is what leads to more rides and more money. All those tweaks and a/b tests and features that solve problems for specific segments that most users won't ever see, getting shipped week after week, increasing the new user conversion rate by 0.5% here, reducing the rate of canceled trips by 0.3% there, it all adds up over time. Many of the features obviously fail too and don't improve anything they just add bloat, but you can't ever know for sure which features those will be in advance. In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure. You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.
notional 6y ago
It's also interesting to look back at the internal politics of Uber at this time. [1] There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela. Reflecting on one very, very strange year at Uber (Susan Fowler) - https://news.ycombinator.com/item?id=13682022 https://news.ycombinator.com/item?id=13682022 I am an Uber survivor - https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-uber-survivor-c6d6541e632f https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-ub... Uber Founder Travis Kalanick Resigns as C.E.O. - https://news.ycombinator.com/item?id=14600873 https://news.ycombinator.com/item?id=14600873 How Uber Used Secret “Greyball” Tool to Deceive Authorities Worldwide - https://news.ycombinator.com/item?id=13785564 https://news.ycombinator.com/item?id=13785564 Suicide of an Uber engineer: Widow blames job stress - https://news.ycombinator.com/item?id=14200486 https://news.ycombinator.com/item?id=14200486 Uber's Design Meltdown - https://news.ycombinator.com/item?id=11109272 https://news.ycombinator.com/item?id=11109272 Uber employees used the platform to stalk celebrities and their exes - https://news.ycombinator.com/item?id=13163895 https://news.ycombinator.com/item?id=13163895 [1] https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&dateStart=1453248000&page=0&prefix=false&query=uber&sort=byPopularity&type=story https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&...
StanTwinB 6y ago
It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.
- rsp1984 6y ago
  OK, I understand. I didn't mean to imply that you didn't see the problem of hypergrowth and I apologize if it came across this way. However, in the sequence of tweets it does sound like the conclusion / lesson is to throw ever more heroic engineering at the problem or to get better at eng. planning, as opposed to solving the organizational issues and messed up incentive structures at the root of it all. I realize this isn't a small ask. I've worked in BigTech myself where the reward / risk ratio of trying to actually solve problems at the root is basically 0 if you're not SVP or higher. Anyway, I enjoyed the tweets nevertheless. Good luck.
  - StanTwinB 6y ago
    It’s interesting that you saw this was a tale of heroics. I view it as a tale of bad choices and organization failure.

rossdavidh 6y ago

While I think your tone is a bit more critical than it needs to be, as many very intelligent people have made the same mistake, I think the sentiment is correct. But 'standing up to management' is literally not possible if they don't want it to be. They get to make the decisions if they want to, more or less by definition of 'management'. He did say 'we need to bail' to his boss at one point. But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.

cactus2093 6y ago

What you're ignoring is that some significant percentage of that "bloat" is what leads to more rides and more money. All those tweaks and a/b tests and features that solve problems for specific segments that most users won't ever see, getting shipped week after week, increasing the new user conversion rate by 0.5% here, reducing the rate of canceled trips by 0.3% there, it all adds up over time. Many of the features obviously fail too and don't improve anything they just add bloat, but you can't ever know for sure which features those will be in advance. In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure. You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.

notional 6y ago

It's also interesting to look back at the internal politics of Uber at this time. [1] There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela. Reflecting on one very, very strange year at Uber (Susan Fowler) - https://news.ycombinator.com/item?id=13682022 https://news.ycombinator.com/item?id=13682022 I am an Uber survivor - https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-uber-survivor-c6d6541e632f https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-ub... Uber Founder Travis Kalanick Resigns as C.E.O. - https://news.ycombinator.com/item?id=14600873 https://news.ycombinator.com/item?id=14600873 How Uber Used Secret “Greyball” Tool to Deceive Authorities Worldwide - https://news.ycombinator.com/item?id=13785564 https://news.ycombinator.com/item?id=13785564 Suicide of an Uber engineer: Widow blames job stress - https://news.ycombinator.com/item?id=14200486 https://news.ycombinator.com/item?id=14200486 Uber's Design Meltdown - https://news.ycombinator.com/item?id=11109272 https://news.ycombinator.com/item?id=11109272 Uber employees used the platform to stalk celebrities and their exes - https://news.ycombinator.com/item?id=13163895 https://news.ycombinator.com/item?id=13163895 [1] https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&dateStart=1453248000&page=0&prefix=false&query=uber&sort=byPopularity&type=story https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&...

StanTwinB 6y ago

It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.

rsp1984 6y ago
OK, I understand. I didn't mean to imply that you didn't see the problem of hypergrowth and I apologize if it came across this way. However, in the sequence of tweets it does sound like the conclusion / lesson is to throw ever more heroic engineering at the problem or to get better at eng. planning, as opposed to solving the organizational issues and messed up incentive structures at the root of it all. I realize this isn't a small ask. I've worked in BigTech myself where the reward / risk ratio of trying to actually solve problems at the root is basically 0 if you're not SVP or higher. Anyway, I enjoyed the tweets nevertheless. Good luck.
- StanTwinB 6y ago
  It’s interesting that you saw this was a tale of heroics. I view it as a tale of bad choices and organization failure.